Automatic navigation of interactive voice response (IVR) trees on behalf of a human user

Abstract

To utilize an assistant to automatically navigate an interactive voice response (IVR) tree to arrive at a target state during an assisted telephone call.SOLUTION: The assistant can receive input to initiate the assisted telephone call, identify an entity to engage with, on behalf of the user and during the assisted telephone call, based on the input, and identify an IVR tree stored in association with the entity. In various implementations, navigation of the IVR tree can be modified based on interactions detected at a client device subsequently to initiating the assisted telephone call. In various implementations, the assisted telephone call can be initiated from a search interface, and the target state can be associated with a given search result. In various implementations, the IVR tree can be dynamic in that only a subset of candidate states of the IVR tree may be available as the target state.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This application relates to automated navigation of interactive voice response (IVR) trees on behalf of a human user. [Background technology]

[0002] Automated assistants can be interacted with by users through a variety of computing devices, such as smartphones, tablet computers, wearable devices, automotive systems, standalone personal assistant devices, etc. The automated assistant receives input (e.g., spoken, touched, and / or typed) from the user and responds with responsive (e.g., visual and / or audible) output.

[0003] A user can interact with an automated assistant to have the automated assistant perform actions on the user's behalf. As an example, the automated assistant can navigate an interactive voice response (IVR) system, place a telephone call on the user's behalf to perform a given action, and engage in discussions with additional users to perform the action. For example, a user can provide user input requesting that the automated assistant make a restaurant reservation over the phone and on the user's behalf. The automated assistant can initiate a telephone call with a particular restaurant, navigate an IVR system associated with the particular restaurant to make the reservation, and / or provide reservation information to an additional human associated with the particular restaurant to make the reservation. The automated assistant can then notify the user when a human associated with the particular restaurant is an active participant in the telephone call, notify the user whether the restaurant reservation was successfully made on the user's behalf during the telephone call, and / or join the user to the telephone call. Summary of the Invention [Means for solving the problem]

[0004] Implementations disclosed herein are directed to navigating an IVR tree of an interactive voice response (IVR) system associated with an entity using an automated assistant to perform an assisted telephone call with the entity and on behalf of a given user. The IVR tree of the IVR system associated with the entity may be stored in a database accessible at a client device of the given user. Performing the assisted telephone call to navigate the IVR tree may include automatically navigating the IVR tree to a target state of the IVR tree. The target state of the IVR tree may be determined from among multiple states based on user input detected at the client device of the given user. The automated assistant can automatically navigate the IVR tree to a target state of the IVR tree with instances of synthetic speech and rendered as part of an assisted telephone call, which can include injecting a synthetic or human voice into the assisted telephone call so that the synthetic or human voice is audibly perceptible to the IVR system (but not necessarily to a given user), and / or instances of emulated button presses can be rendered as part of the assisted telephone call, which can include injecting the emulated button presses into the call so that they are perceptible by the IVR system. The synthetic, human voice, and / or emulated button presses can be determined based on corresponding values ​​for parameters associated with navigating the IVR tree to the target state. The parameters can be stored in association with an IVR tree associated with an entity, and the corresponding values ​​for the parameters can be retrieved from a database and / or requested from a given user of the client device. In some implementations, the automated assistant can cause the client device to render a notification requesting that the user of the client device join the assisted telephone call. In some additional or alternative implementations, the automated assistant can optionally perform tasks on behalf of the user after automatically navigating the IVR tree to the target state.For example, the automated assistant may engage in a discussion with a human agent associated with the entity that is to perform a task on behalf of the user.

[0005] Some implementations are directed to modifying the navigation of an IVR tree based on a user interaction detected at a client device after initiating an assisted telephone call. The automated assistant can generate and render prompts related to modifying the navigation of the IVR tree and can determine whether to modify the navigation of the IVR tree based on user input received in response to the prompts. In some versions of these implementations, modifying the navigation of the IVR tree may include modifying the target state to an alternative target state or adding an additional target state to be navigated to after navigating to the target state. For example, assume that the automated assistant is navigating to a target state related to a painting department in an IVR tree related to a home improvement entity. In this example, if a given user interacts with the home and landscaping section of a webpage related to the home improvement entity, the user may additionally or alternatively be prompted to navigate to a target state related to the home and landscaping department in the IVR tree related to the home improvement entity. In some additional and / or alternative versions of these implementations, modifying the navigation of the IVR tree may include modifying corresponding values ​​for parameters related to navigating the IVR tree to the target state. Continuing with the above example, if a given user interacts with a web page related to a particular brand of paint, the user may be prompted for the paint brand parameter to utilize the corresponding value associated with that particular brand of paint. In these and other ways, client device and / or network resources may be conserved by preventing the reinitiation of another assistive telephone call performed with an alternate target state and / or alternate value for navigating the IVR tree. Furthermore, the assistive telephone call may be terminated more quickly, thereby shortening the overall duration that computer and / or network resources are utilized in performing the assistive telephone call.Additionally, battery resources of the client device may also be conserved in that a given user is not an active participant in the assisted telephone call and the client device may be placed in a locked or low power state during the assisted telephone call.

[0006] Some additional or alternative implementations are directed to providing IVR deep links to candidate states of an IVR tree along with search results received in response to a search query determined based on user input. In some versions of these implementations, a given IVR deep link may be provided based on recent user interactions at a client device prior to issuing the search query. For example, assume that a user of a client device is interacting with search results related to reporting lost baggage to an airline entity and then issues a search for airline entities. In this example, the automated assistant may provide an IVR deep link related to the target state of reporting lost baggage for selection by a given user of a client device based on recent user interactions (and optionally in a more prominent manner than other IVR deep links that may also be provided). In some additional and / or alternative versions of these implementations, a given IVR deep link may be provided with a base term search query. For example, assume that a given user of a client device issues a search query for "report lost baggage to Example Airlines." In this example, the automated assistant may provide an IVR deep link associated with the target state of reporting lost baggage for selection by a given user of a client device based on search query terms including "report lost baggage" and "Example Airlines." In these and other ways, the amount of user input may be reduced, thereby conserving client device resources because a given user only needs to select the "IVR deep link." Additionally, the assisted phone call may be concluded more quickly, thereby shortening the overall duration that computer and / or network resources are utilized in executing the assisted phone call.Additionally, battery resources of the client device may also be conserved in that a given user is not an active participant in the assisted telephone call and the client device may be placed in a locked or low power state during the assisted telephone call.

[0007] Some additional or alternative implementations are directed to determining the availability of candidate states based on contextual information associated with a given user's client device and / or contextual information associated with a given user of the client device. Contextual information associated with a client device may include, for example, device state information (e.g., on, off, locked, sleep), date and / or time information associated with the client device's current location, the client device's current location, and / or other contextual information generated by the client device. For example, candidate states associated with a restaurant entity may include a takeout candidate state and a delivery candidate state if the restaurant entity is open. However, candidate states associated with a restaurant entity may not include a takeout candidate state or a delivery candidate state if the restaurant entity is closed, or may not include a delivery candidate state if the client device is located beyond the delivery zone even if the restaurant is open. Additionally, contextual information associated with a given user of a client device may include, for example, electronic communications created by or received by the user of the client device, user account information associated with various user accounts of the given user of the client device, financial information of the given user of the client device, photo albums of the given user of the client device, social media profiles of the given user of the client device, user preferences of the user of the client device, personal information of the given user of the client device, and / or other information associated with the given user of the client device. For example, candidate states associated with a hotel entity may include a concierge contact state or a free upgrade requested state if the given user is a gold status member or higher at the hotel entity. However, if the given user is not a gold status member or higher at the hotel entity, the candidate states may omit these states.In these and other ways, client device resources and / or network resources may be conserved by preventing the reactivation of another assisted telephone call performed in connection with a given user's selection of a candidate state that may be unavailable. Furthermore, the assisted telephone call may be terminated more quickly, thereby shortening the overall duration for which computer and / or network resources are utilized in performing the assisted telephone call. Moreover, client device battery resources may also be conserved in that a given user is not an active participant in the assisted telephone call and the client device may be placed in a locked or low-power state during the assisted telephone call.

[0008] Some additional or alternative implementations are directed to rendering candidate states in various manners based on a confidence level for the IVR tree associated with the entity. The confidence level for the IVR tree stored in association with the entity may be based on the number of assisted phone calls with the entity that involve navigating an IVR tree that differs from the IVR tree stored in association with the entity, and optionally based on the degree of difference between the navigated IVR tree and the stored IVR tree. For example, if the confidence level for the IVR tree meets a first threshold indicating that the automated assistant is highly confident in the IVR tree associated with the entity, before or upon initiating an assisted telephone call (e.g., upon pressing "1," "3," or "5" to report lost baggage), the automated assistant can cause a sequence of button presses to be rendered at the client device to inform a given user of the client device how to quickly navigate the IVR tree to one or more of the candidate states, or before or upon initiating an assisted telephone call (e.g., upon pressing "5" to report lost baggage), the automated assistant can cause a selectable graphical element (which may not match the sequence of button presses) to be rendered at the client device that is associated with the candidate states and that, when selected, causes the automated assistant to automatically navigate to a target state associated with the selected selectable graphical element. As another example, if the confidence level for the IVR tree does not meet a first threshold but meets a second threshold indicating that the automated assistant is somewhat confident in the IVR tree associated with the entity, the automated assistant may cause the above-mentioned selectable graphical element associated with the candidate state (which may not match the sequence of button presses) to be rendered at the client device and only when the portion of the IVR tree in which the automated assistant is somewhat confident is traversed.As yet another example, if the confidence level for the IVR tree does not meet the first threshold and a second threshold indicating that the automated assistant is not certain of the IVR tree associated with the entity, the automated assistant may cause the above-mentioned selectable graphical elements associated with the candidate states (which may not match the button press sequence) to be rendered at the client device after the automated assistant perceives options for the IVR tree. In these and other ways, client device and / or network resources may be conserved by preventing the relaunch of another assisted telephone call if the automated assistant provides the user with incorrect information for navigating the IVR tree. Furthermore, the assisted telephone call may be terminated more quickly, thereby shortening the overall duration for which computer and / or network resources are utilized in conducting the assisted telephone call by providing the button presses or selectable element sequence associated with the candidate states. Furthermore, battery resources of the client device may also be conserved in that a given user is not an active participant in the assisted telephone call and the client device may be placed in a locked or low-power state during the assisted telephone call.

[0009] The above description is provided merely as a summary of some implementations disclosed herein. These and other implementations are described in additional detail herein. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram of an exemplary environment that demonstrates various aspects of the disclosure and in which implementations disclosed herein may be practiced. [Figure 2] FIG. 1 illustrates an exemplary interactive voice response (IVR) tree according to various implementations. [Figure 3]1 is a flowchart illustrating an example method for modifying navigation of an interactive voice response (IVR) tree based on user interactions at a client device associated with a given user, according to various implementations. [Figure 4A] 1A-1C illustrate various non-limiting examples of user interfaces related to modifying the navigation of an interactive voice response (IVR) tree based on user interactions at a client device associated with a given user, according to various implementations. [Figure 4B] 1A-1C illustrate various non-limiting examples of user interfaces related to modifying the navigation of an interactive voice response (IVR) tree based on user interactions at a client device associated with a given user, according to various implementations. [Figure 4C] 1A-1C illustrate various non-limiting examples of user interfaces related to modifying the navigation of an interactive voice response (IVR) tree based on user interactions at a client device associated with a given user, according to various implementations. [Figure 5] 1 is a flowchart illustrating an exemplary method for navigating an interactive voice response (IVR) tree from a search interface on a client device associated with a given user, according to various implementations. [Figure 6A] 1A-1C illustrate various non-limiting examples of user interfaces associated with navigating an interactive voice response (IVR) tree from a search interface on a client device associated with a given user, according to various implementations. [Figure 6B] 1A-1C illustrate various non-limiting examples of user interfaces associated with navigating an interactive voice response (IVR) tree from a search interface on a client device associated with a given user, according to various implementations. [Figure 6C]1A-1C illustrate various non-limiting examples of user interfaces associated with navigating an interactive voice response (IVR) tree from a search interface on a client device associated with a given user, according to various implementations. [Figure 6D] 1A-1C illustrate various non-limiting examples of user interfaces associated with navigating an interactive voice response (IVR) tree from a search interface on a client device associated with a given user, according to various implementations. [Figure 6E] 1A-1C illustrate various non-limiting examples of user interfaces associated with navigating an interactive voice response (IVR) tree from a search interface on a client device associated with a given user, according to various implementations. [Figure 7] 1 is a flowchart illustrating an example method for navigating a dynamic interactive voice response (IVR) tree, according to various implementations. [Figure 8A] 1A-1C illustrate various non-limiting examples of user interfaces associated with navigating a dynamic interactive voice response (IVR) tree, according to various implementations. [Figure 8B] 1A-1C illustrate various non-limiting examples of user interfaces associated with navigating a dynamic interactive voice response (IVR) tree, according to various implementations. [Figure 8C] 1A-1C illustrate various non-limiting examples of user interfaces associated with navigating a dynamic interactive voice response (IVR) tree, according to various implementations. [Figure 8D] 1A-1C illustrate various non-limiting examples of user interfaces associated with navigating a dynamic interactive voice response (IVR) tree, according to various implementations. [Figure 9] FIG. 1 illustrates an exemplary architecture of a computing device according to various implementations. DETAILED DESCRIPTION OF THE INVENTION

[0011] 1, a block diagram of an exemplary environment is shown that demonstrates various aspects of the present disclosure and in which implementations disclosed herein may be practiced. A client device 110 is shown in FIG. 1 and, in various implementations, includes a user input engine 111, a device state engine 112, a rendering engine 113, a search engine 114, a speech recognition engine 120A1, a natural language understanding (“NLU”) engine 130A1, and a speech synthesis engine 140A1.

[0012] The user input engine 111 can detect various types of user input at the client device 110. The user input detected at the client device 110 can include speech input detected via a microphone of the client device 110 and / or additional speech input transmitted to the client device 110 from an additional client device of an additional user associated with the entity during the assisted telephone call, touch input detected via a user interface input device (e.g., a touchscreen) of the client device 110, and / or typed input detected via a user interface input device (e.g., via a virtual keyboard on the touchscreen) of the client device 110. The additional user associated with the entity can be, for example, an additional human participant associated with an additional client device, an additional automated assistant associated with an additional client device of an additional user, and / or other additional users.

[0013] The assisted telephone calls described herein may be performed using a variety of voice communication protocols (e.g., Voice over Internet Protocol (VoIP), Public Switched Telephone Network (PSTN), and / or other telephone communication protocols. As described herein, synthetic speech may be rendered as part of the assisted telephone call, which may include injecting the synthetic speech into the call so that it is perceptible by at least one of the participants in the assisted telephone call. As also described herein, emulated button presses may be rendered as part of the assisted telephone call, which may include injecting the emulated button presses into the call so that it is perceptible by at least one of the participants in the assisted telephone call. The synthetic speech may be rendered as part of the assisted telephone call, which may include injecting the emulated button presses into the call so that it is perceptible by at least one of the participants in the assisted telephone call. The audible output may be generated and / or injected by a client device 110 that is one of the endpoints of the assisted telephone call, and / or may be generated and / or injected by a server that is in communication with client device 100 and is also connected to the assisted telephone call. As also described herein, the audible output may also be rendered outside of the assisted telephone call that does not involve injecting the audible output into the call, but the audible output may be detected by a microphone of a client device 110 connected to the assisted telephone call and therefore perceptible over the call. In some implementations, the assisted telephone call may be optionally muted and / or filtering may be utilized to mitigate perception within the assisted telephone call of audible output rendered outside of the assisted telephone call.

[0014] In various implementations, automated assistant 115 (schematically illustrated by dashed lines in FIG. 1 ) can perform an assisted telephone call at client device 110 over network 190 (e.g., Wi-Fi, Bluetooth, near-field communications, a local area network, a wide area network, and / or other networks) using assisted telephone call system 160. Assisted telephone call system 160, in various implementations, includes speech recognition engine 120A2, NLU engine 130A2, speech synthesis engine 140A2, and assisted telephone call engine 150. In some implementations, automated assistant 115 can utilize assisted telephone call system 160 to navigate an interactive voice response (IVR) tree of an IVR system associated with an entity and on behalf of the user of client device 110 during an assisted telephone call with an additional user. In some additional or alternative implementations, automated assistant 115 can utilize assisted telephone call system 160 to perform tasks on behalf of the user of client device 110 during an assisted telephone call with an additional user. In some additional or alternative implementations, the automated assistant 115 can utilize the assisted telephone call system 160 to navigate non-interactive systems (e.g., hold systems, voice mail systems) and / or recognize if the IVR system is malfunctioning or if nothing is happening during an assistant telephone call (and optionally terminate the assisted telephone call and notify the user).

[0015] Moreover, in some implementations, after navigating the IVR tree and before performing any task on behalf of the user of client device 110, automated assistant 115 can obtain consent from the additional user to engage in a discussion with automated assistant 115. For example, automated assistant 115 can obtain consent when connecting with a human participant to process audio data corresponding to the human participant's voice. If automated assistant 115 obtains consent from the additional user, automated assistant 115 can verify that a target state of the IVR tree has been reached (e.g., as described below with respect to validation engine 155) and / or can perform the task on behalf of the user (e.g., as described below with respect to task engine 157). However, if the automated assistant 115 does not obtain consent from the additional user, the automated assistant 115 may cause the client device 110 to render (e.g., using the rendering engine 113) a notification to the user of the client device 110 indicating that the user requires that the assisted telephone call be taken over and / or that the assisted telephone call be terminated and a notification to the user of the client device 110 be rendered (e.g., using the rendering engine 113).

[0016] As described in more detail below, the automated assistant 115 can perform an assisted telephone call using the assisted telephone call system 160 in response to detecting user input from a user of the client device 110 to initiate a telephone call using the automated assistant 115. In some implementations, the automated assistant 115 can determine a target state from among multiple candidate states for navigation of an IVR tree associated with an entity to be involved on behalf of the user of the client device 110 and during the assisted telephone call based on user input received at the client device 110. In some versions of these implementations, the automated assistant 115 can receive a selection of the target state before the initiation of the assisted telephone call or once the assisted telephone call is initiated. In some additional or alternative versions of these implementations, the automated assistant 115 can receive a sequence of selections that result in the target state before the initiation of the assisted telephone call or once the assisted telephone call is initiated. The automated assistant 115 can navigate the IVR tree until it arrives at the target state and can cause a notification that the automated assistant 115 has arrived at the target state to be rendered at the client device 110 (e.g., using the rendering engine 113). In various implementations, the automated assistant 115 can additionally perform a task on behalf of the user upon reaching the target state and cause a notification including the results of performing the task to be rendered at the client device 110 (e.g., using the rendering engine 113).

[0017] 1 , assisted telephone call system 160 may be implemented remotely (e.g., via a server and / or other remote client device) and in communication with client device 110. It should be understood that while assisted telephone call system 160 is illustrated in FIG. 1 as being implemented remotely and in communication with client device 110 via network 190, this is for purposes of example and not intended to be limiting. For example, in various implementations, assisted telephone call system 160 may be implemented locally on client device 110. As another example, in various implementations, assisted telephone call system 160 may be implemented remotely in assistant telephone call system 160 without any connection to client device 110 (e.g., cloud-based assisted telephone calling), and client device 110 may optionally be in an off state or not connected to any network (e.g., as determined using device state engine 112). 1 as being implemented both locally on client device 110 and remotely in assisted telephone call system 160, it should also be understood that this is for purposes of example and not intended to be limiting. For example, in various implementations, automated assistant 115 may be implemented locally on client device 110, while in other implementations, automated assistant 115 may be implemented locally on client device 110 and interact with a separate cloud-based automated assistant (e.g., forming a logical instance of automated assistant 115).

[0018] In an implementation in which user input engine 111 detects a user's speech input via a microphone of client device 110 and / or receives audio data capturing additional speech input from an additional user transmitted to client device 110 from an additional client device (e.g., during an assisted phone call), speech recognition engine 120A1 of client device 110 can process the captured speech input and / or audio data capturing the additional speech input using speech recognition model 120A. Speech recognition engine 120A1 can generate recognized text corresponding to the speech input and / or the additional speech input based on processing the audio data. Additionally, NLU engine 130A1 of client device 110 can process the recognized text generated by speech recognition engine 120A1 using NLU model 130A to determine the intent contained in the speech input and / or the additional speech input. For example, if client device 110 detects a speech input from a user of "call Hypothetical Airline to report lost luggage," client device 110 can process audio data capturing the speech input using speech recognition model 120A to generate recognized text corresponding to the speech input of "call Hypothetical Airline to report lost luggage," and can process the recognized text using NLU model 130A to determine at least a first intent to initiate a call and a second intent to report lost luggage.As another example, if client device 110 detects the additional speech input of "tell us briefly why you are calling," client device 110 can process audio data capturing the additional speech input using speech recognition model 120A to generate recognized text corresponding to the additional speech input of "tell us briefly why you are calling," and can process the recognized text using NLU model 130A to determine the intent to request natural language input that includes a request for information related to why the assisted phone call was initiated (e.g., to report lost baggage). In some versions of these implementations, client device 110 can transmit the audio data, the recognized text, and / or the intent to assisted phone call system 160.

[0019] In other implementations, when user input engine 111 detects a user's speech input via the microphone of client device 110 and / or receives audio data capturing additional speech input from an additional user transmitted from an additional client device to client device 110 (e.g., during the assisted phone call and / or during an ongoing call), automated assistant 115 can cause client device 110 to transmit the speech input and / or audio data capturing the additional speech input to assisted phone call system 160. Speech recognition engine 120A2 and / or NLU engine 130A2 of assisted phone call system 160 can process the audio data capturing the speech input and / or audio data capturing the additional oral utterances in a manner similar to that described above with respect to speech recognition engine 120A1 and / or NLU engine 130A1 of client device 110. In some additional and / or alternative implementations, speech recognition engine 120A1 and / or NLU engine 130A1 of client device 110 may be used in conjunction with speech recognition engine 120A2 and / or NLU engine 130A2 of assisted telephone call system 160 in a distributed manner. Moreover, speech recognition model 120A and / or NLU model 130A may be stored locally on client device 110 and / or remotely at a server in communication with client device 110 and / or assisted telephone call system 160 via network 190.

[0020] In various implementations, speech recognition model 120A is an end-to-end speech recognition model such that speech recognition engines 120A1 and / or 120A2 can use the model to directly generate recognized text corresponding to speech input. For example, speech recognition model 120A can be an end-to-end model used to generate recognized text character-by-character (or other token-by-token). One non-limiting example of such an end-to-end model used to generate recognized text character-by-character is a recurrent neural network transducer (RNN-T) model. RNN-T models are a form of sequence-to-sequence model that does not employ an attention mechanism. In other implementations, speech recognition model 120A is not an end-to-end speech recognition model such that speech recognition engines 120A1 and / or 120A2 can instead generate predicted phonemes (and / or other representations). For example, the predicted phonemes (and / or other representations) may then be utilized by speech recognition engines 120A1 and / or 120A2 to determine recognized text that matches the predicted phonemes. In doing so, speech recognition engines 120A1 and / or 120A2 may optionally employ decoding graphs, dictionaries, and / or other resources.

[0021] In implementations in which user input engine 111 detects touch input and / or typed input via a user interface input device of client device 110, automated assistant 115 can cause a display of the touch input and / or the displayed typed input to be sent from client device 110 to assisted telephone call system 160. In some versions of these implementations, the display of the touch input and / or the displayed typed input can include hidden text or a representation of the touch input and / or text of the typed input, and hidden text speech recognition model 120A is an end-to-end speech recognition model and can be processed using NLU model 130A to determine the hidden text and / or the intent for the text.

[0022] As described herein, assisted telephone call engine 150 of assisted telephone call system 160 can further process recognized text generated by speech recognition engines 120A1 and / or 120A2, hidden text or representations of touch input detected at client device 110, hidden text of typed input detected at client device 110, and / or intent determined by NLU engines 130A1 and / or 130A2. Assisted telephone call engine 150, in various implementations, includes entity identification engine 151, interactive voice response (IVR) engine 152, parameter engine 153, user interaction engine 154, verification engine 155, notification engine 156, and task execution engine 157.

[0023] The entity identification engine 151 can identify entities to be engaged with on behalf of a user of the client device 110. The entities may be, for example, person entities, business entities, location entities, and / or other entities. For example, a person entity may be a friend entity, a family member entity, a coworker entity, and / or other specific types of person entities. Furthermore, a business entity may be a restaurant entity, an airline entity, a hotel entity, a salon entity, a doctor's office entity, and / or other specific types of business entities. Moreover, a location entity may be a school entity, a museum entity, a library entity, a park entity, and / or other specific types of location entities. In some implementations, the entity identification engine 151 can also determine a specific entity for the identified entity. For example, a specific entity for a person entity may be a person's name (e.g., Jane Doe, John Doe, etc.), a specific entity for a business entity may be a business name (e.g., virtual airline, exemplary airline, etc.), and a specific entity for a location entity may be a location name (e.g., virtual university, exemplary national park, etc.). The entities described herein may be defined at various levels of granularity, but for simplicity they are collectively referred to herein as "entities."

[0024] In some implementations, the entity identification engine 151 can identify an entity to be engaged on behalf of the user of the client device 110 based on user interaction with the client device 110 prior to initiating an assisted phone call using the automated assistant 115. In some versions of those implementations, the entity can be identified in response to receiving user input to initiate the assisted phone call. For example, if the user of the client device 110 directs input (e.g., spoken or touched) to a call interface element of a software application (e.g., related to a contact in a contacts application, related to a search result in a browser application, and / or related to another callable entity contained in another software application), the entity identification engine 151 can identify the entity associated with the call interface element. For example, if the user input is targeted to a call interface element related to “Hypothetical Airlines” in the browser application, the entity identification engine 151 can identify “Hypothetical Airlines” (or, more generally, a business entity or a restaurant entity) as the entity to be engaged on behalf of the user of the client device 110 during the assisted phone call.

[0025] Moreover, entity identification engine 151 can identify entities from among multiple entities stored in entity database 151A. In some implementations, multiple entities stored in entity database 151A can be indexed by entity and / or specific type of entity. For example, if entity identification engine 151 identifies the entity "Virtual Airlines," "Virtual Airlines" can be indexed in entity database 151A as a business entity and, optionally, further indexed as an airline entity. By storing and indexing the identified entities in entity database 151A, entity identification engine 151 can easily identify and retrieve the entities, thereby reducing subsequent processing to identify the entities when they are encountered in future assisted telephone calls. Moreover, in various implementations, each entity can be associated with a task in entity database 151A.

[0026] IVR engine 152 can identify an IVR tree (and optionally, audio data associated with the IVR tree) associated with an entity identified via entity identification engine 151. In some implementations, the IVR tree associated with the identified entity can be stored in association with the identified entity in IVR tree database 152A. For example, an IVR tree stored in IVR tree database 152A can be mapped to a corresponding entity stored in entity database 151A. In some additional or alternative implementations, the IVR tree can be stored in association with the corresponding entity in entity database 151A, and IVR tree database 152A can be omitted. Each IVR tree can include a corresponding plurality of candidate states, and automated assistant 115 can determine a target state of a given IVR tree from among the corresponding plurality of candidate states based on user input detected via user input engine 111. During an assisted telephone call, the automated assistant can automatically navigate the IVR tree to the target state.

[0027] In some implementations, and before receiving user input to initiate the assisted telephone call, automated assistant 115 (and / or additional instances of the automated assistant) may initiate automated telephone calls with multiple entities stored in entity database 151A. During these automated telephone calls, automated assistant 115 (and / or additional instances of the automated assistant) may navigate an IVR system associated with the entities to partition a corresponding IVR tree associated with each of the multiple entities, and mappings of the corresponding IVR trees may be stored in IVR tree database 152A. In some versions of these implementations, automated assistant 115 (and / or additional instances of the automated assistant) may initiate multiple automated telephone calls to a given entity of the multiple entities to map variations of a given IVR tree associated with the given entity. For example, automated assistant 115 may initiate a first automated telephone call with the restaurant entity when the restaurant entity is open and a second automated telephone call with the restaurant entity when the restaurant is closed. In other words, IVR engine 152 can cause automated assistant 115 to map a given IVR tree associated with a restaurant entity in different contexts (e.g., contexts associated with a given client device or contexts associated with a user) to determine whether the given IVR tree is dynamic. As a result, when IVR tree 152 identifies an IVR tree associated with the entity, an IVR tree that matches the current context of client device 110 and / or a user of client device 110 can be identified (e.g., as described with respect to FIGS. 7 and 8A-8D). In some additional or alternative implementations, the entity can also provide a representation of the IVR tree, and the IVR tree can be stored in IVR tree database 152A, prior to receiving user input to initiate the assisted phone call.

[0028] In various implementations, automated assistant 115 can track changes, and optionally the extent of changes, in an IVR tree over time based on assisted telephone calls initiated in response to user input to initiate the assisted telephone call. These changes can be identified based on state differences between a stored IVR tree and the IVR tree currently being navigated by automated assistant 115 and / or stored audio data associated with the IVR tree and audio data associated with the IVR tree currently being navigated by automated assistant 115. For example, if automated assistant 115 determines that the IVR tree associated with a given entity and traversed during the assisted telephone call differs from the IVR tree stored in IVR tree database 152A (e.g., the currently navigated IVR tree includes a new introductory message, a different internal or top node, etc.), automated assistant 115 can store the different state of the currently navigated IVR tree together with the IVR tree already stored associated with the given entity in IVR tree database 152A. Additionally, the inclusion of a new introduction message may be used as a signal to automated assistant 115 that the IVR tree may contain other downstream changes and to monitor for those changes (if possible) rather than attempting to navigate the IVR tree immediately. As a result, automated assistant 115 may. As another example, if automated assistant 115 determines that audio data received during navigation of an IVR tree differs from audio data stored in association with the IVR tree (e.g., based on comparing the acoustic characteristics of the received audio data with the stored audio data and / or comparing the stored audio data with recognized text corresponding to the received audio data), automated assistant 115 may store the different audio data of the currently navigated IVR tree along with the IVR trees already stored in association with the given entity in IVR tree database 152A.If the difference between the stored IVR tree and the navigated IVR tree persists over a threshold number of assisted phone calls with the given entity, a different IVR tree may replace the IVR tree associated with the given entity. In some versions of these implementations, different IVR trees may only be considered for replacing an IVR tree if they are encountered within the same context (e.g., determined based on contextual information associated with the client device 110 and / or the user of the client device 110). In some versions of these implementations, the confidence level associated with the IVR tree associated with the given entity may be based on how frequently the IVR tree traversed during the assisted phone call matches the IVR tree stored in association with the IVR tree in the IVR tree database 152A. In some further versions of these implementations, and as described with respect to FIGS. 6B-6E , the automated assistant 115 may render multiple candidate states in different manners based on the confidence level and whether the confidence level meets one or more thresholds.

[0029] In some implementations, parameter engine 153 can identify parameters associated with navigating an IVR tree identified via IVR engine 152 based on entities identified via entity identification engine 151. The parameters for the IVR tree can be stored in association with the IVR tree identified via IVR engine 152 and can vary based on a navigation path for navigating the IVR tree to a target state. Automated assistant 115 can navigate the IVR tree using corresponding values ​​for the parameters. In some implementations, corresponding values ​​for the parameters can be stored in user profile database 153A, and automated assistant 115 can retrieve corresponding values ​​for the parameters from user profile database 153A in response to identifying parameters associated with navigating an IVR tree and without requesting any values ​​from the user of client device 110. In some additional or alternative versions of these implementations, automated assistant 115 can prompt the user for corresponding values ​​for the parameters in response to identifying parameters associated with navigating an IVR tree and in response to determining that the corresponding values ​​cannot be retrieved from user profile database 153A. For example, the automated assistant 115 can initiate a discussion with the user of the client device and cause one or more prompts to be rendered at the client device 110 via the rendering engine 113 to request corresponding values ​​for any parameters that cannot be resolved using information from the user profile database 153A.

[0030] User profile database 153A may contain information about the user from a variety of sources including, for example, electronic communications (e.g., emails, text messages, voice messages, etc.) related to the user of client device 110, user account information (e.g., account status information, account numbers, account usernames, account passwords, etc.) related to various user accounts of the user of client device 110, application information from applications (e.g., smart device applications, notes applications, reminder applications, phone applications, calendar applications, and / or other software applications) installed on and / or accessible by client device 110, payment information of the user of client device 110 (e.g., bank account numbers, credit card numbers, payment application information, etc.), photo albums of the user of client device 110, social media profiles of the user of client device 110, user preferences of the user of client device 110, personal information of the user of client device 110 (e.g., name, phone number, physical address, email address, social security number, date of birth, etc.), and / or other restricted access information. For example, to navigate an IVR tree associated with an airline entity to a target state associated with canceling a flight reservation, parameter engine 153 can identify parameters for canceling a flight reservation and can determine values ​​for the parameters. For example, a flight confirmation number parameter can be identified, where the flight confirmation number can be retrieved from a flight confirmation email, and a frequent flyer number parameter can be identified, where the frequent flyer number for the user of client device 110 associated with the airline whose flight is being canceled can be retrieved from an application associated with that airline.In contrast, to navigate an IVR tree associated with an airline entity to a different target state associated with changing a flight reservation, the parameter engine 153 may identify the parameters and corresponding values ​​described above, but may also identify a date or time parameter for the new flight and a destination parameter for the new flight, and corresponding values ​​for those parameters may be requested and provided to the user of the client device 110.

[0031] For example, briefly referring to FIG. 2, an exemplary IVR tree 200 is shown. The IVR tree 200 shown in FIG. 2 includes multiple nodes connected by multiple edges. The multiple nodes may include, for example, at least one root node and at least one leaf node. For example, as shown in FIG. 2, the IVR tree includes a root node 211, multiple internal nodes 221, 222, and 223, and multiple leaf nodes 231, 232, 233, 234, 235, and 236. The leaf nodes 231, 232, 233, 234, 235, and 236 can correspond to multiple candidate states, and a given one of the leaf nodes 231, 232, 233, 234, 235, and 236 can correspond to a target state. In particular, the multiple nodes are connected by multiple corresponding edges. These nodes and / or edges may be stored as an IVR tree in IVR tree database 152A (and optionally, along with audio data associated with each of the nodes and / or edges). Each of the corresponding edges may be traversed based on input generated by automated assistant 115 during an assisted telephone call. For example, the corresponding edges may be traversed based on input corresponding to numeric indicators that may be emulated as button presses by automated assistant 115 (e.g., pressing “1” to traverse from root node 211 to internal node 1 221, pressing “2” to traverse from root node 211 to internal node 2 222, pressing “3” to traverse from root node 211 to leaf node 1 231, etc., as shown by the edges in FIG. 2 ), or based on input corresponding to free-form natural language input (e.g., saying “book a flight” to traverse from root node 211 to internal node 1 221).

[0032] In some implementations, the free-form natural language input can be synthesized speech audio data generated by the automated assistant 115, including synthesized speech to be rendered at an additional client device for traversing along the edges of the IVR tree. For example, the automated assistant can generate synthesized speech audio data including the display of the numerical indicators described above and / or a short summary of why the assisted telephone call was placed. In some additional or alternative implementations, the free-form natural language input can be audio data that captures the speech input of a user of the client device 110 and is provided before initiating the assisted telephone call and / or during the assisted telephone call. For example, if the IVR tree requires the user to provide verbal utterances to verify the identity of the user of the client device 110 (e.g., using voice recognition), the audio data can capture the user's speech input.

[0033] In various implementations, the automated assistant 115 can track the navigation of the IVR tree 200. By tracking the navigation of the IVR tree 200, the automated assistant 115 can determine when to provide input (e.g., synthesized speech, emulated button presses, and / or audio data) to navigate the IVR tree 200. For example, the automated assistant 115 can track the initiation of an assisted telephone call beginning at the root node 211 and begin traversing along the edges of the IVR tree 200 to the internal nodes 221, 222, and 223 and / or the leaf nodes 231, 232, 233, 234, 235, and 236 to navigate the IVR tree 200 to a target state determined based on user input detected at the client device 110. As the automated assistant 115 traverses along the edges to different nodes to arrive at the target state, the traversed edges and / or nodes can be tracked by the automated assistant 115. In this example, if the IVR tree 200 includes an introductory message before presenting the main menu associated with the root node 211, the automated assistant 115 can determine that the introductory message is new and can withhold any input for navigating the IVR tree until the main menu associated with the root node 211 is presented (or until any other change is presented). As another example, the automated assistant 115 can determine that any input to be provided for navigating the IVR tree to a target state may not be provided until the IVR system provides all of the options for navigating the IVR tree (e.g., the IVR tree does not support providing input at any time). In this example, the automated assistant 115 may withhold any input for navigating the IVR tree until all of the options for navigating the IVR tree are presented. In some implementations, the automated assistant 115 may return to a previous node traversed during navigation of the IVR tree 200.

[0034] As a non-limiting example, assume that IVR tree 200 corresponds to an IVR tree associated with a hypothetical entity, namely, an exemplary airline. In this example, root node 211 may correspond to a main menu of an IVR system associated with the exemplary airline, interior nodes may correspond to intermediate states 221, 222, and 223, and leaf nodes 231, 232, 233, 234, 235, and 236 may correspond to target states for the IVR tree. Assume further that leaf node 1 231 is associated with the target state "speak with a representative," leaf node 2 232 is associated with the target state "upcoming flight information," leaf node 3 233 is associated with the target state "report lost baggage," leaf node 4 234 is associated with the target state "book a flight," leaf node 5 235 is associated with the target state "change flight reservation," and leaf node 6 236 is associated with the target state "cancel flight reservation." In some implementations, the target states to which the automated assistant 115 navigates the IVR tree during the assisted phone call can be determined based on user input detected via the user input engine 111 of the client device 110 before initiating the assisted phone call and / or during the assisted phone call.

[0035] For example, assume that user input is received at client device 110 indicating that the user wishes to report lost baggage. In this example, automated assistant 115 may determine that it should navigate to leaf node 3 233, which is associated with the target state "Report Lost Baggage." In this example, because IVR tree 200 was previously mapped by automated assistant 115 (and / or additional automated assistants), as described above with respect to IVR engine 152, automated assistant 115 may know that to report the lost baggage, it should traverse from root node 211 to internal node 1 221 and from internal node 1 221 to leaf node 3 233. In navigating from root node 211 to leaf node 3 233 and / or upon arriving at leaf node 3 233, automated assistant 115 may cause corresponding values ​​for parameters required while navigating IVR tree 200 to be rendered at the additional client device associated with the entity. In this example, in response to the corresponding values ​​for the parameters being requested, the automated assistant 115 can render, at additional client devices associated with the entity, corresponding values ​​for flight information (e.g., flight confirmation number, departure destination, arrival destination) related to a recent flight, personal information (e.g., name and phone number) of the user of the client device 110, and / or other corresponding values ​​for the parameters. Upon arriving at leaf node 3 233, the automated assistant 115 can optionally generate (e.g., via notification engine 156) a notification to be rendered at the client device 110 (e.g., via rendering engine 113) requesting that the user join an assisted phone call and / or alerting the user of the client device 110 of the results of reporting lost baggage.

[0036] As another example, assume that user input is received at client device 110 indicating that the user wishes to cancel a lost reservation. In this example, automated assistant 115 may determine that it should navigate to leaf node 6 236, which is associated with the target state "Report Lost Baggage." In this example, because IVR tree 200 was previously mapped by automated assistant 115 (and / or additional automated assistants), as described above with respect to IVR engine 152, automated assistant 115 may know to traverse from root node 211 to internal node 1 221, from internal node 1 221 to internal node 3 223, and from internal node 3 223 to leaf node 6 236 to cancel the flight reservation. In navigating from root node 211 to leaf node 6 236 and / or upon arriving at leaf node 6 236, automated assistant 115 may cause corresponding values ​​for parameters required while navigating IVR tree 200 to be rendered at the additional client device associated with the entity. In this example, in response to the corresponding values ​​for the parameters being requested, the automated assistant 115 can render, at additional client devices associated with the entity, corresponding values ​​for flight information associated with the upcoming flight (e.g., flight confirmation number, departure destination, arrival destination), account information associated with the exemplary airline account (e.g., frequent flyer number), and / or other corresponding values ​​for the parameters. Upon arriving at leaf node 6 236, the automated assistant 115 can optionally generate (e.g., via notification engine 156) a notification to be rendered at the client device 110 (e.g., via rendering engine 113) requesting that the user join an assisted phone call and / or alerting the user of the client device 110 of the result of canceling the flight.

[0037] While IVR tree 200 in FIG. 2 is shown as having particular nodes configured in a particular manner based on corresponding edges, it should be understood that this is for purposes of example and not limitation, and that IVR trees can be configured in a virtually infinite variety of ways. Moreover, some nodes of IVR tree 200 may or may not be available to a user of client device 110 based on some context (e.g., as described with respect to FIGS. 7 and 8A-8D). For example, assume a given IVR tree is associated with a restaurant entity. In this example, when the restaurant entity is open, a target state related to placing a takeout order may be available. In contrast, when the restaurant entity is closed, a target state related to placing a takeout order may not be available.

[0038] 1 , user interaction engine 154 can detect user interactions at client device 110. User interactions can include, for example, search interactions with a web browser or web browser application, browsing interactions with a web browser or web browser application, navigation interactions with a web browser or navigation application, assistant interactions with automated assistant 115, and / or other user interactions. In some implementations, user interaction engine 154 can detect user interactions of a user of client device 110 at client device 110 or at an additional client device in communication with client device 110 (e.g., via network 190) after automated assistant 115 begins executing an assisted telephone call. In some versions of these implementations, automated assistant 115 can determine whether the detected user interaction relates to an entity engaged on behalf of the user of client device 110 during the assisted telephone call. If the automated assistant 115 determines that the detected user interaction is related to an entity, the automated assistant 115 may generate one or more prompts (e.g., via the notification engine 156) asking the user of the client device 110 whether the navigation of the IVR tree should be modified. In some further versions of these implementations, modifying the navigation of the IVR tree may include navigating to a different target state. For example, if the automated assistant 115 initiates an assisted telephone call to navigate an IVR tree related to a home improvement entity to navigate the IVR tree to a target state related to the painting department of the home improvement entity, and the user of the client device 110 subsequently interacts with a web page related to the home and garden department of the home improvement entity, the automated assistant 115 may additionally or alternatively generate one or more prompts asking the user whether to navigate to the target state related to the home and garden department.In other further versions of these implementations, modifying the navigation of the IVR tree may include navigating the same IVR tree using different parameters. For example, if the automated assistant 115 initiates an assisted telephone call to navigate an IVR tree associated with a home improvement entity to navigate the IVR tree to a target state associated with the home improvement entity's paint department, and the user of the client device 110 subsequently interacts with a web page associated with a particular paint manufacturer, the automated assistant 115 may generate one or more prompts asking the user whether to use the particular paint manufacturer as the corresponding value for the parameter in navigating to the target state associated with the paint department. Modifying the navigation of the IVR tree is described in more detail below (e.g., with respect to FIGS. 3 and 4A-4C).

[0039] In some additional or alternative implementations, the user interaction engine 154 can detect a user interaction of the user of the client device 110 at the client device 110 or at an additional client device in communication with the client device 110 (e.g., via the network 190) before the automated assistant 115 begins executing the assisted phone call. In some versions of these implementations, the automated assistant 115 can identify an entity associated with the user interaction and, if an assisted phone call is subsequently initiated with the identified entity, can cause candidate states of the IVR tree to be rendered based on the user interaction. For example, if a user is navigating a web page associated with a hypothetical airline entity, i.e., an exemplary airline, while re-searching how to report lost baggage before initiating an assisted phone call with the exemplary airline, the automated assistant 115 can render a selectable graphical element (alone or more prominently than other selectable elements) associated with a target state related to reporting lost baggage for the IVR tree associated with the exemplary airline. In other words, the automated assistant 115 can cause a selectable graphical element to be rendered as an “IVR deep link” to a target state in the IVR tree based on the user re-searching how to report lost baggage before initiating an assisted telephone call (e.g., as described with respect to Figures 5 and 6A).

[0040] In some implementations, the validation engine 155 can validate whether a state navigated by the automated assistant 115 during an assisted phone call is actually a target state. In some versions of these implementations, when a human agent associated with an entity joins the assisted phone call, the automated assistant 115 can process received audio data capturing the speech input of the human agent to verify that the state navigated by the automated assistant 115 is actually the target state. For example, assume that the automated assistant 115 is navigating an IVR tree associated with a home improvement entity to a target state associated with the home improvement entity's painting department. Further assume that the human agent takes over the assisted phone call from the IVR system associated with the home improvement entity and provides the speech input, "Thanks for calling the paint department, how can I help today?" In this example, verification engine 155 can cause automated assistant 115 to process audio data capturing the speech input using speech recognition engines 120A1 and / or 120A2 and can determine that the speech input includes “paint department” based on processing the audio data capturing the speech input. In some additional or alternative versions of these implementations, the speech input can be received in response to synthesized speech audio data including synthesized speech to verify the target state of navigating an IVR tree. Continuing with the above example, assume that a human agent takes over an assisted phone call from an IVR system associated with a home improvement entity and provides the speech input “Hello.” In this example, verification engine 155 can cause automated assistant 115 to render, at an additional client device associated with the human agent, the synthesized speech of “Hello, is this the paint department?”Verification engine 155 can use speech recognition engines 120A1 and / or 120A2 to have automated assistant 115 process audio data capturing speech input responsive to synthesized speech and determine whether the speech input verifies that the target state has been reached (e.g., "yes," "this is the paint department," or some other affirmation).

[0041] In various implementations, if a state arrived at when navigating the IVR does not correspond to the target state, the verification engine 155 can cause the automated assistant 115, at an additional client device associated with the human agent, to render a synthesized voice requesting that the assisted telephone call be transferred to the target state. Continuing with the example above, if it is determined that the speech input indicates that the automated assistant 115 has arrived at a state associated with the home and garden department (rather than the paint department), the verification engine 155 can cause the automated assistant 115, at an additional client device associated with the human agent, to render a synthesized voice saying, "Can you transfer me to the paint department?" and can again verify the state after the transfer, corresponding to the desired target state of the user of the client device 110.

[0042] The notification engine 156 can generate various notifications during the assisted telephone call based on navigating the IVR tree associated with the identified entity without interrupting the assisted telephone call and / or based on tasks performed by the automated assistant 115 and on behalf of the user of the client device 110. For example, upon reaching a target state, the automated assistant 115 can cause the notification engine 156 to generate a notification requesting that the user of the client device 110 join the assisted telephone call. In some implementations, the notification engine 156 can cause the automated assistant 115 to generate a notification requesting that the user of the client device 110 join the assisted telephone call only after the verification engine 155 verifies that the automated assistant has indeed reached the target state, as described above with respect to the verification engine 155. As another example, if the automated assistant 115 cannot resolve corresponding values ​​for parameters during the assisted telephone call, the automated assistant 115 can cause the notification engine 156 to generate one or more prompts to request corresponding values ​​for any unresolved parameters that are needed to continue navigating the IVR tree associated with the entity. As yet another example, when the automated assistant performs a task on behalf of the user of the client device 110 during an assisted telephone call, the automated assistant 115 can cause the notification engine 156 to generate a notification that includes the results of performing the task.

[0043] In some implementations, the type of notification generated by notification engine 157 and rendered at client device 110 via rendering engine 113, and / or one or more characteristics (e.g., volume, brightness, size) of the rendering notification, may be based on the state of client device 110 (e.g., determined using device state engine 112) and / or the state of an ongoing assisted phone call. The state of an ongoing assisted phone call may be based on, for example, whether a software application is running in the foreground of client device 110, whether a software application is running in the background of client device 110, whether client device 110 is in a locked state, whether client device 110 is in a sleep state, whether client device 110 is in an off state, sensor data generated by sensors of client device 110, and / or other data related to client device 110. For example, if the state of client device 110 indicates that a software application (e.g., an automated assistant application, a phone application, an assisted phone call application, and / or other software application) that displays a transcript of an assisted phone call is running in the foreground of client device 110, the type of notification may be a banner notification, a pop-up notification, and / or other type of visual notification. As another example, if the state of client device 110 indicates that client device 110 is asleep or locked, the type of notification may be an audible indication via a speaker and / or a vibration via a speaker or other hardware component of client device 110 that mimics a phone call.As yet another example, if sensor data from a presence sensor, accelerometer, and / or other sensors on the client device indicates that the user is not currently near the client device and / or is not currently holding the client device, a more intrusive notification (e.g., visual and audible at a first volume level) may be provided. On the other hand, if such sensor data indicates that the user is currently near the client device and / or currently holding the client device, a less intrusive notification (e.g., visual only, or visual and audible at a second volume level less than the first volume level) may be provided. As yet another example, when the discussion state indicates that the discussion is nearly complete, a more intrusive notification may be provided, but when the discussion state indicates that the discussion is not nearly complete, a less intrusive notification may be provided.

[0044] Moreover, in implementations in which automated assistant 115 causes a notification at client device 110 requesting that the user join an assisted telephone call, but the user does not join the assisted telephone call within a threshold duration (e.g., 15 seconds, 30 seconds, 60 seconds, and / or other duration), the automated assistant may terminate the assisted telephone call. In some versions of these implementations, the threshold duration may begin when a notification at client device 110 requesting that the user join an assisted telephone call is rendered. In some additional or alternative versions of these implementations, the state of the assisted telephone call may also be stored in one or more databases of client device 110 (e.g., random access memory of the client device) before terminating the assisted telephone call. If the user of client device 110 subsequently interacts with the notification to join the terminated assisted telephone call, automated assistant 115 may restart execution of the assisted telephone call by loading the stored state of the assisted telephone call, such that the assisted telephone call is resumed in a given state corresponding to the stored state of the assisted telephone call. In this way, the automated assistant 115 does not need to navigate portions of the IVR tree that have already been navigated by the automated assistant 115 .

[0045] In various implementations, and after the automated assistant 115 navigates to the target state of the IVR tree associated with the entity, the automated assistant can perform a task on behalf of the user of the client device 110. In some versions of those implementations, the task engine 157 can determine a task to be performed on behalf of the user of the client device 110 based on a user input detected at the client device 110 to initiate the assisted phone call and based on additional user input detected at the client device 110 before the automated assistant 115 initiates the assisted phone call. For example, if the user of the client device 110 provides a speech input of "call Example Cafe to make a reservation for tonight," the task engine 157 can use the intents of initiating a call and making a restaurant reservation (e.g., determined using the NLU model 130A) to determine the task of making a restaurant reservation based on the speech input. The parameters described herein that are determined by parameter engine 153 may also identify parameters for tasks that are identified by task engine 157, which may determine corresponding values ​​for the parameters for the tasks in a similar manner as described above with respect to parameter engine 153.

[0046] The task engine 157 can cause the automated assistant 115 to engage in a discussion with a human agent associated with the identified entity during an assisted telephone call to perform the task. For example, the task engine 157 can provide text and / or phonemes based on corresponding values ​​for parameters to the speech synthesis engine 140A1 and / or 140A2 to generate synthetic speech audio data. The synthetic speech audio data can be transmitted to an additional client device of the human agent for audible rendering on the additional client device. For example, the speech synthesis engine 140A1 and / or 140A2 can determine a sequence of phonemes determined to correspond to values ​​for parameters requested by the human agent and can process the sequence of phonemes using the speech synthesis model 140A to generate synthetic speech audio data. The synthetic speech audio data can be in the form of, for example, an audio waveform. In determining the sequence of phonemes corresponding to the value for the parameter, speech synthesis engine 140A1 and / or 140A2 can access token-to-phoneme mappings stored locally at client device 110 or stored at a server (e.g., via network 190). Task engine 157 can have speech recognition engine 120A1 and / or 120A2 process any audio data capturing speech input provided by the human agent, and automated assistant 115 can then use speech synthesis engine 140A1 and / or 140A2 to generate further synthetic speech audio data.

[0047] As described herein, the rendering engine 113 can render various notifications or other outputs at the client device 110. The rendering engine 113 can audibly and / or visually render the various notifications described herein. Additionally, the rendering engine 113 can cause a transcript of the conversation to be rendered on the user interface of the client device 110. In some implementations, the transcript can correspond to a conversation between a user of the client device 110 and the automated assistant 115. In some additional or alternative implementations, the transcript can correspond to a conversation between an IVR system associated with the identified entity and the automated assistant 115. In yet other implementations, the transcript can correspond to a conversation between a human agent associated with the identified entity and the automated assistant 115.

[0048] In some implementations, the user input detected at the client device 110 via the user input engine 111 may be a search query received via a search interface of the client device 110. The search interface may be implemented, for example, as part of a phone or contacts application, a browser application, an automated assistant application, a web-based browser, and / or any other interface that enables search functionality. The search engine 114 may identify and retrieve search results that are rendered at the client device 110 via the rendering engine 113. In some versions of these implementations, the search results may include content responsive to the search query and may optionally include one or more selectable elements. The one or more selectable elements may, for example, include deep links to one or more candidate states of the IVR tree that are related to the entity and that, when selected based on additional user input (e.g., touch, type, and / or speech), cause the automated assistant 115 to navigate to a target state of the IVR tree that corresponds to the selected candidate state of the IVR tree included in the search results. For example, if a user of client device 110 submits a search query for "home improvement store" via a search interface of a telephone application, selectable elements associated with various departments of an IVR tree associated with the home improvement store that correspond to one or more of the candidate states may be included in the search results responsive to the search query. Moreover, a particular selectable element (e.g., "paint department") may be rendered more prominently than other selectable elements if the user has recently interacted with other search results related to "paint" (e.g., as described above with respect to user interaction engine 154).

[0049] By using the techniques described herein, various technical advantages may be achieved. As one non-limiting example, since an IVR tree is stored in association with an entity, the automated assistant 115 may be able to more quickly complete an assisted telephone call because the automated assistant does not need to listen to the IVR system's options before navigating the IVR tree. Because the length of an assisted telephone call may be shortened by using the techniques disclosed herein, both network and computational resources may be conserved. Furthermore, the amount of user input may be reduced, thereby conserving client device resources because, in various implementations, a given user may only need to select an “IVR deep link” to begin execution of an assisted telephone call. Furthermore, battery resources of the client device 110 may be conserved in that the user is not an active participant in the assisted telephone call and the client device 110 may be placed in a locked or low-power state during the assisted telephone call.

[0050] Referring now to FIG. 3 , a flowchart illustrating an example method 300 for modifying navigation of an IVR tree based on user interactions at a client device associated with a given user is shown. For convenience, operations of method 300 are described with reference to a system that performs the operations. This system of method 300 includes one or more processors and / or other components of a computing device (e.g., client device 110 of FIGS. 1 , 4A-4D , 6A-6E , and / or 8A-8D , and / or computing device 910 of FIG. 9 , one or more servers, and / or other computing devices). Moreover, while the operations of method 300 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0051] In block 352, the system receives user input to initiate an assisted telephone call from a given user and via a client device associated with the given user. The user input to initiate an assisted telephone call may be one or more of a spoken input to initiate an assisted telephone call, a touch input to initiate an assisted telephone call, or a typed input to initiate an assisted telephone call. Furthermore, the user input to initiate an assisted telephone call may be received in a variety of interfaces (e.g., a search interface, a voice interface, an automated assistant interface, and / or other interface from which an assisted telephone call may be initiated).

[0052] In block 354, the system identifies an entity to be engaged on behalf of a given user during the assisted phone call based on the user input. The system may identify an entity to be engaged during the assisted phone call based on the user input received in block 352 and / or the user's interaction with the client device immediately prior to the initiation of the assisted phone call. For example, the system may identify the entity "exemplary airline" based on processing the speech input "I want to report my lost baggage to an exemplary airline" received at the client device. As another example, the system may identify the entity "exemplary airline" based on determining that a given user of the client device selected a call interface element associated with an exemplary airline or a particular graphical element associated with a candidate state of an IVR tree associated with an exemplary airline.

[0053] In block 356, the system identifies an IVR tree associated with the identified entity, the IVR tree including a plurality of candidate states. The IVR tree associated with the identified entity may be stored in association with the entity identified in block 354 (e.g., in entity database 151A of FIG. 1 and / or in IVR tree database 152A of FIG. 1). Based on the system (and / or additional instances of the system) previously crawling the IVR system associated with the identified entity, the stored IVR tree associated with the entity may have been previously stored in one or more databases, and / or the identified entity may provide the IVR tree for the IVR system associated with the identified entity. Furthermore, the system may continuously update the IVR tree for the IVR system associated with the identified entity based on multiple assisted telephone calls that navigate the IVR tree. The plurality of candidate states may correspond to specific nodes in the IVR tree to which the system can navigate during the assisted telephone call.

[0054] In block 358, the system determines a target state of the IVR tree from among the plurality of candidate states based on the user input or additional user input. In some implementations, the target state may be determined based on the user input received in block 352. For example, assume that the IVR tree is associated with an exemplary airline entity and that a given candidate state of the IVR corresponds to the “report lost baggage” candidate state for the IVR tree associated with the exemplary airline entity. Further assume that a speech input of “report my lost baggage to the exemplary airline” is received at the client device in block 352. In this example, the target state of “report lost baggage” may be identified based on the speech input. In some additional or alternative implementations, the target state may be determined based on additional user input in addition to the user input received in block 352. For example, assume that the IVR tree is associated with an exemplary airline entity and that a given candidate state of the IVR corresponds to the “report lost baggage” candidate state for the IVR tree associated with the exemplary airline entity. It is further assumed that a speech input of "call an exemplary airline" is received at the client device at block 352, or a selection of a selectable graphical element related to calling an exemplary airline is received at the client device at block 352. In this example, one or more of a plurality of candidate states may be presented as output at the client device in response to receiving the user input, and the target state may be based on an additional user input that selects the target state of "report lost baggage" from among one or more of the plurality of presented candidate states. The output may be presented prior to initiating the assisted phone call or upon initiating the assisted phone call.

[0055] In block 360, the system initiates execution of an assisted telephone call to navigate the IVR tree to the target state. The system can initiate execution of the assisted telephone call by establishing a communication session between the client device associated with the given user and the additional client device associated with the entity identified in block 354. The communication session can be established using various voice communication protocols, including, for example, VoIP, PSTN, and / or other telephony communication protocols. In some implementations, the system can navigate the IVR tree to the target state by emulating button presses to traverse nodes of the IVR tree, by rendering synthesized voice audio data including synthesized voice to traverse nodes of the IVR tree, and / or by rendering audio data including speech input of a given user associated with the client device (e.g., as described with respect to FIG. 2). In some versions of these implementations, the system can also utilize corresponding values ​​for parameters related to navigating the IVR tree. The corresponding values ​​for the parameters can be retrieved from one or more databases (e.g., user profile database 153A of FIG. 1) and / or can be requested from a given user associated with the client device.

[0056] In block 362, the system determines whether a user interaction is detected at the client device. The user interaction may include, for example, a search interaction with a web browser or web browser application, a browsing interaction with a web browser or web browser application, a navigation interaction with a web browser or navigation application, an assistant interaction with automated assistant 115, and / or other user interaction with the client device. In particular, while the system performs the assisted telephone call, a given user associated with the client device may still be interacting with the client device such that the assisted telephone call runs in the background of the client device. If, in an iteration of block 362, the system determines that no user interaction is detected at the client device, the system may proceed to block 370. Block 370 is described in more detail below. If, in an iteration of block 362, the system determines that a user interaction is detected at the client device, the system may proceed to block 364.

[0057] In block 364, the system determines whether the user interaction detected in block 362 was related to the entity identified in block 354. For example, the system may determine whether the user interaction was directed to a web page related to the entity identified in block 354, whether the user interaction was directed to obtaining directions to a brick-and-mortar store related to the entity identified in block 354, and / or whether other user interactions are directed to the entity. If, in an iteration of block 364, the system determines that the user interaction detected in block 362 was not related to the entity identified in block 354, the system may proceed to block 370. Block 370 is described in more detail below. If, in an iteration of block 364, the system determines that the user interaction detected in block 362 was related to the entity identified in block 354, the system may proceed to block 366.

[0058] In block 366, the system generates one or more prompts related to modifying the navigation of the IVR tree based on the user interaction detected in block 362. In some implementations, modifying the navigation of the IVR tree can include modifying the target state to navigate the IVR tree to an alternate target state. For example, assume that an assisted phone call is initiated to navigate to a target state related to a flooring department in an IVR tree associated with a home improvement entity, and further assume that a user interaction targeted to a portion of a web page related to the lighting department of the home improvement entity is detected. In this example, the system can prompt a given user to change the target state from related to the flooring department to an alternate target state related to the lighting department.

[0059] In some additional or alternative implementations, modifying the navigation of the IVR tree can include adding an additional target state for navigating the IVR tree. Continuing with the above example, rather than prompting a given user to change the target state from one related to the flooring department to an alternative target state related to the lighting department, the system can prompt the given user to add the lighting department as an additional target state in addition to the target state related to the flooring department. In this example, upon arriving at the target state related to the flooring department, the user can be notified to join the assistance phone call. After the given user interacts with the flooring department, the given user can end the assistance phone call. However, the system can leave the assistance phone call connected and request that a human agent related to the flooring department transfer the assistance phone call to the lighting department, or the system can re-navigate the IVR tree to the additional target state related to the lighting department.

[0060] In some additional or alternative implementations, modifying the navigation of the IVR tree can include modifying corresponding values ​​for parameters for navigating the IVR tree. For example, assume that an assisted phone call is initiated to navigate to a target state associated with a lighting department of an IVR tree associated with a home improvement entity, and further assume that a user interaction targeted to a portion of a web page associated with a particular brand of smart light bulbs sold at the home improvement entity is detected. In this example, the system can prompt a given user to include corresponding values ​​associated with the particular brand of smart light bulbs for parameters associated with navigating to the target state associated with the lighting department. Additionally, the system can cause one or more prompts to be rendered visually and / or audibly at a client device associated with the given user.

[0061] In block 368, the system determines whether to modify the navigation of the IVR tree based on additional user input received in response to one or more of the prompts. For example, if the user responds affirmatively to one or more of the prompts rendered at the client device, the navigation of the IVR tree may be modified based on the user interaction detected in block 362. However, if the user does not respond affirmatively (or responds negatively) to one or more of the prompts, the navigation of the IVR tree may not be modified. If, in a repetition of block 368, the system determines based on additional user input that the navigation of the IVR tree should not be modified, the system may proceed to block 370.

[0062] In block 370, the system navigates the IVR tree to the target state. In implementations in which the system proceeds to block 370 from blocks 362, 364, and / or 368, the system may continue navigating the IVR tree based on the target state determined in block 358. The system may navigate the IVR tree based on emulating button presses to navigate the IVR tree, causing additional client devices associated with the entity identified in block 354 to render synthetic speech audio data including synthetic speech, and / or causing additional client devices associated with the entity identified in block 354 to render audio data including speech input. If, in a repetition of block 368, the system determines that the navigation of the IVR tree should be modified based on the additional user input, the system may proceed to block 372. In block 372, the system modifies corresponding values ​​for parameters for navigating the IVR tree or modifies the target state to additionally or alternatively include the additional target state, as described above with respect to block 366. Navigating the IVR tree is described above with respect to FIG.

[0063] From block 370 or block 372, the system may proceed to block 374. In block 374, the system may verify whether the assisted telephone call has reached the target state. The system may process speech input received at the client device from a human agent associated with the entity subscribing to the call or an additional automated assistant in an IVR system associated with the entity to determine whether the system has reached the target state. For example, assume that the target state of the IVR tree is associated with the lighting department of a home improvement store entity. Further assume that the human agent or additional automated assistant provides the speech input, "Thank you for calling the lighting department, how may I help?" In this example, to verify that the system has indeed reached the desired target state for a given user associated with the client device, the system may determine that the speech input includes the term "lighting department." In some implementations, the speech input received at the client device may be provided in response to the system soliciting speech input via a synthetic voice rendered at an additional client device associated with a human agent or additional automated assistant (e.g., "Hello, is this the lighting department," or other synthetic voice requesting verification of the target state). If, in the iteration of block 374, the system determines that the assisted phone call is arriving in another state other than the target state, the system may proceed to block 376. In block 376, the system requests that the assisted phone call be transferred to an agent associated with the target state.The request to transfer the assistive phone call may be included in a synthesized voice rendered at an additional client device associated with the human agent or additional automated assistant (e.g., "Can you transfer me to the lighting department," or other synthesized voice requesting that the assistive phone call be transferred). The system may then return to block 374 to verify whether the agent associated with the target state corresponds to the target state after the transfer. If, in the iteration of block 374, the system determines that the assistive phone call has arrived at the target state, the system may proceed to block 378.

[0064] In block 378, the system renders a notification via the client device indicating the results of the execution of navigating the IVR tree. For example, the notification may include a request that a given user associated with the client device join an assisted telephone call. In implementations described herein in which the system also performs a task on behalf of a given user and entity associated with the client device, the notification may additionally or alternatively include the results of the execution of the task.

[0065] 4A-4C, various non-limiting examples of user interfaces associated with modifying IVR tree navigation based on user interactions at a client device 110 associated with a given user are shown. Each of FIGS. 4A-4C illustrates a client device 110 having a graphical user interface 180, which may include one or more of the components of the client device of FIG. 1. One or more aspects of an automated assistant associated with the client device 110 (e.g., automated assistant 115 of FIG. 1) may be implemented locally on the client device 110 and / or on other client devices in network communication with the client device 110 in a distributed manner (e.g., via network 190 of FIG. 1). For simplicity, the operations of FIGS. 4A-4C are described herein as being performed by an automated assistant. While the client device 110 in FIGS. 4A-4C is shown as a mobile phone, it should be understood that this is not intended to be limiting. Client device 110 may be, for example, a standalone assistant device (e.g., having a speaker and / or a display), a laptop, a desktop computer, a vehicle computing device, and / or any other client device capable of making telephone calls.

[0066] 4A-4C further includes a text response interface element 184 that a user may select to generate user input via a virtual keyboard or other touch and / or typed input, as well as a voice response interface element 185 that a user may select to generate user input via a microphone of client device 110. In some implementations, a user may generate user input via the microphone without selecting a voice response interface element 185. For example, active monitoring for audible user input via the microphone may be performed to eliminate the need for a user to select voice response interface element 185. In some of these and / or other implementations, voice response interface element 185 may be omitted. Moreover, in some implementations, text response interface element 184 may additionally and / or alternatively be omitted (e.g., a user may provide only audible user input). The graphical user interface 180 of FIGS. 4A-4C also includes system interface elements 181, 182, 183 that may be interacted with by a user to cause the computing device 110 to perform one or more actions.

[0067] In various implementations described herein, user input can be received to initiate a telephone call (e.g., an assisted telephone call) with an entity using the automated assistant. The user input can be speech input, touch input, and / or typed input, including an indication for initiating the assisted telephone call. Furthermore, the automated assistant can navigate an IVR tree associated with the entity to a target state from among multiple candidate states and / or perform a task on the entity on behalf of a given user of client device 110. As shown in FIG. 4A , user interface 180 includes search results for a grocery store entity from a browser application accessible at client device 110 (e.g., as indicated by URL 411, “www.exampleurl0.com / ”). Furthermore, the search results include a first search result 420 for a “Hypothetical Grocer” located at a first location and a second search result 430 for a “Hypothetical Grocer” located at a second location.

[0068] In some implementations, search results 420 and / or 430 may be associated with various selectable graphical elements that, when selected, cause client device 110 to perform a corresponding action. For example, when call graphical element 421 and / or 431 associated with a given one of search results 420 and / or 430 is selected, user input may indicate that a phone call action to the grocery store entity associated with search result 420 and / or 430 should be performed. As another example, when directions graphical element 422 and / or 432 associated with a given one of search results 420 and / or 430 is selected, user input may indicate that a navigation action to the grocery store entity associated with search result 420 and / or 430 should be performed. As yet another example, when departments graphical element 423 and / or 433 associated with a given one of search results 420 and / or 430 is selected, user input may indicate that a browser-based action to display the departments for the grocery store entity associated with search result 420 and / or 430 should be performed. 4A , it should be understood that while the assisted phone call is initiated from a browser application in FIG. 4A , this is by way of example and not intended to be limiting. For example, the assisted phone call may be initiated from various software applications accessible at client device 110 (e.g., an automated assistant application, a phone application, a contacts application, an email application, a text or SMS messaging application, and / or other software applications), from the home screen of client device 110, from the lock screen of client device 110, and / or from other states of client device 110 if the assisted phone call is initiated using speech input.

[0069] For the sake of example, assume that user input is detected at client device 110 to initiate a phone call with first search result 420 of “Virtual Grocer” at a first location. The user input may be, for example, a spoken input of “call Hypothetical Grocer” (and selected based on the proximity of the first location to client device 110) or may be a touch input targeted at call graphical element 421, as indicated by 490A1. In some implementations, call details interface 470 may be rendered at client device 110 prior to receiving user input to initiate the phone call with “Virtual Grocer,” or may be rendered at client device 110 in response to receiving user input to initiate the phone call with “Virtual Grocer,” as indicated by 490A1. In some versions of those implementations, call details interface 470 may be rendered at client device 110 as part of user interface 180. In some other versions of those implementations, call details interface 470 may be a separate interface from user interface 180 that overlays user interface 180 and may include a call details interface element 186 that allows a user to expand call details interface 470 to display additional call details (e.g., by swiping up on the call details interface element 186) and / or dismiss call details interface 470 (e.g., by swiping down on the call details interface element 186). While call details interface 470 is shown as being at the bottom of user interface 180, it should be understood that this is for purposes of example and not intended to be limiting. For example, call details interface 470 may be rendered at the top of user interface 180, on the side of user interface 180, or in an interface that is entirely separate from user interface 180.

[0070] In various implementations, the call details interface 470 may include multiple graphical elements. In some versions of those implementations, the graphical elements may be selectable, such that when a given one of the graphical elements is selected, the client device 110 may perform a corresponding action. As shown in FIG. 4A , the call details interface 470 includes a first graphical element 471A, "Assisted Call, Hypothetical Grocer, First Location," a second graphical element 472A, "Regular Call," and a third graphical element 473A, "Save Contact 'Hypothetical Grocer'." Notably, in some versions of those implementations, the graphical elements may include subelements to provide a display of candidate states for navigating an IVR tree related to the "virtual grocery store." For example, the first graphical element 471A may include a first sub-element 471A1 called "Produce Department" that is associated with the target state of navigating an IVR tree associated with the virtual grocery store to a portion corresponding to the produce department, a second sub-element 471A2 called "Bakery Department" that is associated with the target state of navigating an IVR tree associated with the virtual grocery store to a portion corresponding to the bakery department, and a third sub-element 471A3 called "Deli Department" that is associated with the target state of navigating an IVR tree associated with the virtual grocery store to a portion corresponding to the prepared foods department.

[0071] For purposes of example, assume that user input is detected at client device 110 to initiate execution of an assisted phone call with a virtual grocery store to navigate an IVR tree associated with the virtual grocery store to a target state associated with the produce department of the virtual grocery store. The user input may be, for example, a spoken input of "call the produce department of Hypothetical Grocer" or a touch input targeted to first sub-element 471A1, as indicated by 490A2. In response to detecting the user input, the automated assistant can determine a target state of "produce department" to navigate the IVR tree associated with the virtual grocery store. In some implementations, the automated assistant can identify corresponding values ​​for parameters associated with navigating the IVR tree to the target state and / or corresponding values ​​for tasks to be performed upon arriving at the target state associated with the produce department, as described herein (e.g., with respect to parameter engine 153 of FIG. 1 and with respect to FIG. 2).

[0072] In some implementations, and as shown in FIG. 4B , while an automated assistant navigates an IVR tree to a target state during an assisted telephone call, a given user of client device 110 may still be able to interact with client device 110. In some versions of those implementations, user interactions detected at client device 110 while the automated assistant navigates the IVR tree may be utilized to modify the navigation of the IVR tree. For example, and as shown in FIG. 4B , assume that a given user associated with client device 110 views a web page related to a first search result 420 for "virtual grocery store" in a browser application accessible at client device 110 (e.g., as indicated by URL 411, "www.exampleurl1.com / "). In particular, the web page may include selectable graphical elements corresponding to information related to various departments of the virtual grocery store. For example, a given user associated with client device 110 may select a first graphical selectable element 471B1 to view produce availability in the produce section of the virtual grocery store, may select a second graphical selectable element 471B2 to view cake availability in the bakery section of the virtual grocery store, or may select a third graphical selectable element 471B3 to view meat availability in the prepared foods section of the virtual grocery store, or any other graphical selectable element that may be presented.

[0073] For example, assume that while the automated assistant is navigating an IVR tree associated with a virtual grocery store to a target state associated with the produce department, the user selects third graphical selectable element 471B3 to view meat availability in the prepared foods section of the virtual grocery store, as indicated by 490B1. While the automated assistant is navigating an IVR tree associated with the virtual grocery store to a target state not associated with the meat department, in response to determining that a given user associated with client device 110 has interacted with a search result related to the prepared foods section of the virtual grocery store, the automated assistant can generate one or more prompts and can render one or more of the prompts as notifications in user interface 180 of client device 110. For example, the automated assistant can cause notification 479 to be rendered visually in call details interface 470. Notification 479 includes an indication that "Would you rather connect with the deli department during the ongoing assisted telephone call," and also includes a first suggestion 479B1 of "yes" and a second suggestion 479B2 of "no." As shown in FIG. 4B , and in response to the rendering of notification 479, it is further assumed that additional user input is received at client device 110 indicating that a given user of client device 110 will actually connect with the prepared foods department rather than the produce department as originally intended (e.g., as indicated by 490B2 as a result of selecting first suggestion 479B1 of "yes"). In this example, navigation of an IVR tree associated with the virtual grocery store may be modified based on user interaction with a web page at client device 110.For example, rather than navigating the IVR to a target state associated with the produce department as originally intended by a given user associated with client device 110, the automated assistant may instead navigate to an alternate target state associated with the prepared foods department.

[0074] In various implementations, the automated assistant may prompt a given user of client device 110 to modify the navigation of the IVR tree only if the user interaction is related to an entity involved during the assisted phone call. For example, a given user of client device 110 may be prompted to modify the navigation of the IVR tree based only on user interactions related to a virtual grocery store. In some additional or alternative implementations, the automated assistant may still prompt a given user of client device 110 to modify the navigation of the IVR tree even if the user interaction is not related to an entity involved during the assisted phone call. For example, a given user of client device 110 may be prompted to modify the navigation of the IVR tree based on user interactions related to a web page for a particular brand of deli meat that is different from the virtual grocery store. In this example, one or more prompts may also include a coupon or advertisement for the particular brand of deli meat.

[0075] In some implementations, modifying the navigation of the IVR system can be based on the state of navigating the IVR tree. For example, if the automated assistant has not arrived at the target state when additional user input is received in response to one or more prompts for modifying the navigation of the IVR tree, the automated assistant can modify corresponding values ​​for parameters utilized in navigating the IVR tree to navigate to the target state, modify internal nodes traversed in navigating the IVR tree (e.g., as described with respect to FIG. 2) to navigate to an alternate target state, and / or restart navigation of the IVR tree. As another example, if the automated assistant has arrived at the target state when additional user input is received in response to one or more prompts for modifying the navigation of the IVR tree, the automated assistant can request that the assisted telephone call be transferred to an alternate target state determined based on the additional user input and / or restart navigation of the IVR tree.

[0076] For example, and referring in detail to FIG. 4C , assume that the automated assistant has navigated the IVR tree to a target state as indicated by spoken utterance 452C1 of “Now connecting you with the produce department” from the virtual grocery store IVR and a subsequent spoken utterance 452C2 of “Hello, produce department” from the virtual grocery store produce agent. Further assume that the automated assistant arrived at the target state before receiving user input, indicated by 490B1 in FIG. 4B , to modify the IVR navigation to an alternate target state associated with the prepared foods department. Generally, at this stage of navigating the IVR tree associated with the virtual grocery store, the automated assistant may verify that the target state has been reached (e.g., based on the subsequent spoken utterance 452C2), generate a notification requesting that the user subscribe to an assisted call, and / or perform a task on behalf of a given user of client device 110. However, because a given user has provided user input to modify the navigation of the IVR tree in FIG. 4B, the automated assistant may invoke synthesized voice 454C1 saying, "Hello, can you transfer me to the deli department," and because the virtual grocery store produce agent places the automated assistant on hold to transfer the call to the prepared foods department, spoken utterance 452C3 "One moment please" may be received in response to synthesized voice 452C3, and the automated assistant may remain on hold until spoken utterance 452C4 "Hello, deli department" is received.Thus, at this stage of navigating the IVR tree associated with the virtual grocery store, the automated assistant may verify that a target state has been reached (e.g., based on verbal utterance 452C4), may generate a notification (e.g., as indicated by 454C2) requesting that the user subscribe to an assisted call, and / or may perform a task on behalf of a given user of client device 110.

[0077] Moreover, in various implementations, the automated assistant may cause transcripts of various discussions to be visually rendered in the user interface 180 of the client device 110 (e.g., as shown in FIG. 4C ). The transcripts may be displayed, for example, on the home screen of the client device 110, in various software applications (e.g., an automated assistant application, a call application, and / or other applications). In some implementations, the transcripts may include discussions between the automated assistant, an IVR system, and / or a human agent associated with the entity. Furthermore, in various implementations, the automated assistant may also cause various graphical elements to be presented in the call details interface 470. For example, a first selectable graphical element 442C associated with ending an assisted telephone call may be provided, a second selectable graphical element 443C associated with ending a given user's participation in an assisted telephone call may be provided, a third selectable graphical element 444C associated with the assisted telephone call being audibly perceptible via a speaker of the client device 110 may be provided, and / or other selectable graphical elements associated with the assisted telephone call may be provided. In various implementations, these selectable graphical elements may be rendered in the call details interface 470 only in response to the generation of a notification such as that indicated by 452C2.

[0078] 4A-4C are described herein with respect to modifying the navigation of an IVR tree associated with a virtual grocery store from an original target state associated with a produce department to an alternative target state associated with a prepared foods department, it should be understood that this is by way of example and not intended to be limiting. For example, and as described above (e.g., with respect to user interaction engine 154 of FIG. 1 , with respect to IVR tree 200 of FIG. 2 , and with respect to method 300 of FIG. 3 ), modifying the navigation of the IVR tree may include adding a prepared foods department as an additional target state that may be navigated to after navigating to the original target state associated with the produce department. As another example, and also as described above (e.g., with respect to user interaction engine 154 of FIG. 1 , with respect to IVR tree 200 of FIG. 2 , and with respect to method 300 of FIG. 3 ), modifying the navigation of the IVR tree may include modifying corresponding values ​​for parameters for navigating the IVR tree to the original target state associated with the produce department. For example, if, when navigating to a target state related to the produce department, a given user indicates that the automated assistant should query about avocados, but begins to view tomatoes on the produce section of a web page related to a virtual grocery store, the user may be prompted as to whether the automated assistant should additionally or alternatively query about tomatoes.

[0079] 4C is illustrated as including a transcript of an automated assistant navigating an IVR tree associated with a virtual grocery store, it should be understood that this is by way of example and not intended to be limiting. For example, it should be understood that the assisted phone call described above can be performed while client device 110 is asleep, locked, when other software applications are running in the foreground, or in other states. Furthermore, in implementations in which the automated assistant causes notifications to be rendered at client device 110, the type of notification rendered at the client device is based on the state of client device 110 as described herein. Furthermore, it should be understood that while FIGS. 4A-4C are described herein with respect to navigating an IVR tree associated with a virtual grocery store, this is also not intended to be limiting, and that the techniques described herein can be utilized in navigating any previously stored IVR tree and with respect to multiple different entities.

[0080] Referring now to FIG. 5, a flowchart illustrating an example method 500 for navigating an IVR tree from a search interface on a client device associated with a given user is shown. For convenience, operations of method 500 are described with reference to a system that performs the operations. This system of method 500 includes one or more processors and / or other components of a computing device (e.g., client device 110 of FIGS. 1, 4A-4D, 6A-6E, and / or 8A-8D, and / or computing device 910 of FIG. 9, one or more servers, and / or other computing devices). Moreover, while the operations of method 500 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0081] At block 552, the system receives user input from a given user and via a search interface of a client device associated with the given user. The user input to initiate an assisted phone call may be one or more of a speech input to initiate an assisted phone call, a touch input to initiate an assisted phone call, or a typed input to initiate an assisted phone call. Furthermore, the user input to initiate an assisted phone call may be received in a variety of interfaces (e.g., a search interface, a voice interface, an automated assistant interface, and / or other interface from which an assisted phone call may be initiated). Furthermore, the search interface may be implemented, for example, as part of a phone or contacts application, a browser application, an automated assistant application, a web-based browser, and / or any other interface that enables search functionality.

[0082] At block 554, the system issues a search query based on the user input in a search interface of the client device. The search query may be issued to one or more databases accessible by the client device (e.g., restricted-access databases and / or public databases), search engines, and / or any other search-based system.

[0083] In block 556, the system identifies a given search result related to an entity to be engaged on behalf of the given user during the assisted telephone call based on the issued search query, the given search result related to a target state from among multiple candidate states for an IVR tree related to the identified entity. In block 558, the system causes at least the given search result to be rendered at the client device. In block 560, the system receives a selection of the given search result from the given user via a search interface of the client device. The given search result may include content responsive to the search query issued in block 554. In some implementations, additional given search results corresponding to other candidate states may also be rendered along with the given search result. In some versions of these implementations, the given search result may be presented more prominently than additional given search results related to other candidate states based on recent user interactions at the client device. For example, if a user is interacting with other search results related to various brands of paint or painting equipment and the issued search query is for a home improvement entity, the given search result may be related to the home improvement entity, or more specifically, to the painting department of the home improvement entity. Thus, upon selection of the given search result, the system may determine the painting department as being the target state for navigating the IVR tree related to the home improvement entity. Selection of the given search result may be based on touch input, speech input, and / or typed input.

[0084] In block 562, the system, in response to the selection, initiates execution of an assisted telephone call to navigate the IVR tree to the target state. The system can initiate execution of the assisted telephone call by establishing a communication session between the client device associated with the given user and the additional client device associated with the entity identified in block 556. The communication session can be established using various voice communication protocols, including, for example, VoIP, PSTN, and / or other telephony communication protocols. In some implementations, the system can navigate the IVR tree to the target state by emulating button presses to traverse nodes of the IVR tree, by rendering synthesized speech audio data including synthesized speech to traverse nodes of the IVR tree, and / or by rendering audio data including speech input of the given user associated with the client device (e.g., as described with respect to FIG. 2). In some versions of these implementations, the system can also utilize corresponding values ​​for parameters related to navigating the IVR tree. Corresponding values ​​for the parameters may be retrieved from one or more databases (eg, user profile database 153A of FIG. 1) and / or may be requested from a given user associated with the client device.

[0085] In block 564, the system renders a notification via the client device indicating the results of the execution of navigating the IVR tree. For example, the notification may include a request that a given user associated with the client device join an assisted telephone call. In implementations described herein in which the system also performs a task on behalf of a given user associated with the client device and for an entity, the notification may additionally or alternatively include the results of the execution of the task. In various implementations, the navigation of the IVR tree described with respect to FIG. 5 may be modified based on user interactions at the client device, as described above with respect to FIGS. 3 and 4A-4C.

[0086] 6A-6E, various non-limiting examples of user interfaces associated with navigating an IVR tree from a search interface on a client device 110 associated with a given user are shown. Each of Figures 6A-6E illustrates a client device 110 having a graphical user interface 180, which may include one or more of the components of the client device of Figure 1. The client device 110 may be substantially similar to the client device of Figures 4A-4C and may include many of the same components described above with respect to Figures 4A-4C, including an instance of an automated assistant (e.g., automated assistant 115 of Figure 1).

[0087] In various implementations described herein, user input may be received and a search query may be issued based on the user input. The user input may be spoken, touch, and / or typed input including the search query. For example, and as shown in FIG. 6A , assume that a user provides a search query 684A for “lost baggage virtual airlines” (e.g., as included in text response interface element 184) in a browser application accessible at client device 110 (e.g., as indicated by URL 611 of “www.exampleurl0.com / ”). Further, the search results include a first search result 620 for “virtual airlines” offering flights serving North America. In some implementations, search result 620 may be associated with various selectable graphical elements that, when selected, cause client device 110 to perform a corresponding action. For example, when call graphical element 621 associated with search result 620 is selected, the user input may indicate that a phone call action to virtual airlines should be performed. As another example, when a user account graphical element 622 associated with a search result 620 is selected, the user input may indicate that an account lookup action should be performed for a user account associated with the virtual airline. As yet another example, when a flight graphical element 623 associated with a search result 620 is selected, the user input may indicate that a flight lookup action should be performed for a flight associated with the virtual airline.

[0088] In various implementations, the search result 620 may also include one or more IVR deep links to candidate states of an IVR tree associated with the entity. For example, as shown in FIG. 6A , a first IVR deep link 620A1, “report last baggage,” may be provided when navigating an IVR tree associated with a virtual airline, corresponding to a candidate state for reporting lost baggage, and a second IVR deep link 620A2, “speak with agent,” may be provided when navigating an IVR tree associated with the virtual airline, corresponding to a candidate state for speaking with a virtual airline agent. An additional “see more” graphical element 620A3 may optionally be provided and, when selected, may cause one or more additional IVR deep links associated with the candidate states to be rendered along with the first and second deep links 620A1 and 620A2. In some versions of these implementations, one or more of the rendered IVR deep links 620A1 and / or 620A2 may be based on recent user interactions at the client device 110. For example, a first IVR deep link 620A1, "Report Lost Baggage," may be rendered at the client device 110 based on the search query 684A including "lost baggage" or based on a given user of the client device 110's previous interactions with other search results related to reporting lost baggage. As another example, a second IVR deep link 620A2, "Speak with an Agent," may be rendered at the client device 110 based on it being the most popular target state from among the candidate states. In these examples, the first IVR deep link 620A1 may be rendered more prominently than the second IVR deep link 620A2 based on determining it to be more important to the given user of the client device 110 (e.g., determined based on the search query 684A and / or based on a given user of the client device 110's previous interactions with other search results).

[0089] An assistive phone call to navigate an IVR tree associated with the virtual airline to a target state associated with reporting lost baggage can be initiated based on user input targeted at the first IVR deep link 620A1, and execution of the assistive phone call to navigate the IVR tree to the target state can be initiated. In some implementations, the automated assistant can utilize resources of the client device 110 to initiate execution of the assistive phone call. In some additional or alternative implementations, the automated assistant can utilize resources of a remote system to initiate execution of the assisted phone call (e.g., a cloud-based assisted phone call). For example, assume that a user selects the first IVR deep link 620A1, but the client device 110 is not capable of executing phone calls using VoIP, PSTN, or other telephony protocols (e.g., via a laptop or desktop computer). In this example, the automated assistant 115 can have a remote system (which may or may not be in communication with the client device 110) initiate and execute the assisted phone call on behalf of the user.

[0090] In various implementations, call details interface 670 may be rendered at client device 110 before receiving user input to initiate a phone call with the “virtual airline,” or may be rendered at client device 110 in response to receiving user input to initiate a phone call with the virtual airline. In some versions of those implementations, call details interface 670 may be rendered at client device 110 as part of user interface 180. In some other versions of those implementations, call details interface 670 may be a separate interface that overlays user interface 180 and may include a call details interface element 186 that allows a user to expand call details interface 670 to display additional call details (e.g., by swiping up on call details interface element 186) and / or dismiss call details interface 670 (e.g., by swiping down on call details interface element 186). It should be understood that while call details interface 670 is shown as at the bottom of user interface 180, this is by way of example and not intended to be limiting. For example, call details interface 670 may be rendered on top of user interface 180 , on the side of user interface 180 , or in an interface that is entirely separate from user interface 180 .

[0091] The call details interface 670, in various implementations, can include multiple graphical elements. In some versions of those implementations, the graphical elements can be selectable, such that when a given one of the graphical elements is selected, the client device 110 can perform a corresponding action. As shown in FIG. 6A , the call details interface 670 includes a first graphical element 671 titled "Assisted Call, Hypothetical Airline," a second graphical element 672A titled "Normal Call," and a third graphical element 673A titled "Save Contact 'Hypothetical Airline'." An assisted phone call can additionally or alternatively be initiated from the call details interface 670 based on user input targeted at the first selectable graphical element 671, and execution of the assisted phone call is based on selection of the first selectable graphical element 671.

[0092] In various implementations, one or more IVR deep links associated with a candidate state may not be rendered. In some versions of these implementations, the automated assistant may visually render various nodes (e.g., internal nodes and / or leaf nodes as described with respect to FIG. 2) for navigating an IVR tree associated with an entity. The visual rendering of the various nodes may be based on a confidence level for the IVR tree stored in association with the entity. The confidence level for the IVR tree stored in association with the entity may be based on the number of assisted phone calls with the entity that involve navigating an IVR tree different from the IVR tree stored in association with the entity, and optionally on the degree of difference between the navigated IVR tree and the stored IVR tree. In particular, FIGS. 6B-6E show various non-limiting examples of how these various nodes may be presented to a user based on differential confidence levels for IVR trees associated with a virtual airline.

[0093] 6B-6E, for example, assume that the entity to be engaged during the assisted phone call is the virtual airline described above with respect to FIG. 6A, and that the user is viewing a first search result 610 related to the virtual airline, but that no user input including search query 684A is received. Referring specifically to FIG. 6B, further assume that the automated assistant is highly confident in the IVR tree stored in association with the virtual airline. The automated assistant may be highly confident in the IVR tree stored in association with the virtual airline based on a confidence level determined based on the nodes of the IVR tree remaining the same for most (if not all) of the assisted phone calls involving the same IVR tree. In this example, rather than providing one or more IVR deep links, the automated assistant may invoke information including an indication of how to navigate the IVR tree associated with the virtual airline before receiving user input to begin executing the assisted phone call. For example, call details interface 670 may include first information 671A, "Press '1,' '3,' '5' to report lost baggage," indicating a sequence of button presses a given user of client device 110 can provide to reach a target state of an IVR tree associated with reporting lost baggage, and second information 671B, "Press '5' to speak with an agent," indicating a sequence of button presses a given user of client device 110 can provide to reach a target state of an IVR tree associated with speaking with an agent. A "More" graphical element 671C may optionally be provided and, when selected, may cause additional information associated with other candidate states to be rendered along with first information 671A and second information 671B. Similar to FIG. 6A , the information associated with the candidate states rendered in call details interface 670 may optionally be rendered based on recent user interactions with client device 110 and / or the overall popularity of the candidate states.

[0094] 6C in detail, and similar to FIG. 6B, further assume that the automated assistant is highly confident in the IVR tree stored in association with the virtual airline. In this example, rather than providing information related to navigating directly to a desired target state through one or more IVR deep links or a sequence of button presses before initiating an assisted phone call, the automated assistant may cause candidate states to be rendered in a notification 679C in the call details interface 670, as indicated by 652C1, prior to receiving a verbal utterance 654C1 from the virtual airline IVR containing the candidate states. In particular, the candidate states may include candidate states indicated by 679C1, 679C2, 679C3, 679C4, and 679C5. Each of the candidate states included in the notification 679C may be selectable, such that a given user associated with the client device 110 can simply select a given one of the candidate states as the target state, and the automated assistant automatically navigates the assisted phone call to the target state.

[0095] 6D in detail, and in contrast to FIG. 6C, further assume that the automated assistant is moderately confident in the stored IVR tree associated with the virtual airline. The automated assistant may be moderately confident in the stored IVR tree associated with the virtual airline based on a confidence level determined based on the inclusion of some variances in the nodes of the IVR tree for some assisted telephone calls involving the same IVR tree. In this example, rather than providing notification 679D in call detail interface 670 before receiving a verbal utterance from the virtual airline IVR that includes the candidate states, the automated assistant may wait to render notification 679C that includes the candidate states until a first portion of a verbal response 652D1 that includes a subset of the candidate states is provided by the virtual airline IVR. For example, the degree of variance between the stored IVR trees associated with the virtual airline may be such that the automated assistant is only moderately confident in a subset of the candidate states included in the first portion of verbal utterance 652D1, but is highly confident in the remaining candidate states. Thus, after the subset of candidate states is included in the first portion of spoken utterance 652D1, the automated assistant may generate notification 679D, as indicated by 654D1, before receiving the second portion of spoken utterance 652D2. Similar to FIG. 6C , each of the candidate states included in notification 679D may be selectable, such that a given user associated with client device 110 can simply select a given one of the candidate states as the target state, and the automated assistant will automatically navigate the assisted phone call to the target state.

[0096] Referring specifically to FIG. 6E , and in contrast to FIGS. 6C and 6D , assume further that the automated assistant is unsure of the IVR tree stored in association with the virtual airline. The automated assistant may be unsure of the IVR tree stored in association with the virtual airline based on a confidence level determined based on the inclusion of differences in nodes of the IVR tree for a threshold number of assisted phone calls involving different IVR trees. In this example, rather than providing notification 679E in call details interface 670 before receiving a verbal utterance, the automated assistant may wait until a spoken response 652E1 including the candidate states is provided by the virtual airline IVR before rendering notification 679D including the candidate states, as indicated by 654E1. As with FIG. 6C , each of the candidate states included in notification 679E may be selectable, such that a given user associated with client device 110 can simply select a given one of the candidate states as a target state, and the automated assistant automatically navigates the assisted phone call to the target state.

[0097] 6A-6E are described herein with respect to the entity virtual airline, and while some graphical elements and information are shown along with the search results 620 and call detail interface, it should be understood that this is by way of example and not intended to be limiting. It should be understood that an IVR tree associated with any entity that employs an IVR system can be stored and subsequently utilized in conducting an assisted telephone call. It should further be understood that the graphical elements and information provided by the automated assistant can be based on an IVR tree, and that the configurations of IVR trees associated with an entity are virtually infinite.

[0098] Referring now to FIG. 7 , a flowchart illustrating an example method 700 for navigating a dynamic interactive voice response (IVR) tree is shown. For convenience, the operations of method 700 are described with reference to a system that performs the operations. This system of method 700 includes one or more processors and / or other components of a computing device (e.g., client device 110 of FIGS. 1 , 4A-4D , 6A-6E , and / or 8A-8D , and / or computing device 910 of FIG. 9 , one or more servers, and / or other computing devices). Moreover, while the operations of method 700 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0099] At block 752, the system receives user input to initiate an assisted telephone call from a given user and via a client device associated with the given user. The user input to initiate an assisted telephone call may be one or more of a spoken input to initiate an assisted telephone call, a touch input to initiate an assisted telephone call, or a typed input to initiate an assisted telephone call. Furthermore, the user input to initiate an assisted telephone call may be received in a variety of interfaces (e.g., a search interface, a voice interface, an automated assistant interface, and / or other interface from which an assisted telephone call may be initiated).

[0100] At block 754, the system identifies an entity to be engaged on behalf of the given user during the assisted phone call based on the user input. The system may identify an entity to be engaged during the assisted phone call based on the user input received at block 752 and / or the user's interaction with the client device immediately prior to the initiation of the assisted phone call. For example, the system may identify the entity "exemplary airline" based on processing the speech input "I want to report my lost baggage to the exemplary airline" received at the client device. As another example, the system may identify the entity "exemplary airline" based on determining that a given user of the client device selected a call interface element associated with the exemplary airline or a particular graphical element associated with a candidate state of an IVR tree associated with the exemplary airline.

[0101] In block 756, the system identifies an IVR tree associated with the entity identified in block 754, the IVR tree including a plurality of candidate states. The IVR tree associated with the identified entity may be stored in association with the entity identified in block 754 (e.g., in entity database 151A of FIG. 1 and / or in IVR tree database 152A of FIG. 1). Based on the system (and / or additional instances of the system) previously crawling the IVR system associated with the identified entity, the stored IVR tree associated with the entity may have been previously stored in one or more databases, and / or the identified entity may provide the IVR tree for the IVR system associated with the identified entity. Furthermore, the system may continuously update the IVR tree for the IVR system associated with the identified entity based on multiple assisted telephone calls that navigate the IVR tree. The plurality of candidate states may correspond to particular nodes in the IVR tree to which the system can navigate during the assisted telephone call.

[0102] In block 758, the system receives context information. In some implementations, block 756 may include optional sub-block 758A and / or optional sub-block 758B. If included, in optional sub-block 758A, the system receives context information related to the client device. The context information related to the client device may include, for example, device state information (e.g., determined via device state engine 112), date and / or time information related to the client device's current location, the client device's current location, and / or other context information generated by the client device. If included, in optional sub-block 758B, the system receives context information related to a given user of the client device. Contextual information associated with a given user of a client device may include, for example, electronic communications (e.g., emails, text messages, voice messages, etc.) created by or received by the user of the client device, user account information associated with various user accounts of the given user of the client device (e.g., account status information, account numbers, account usernames, account passwords, etc.), financial information of the given user of the client device (e.g., bank account numbers, credit card numbers, payment application information, etc.), photo albums of the given user of the client device, social media profiles of the given user of the client device, user preferences of the user of the client device, personal information of the given user of the client device (e.g., name, telephone number, physical address, email address, social security number, date of birth, etc.), and / or other information associated with the given user of the client device.

[0103] In block 760, the system determines the availability of multiple candidate states as target states of the IVR tree based on context information associated with the client device and / or a given user of the client device. In various implementations, some candidate states of the IVR tree may or may not be available to a given user of the client device based on the context information. For example, assume that the entity identified in block 754 is a restaurant entity. In this example, some candidate states may be available only during the restaurant's business hours (e.g., candidate states related to placing a takeout order). As another example, assume that the entity identified in block 754 is an airline entity. In this example, some candidate states may be available only if the user account status exceeds a threshold membership status (e.g., only for gold status members or above). In other words, the IVR tree associated with the identified entity may be dynamic in that candidate states may or may not be available in some contexts.

[0104] In block 762, the system initiates execution of an assisted telephone call to navigate the IVR tree to the target state. The system may initiate execution of the assisted telephone call in response to additional user input selecting the target state. Further, the system may initiate execution of the assisted telephone call by establishing a communication session between a client device associated with the given user and an additional client device associated with the entity identified in block 754. The communication session may be established using various voice communication protocols, including, for example, VoIP, PSTN, and / or other telephony communication protocols. In some implementations, the system may navigate the IVR tree to the target state by emulating button presses to traverse nodes of the IVR tree, by rendering synthesized speech audio data including synthesized speech to traverse nodes of the IVR tree, and / or by rendering audio data including speech input of a given user associated with a client device (e.g., as described with respect to FIG. 2). In some versions of these implementations, the system may also utilize corresponding values ​​for parameters related to navigating the IVR tree. Corresponding values ​​for the parameters may be retrieved from one or more databases (eg, user profile database 153A of FIG. 1) and / or may be requested from a given user associated with the client device.

[0105] In block 764, the system renders a notification via the client device indicating the results of the execution of navigating the IVR tree. For example, the notification may include a request that a given user associated with the client device join an assisted telephone call. In implementations described herein in which the system also performs a task on behalf of a given user associated with the client device and for an entity, the notification may additionally or alternatively include the results of the execution of the task. In various implementations, the navigation of the IVR tree described with respect to FIG. 5 may be modified based on user interactions at the client device, as described above with respect to FIGS. 3 and 4A-4C.

[0106] 8A-8D, various non-limiting examples of user interfaces associated with navigating a dynamic IVR tree are shown. Each of Figures 8A-8D illustrates a client device 110 having a graphical user interface 180, which may include one or more of the components of the client device of Figure 1. Client device 110 may be substantially similar to the client device of Figures 4A-4C and may include many of the same components described above with respect to Figures 4A-4C, including an instance of an automated assistant (e.g., automated assistant 115 of Figure 1).

[0107] In various implementations described herein, an IVR tree associated with an entity may be a dynamic IVR tree in that one or more candidate states associated with the IVR tree may or may not be available. The availability of one or more of the candidate states may be determined based on context information. In some implementations, the context information may be context information associated with the client device 110 and may include, for example, device state information of the client device 110 (e.g., determined via the device state engine 112), date and / or time information associated with the current location of the client device 110, the current location of the client device, and / or other context information generated by the client device 110. In some additional or alternative implementations, the context information may be context information related to a given user of client device 110 and may include, for example, electronic communications (e.g., emails, text messages, voice messages, etc.) created by or received by the user of the client device, user account information (e.g., account status information, account numbers, account usernames, account passwords, etc.) related to various user accounts of the given user of client device 110, financial information of the given user of client device 110 (e.g., bank account numbers, credit card numbers, payment application information, etc.), photo albums of the given user of client device 110, social media profiles of the given user of client device 110, user preferences of the user of client device 110, personal information of the given user of client device (e.g., name, phone number, physical address, email address, social security number, date of birth, etc.), and / or other information related to the given user of client device.

[0108] 8A and 8B in detail, assume that a given user of client device 110 is viewing search results for restaurant entities in a browser application accessible at client device 110 (e.g., as indicated by URL 811A of "www.exampleurl0.com / "). Further assume that the search results include at least search result 820A for "Hypothetical Restaurant" that is open from 9:00 AM to 11:00 PM. In some implementations, search result 820A may be associated with various selectable graphical elements that, when selected, cause client device 110 to perform a corresponding action. For example, when call graphical element 821A associated with search result 820A is selected, user input may indicate that a phone call action to the virtual restaurant should be performed. As another example, when directions graphical element 822A associated with search result 820A is selected, user input may indicate that a navigation action to the virtual restaurant's real location should be performed. As yet another example, when menu graphical element 823A associated with search result 820A is selected, the user input may indicate that a menu lookup action should be performed for a menu associated with the virtual airline.

[0109] In various implementations, call details interface 870 may be rendered at client device 110 before receiving user input to initiate a phone call with the “virtual restaurant,” or may be rendered at client device 110 in response to receiving user input to initiate a phone call with the virtual restaurant. In some versions of those implementations, call details interface 870 may be rendered at client device 110 as part of user interface 180. In some other versions of those implementations, call details interface 870 may be a separate interface that overlays user interface 180 and may include call details interface element 186 that allows a user to expand call details interface 870 to display additional call details (e.g., by swiping up on call details interface element 186) and / or dismiss call details interface 870 (e.g., by swiping down on call details interface element 186). It should be understood that while call details interface 870 is shown as at the bottom of user interface 180, this is by way of example and not intended to be limiting. For example, call details interface 870 may be rendered on top of user interface 180 , on the side of user interface 180 , or in an interface that is entirely separate from user interface 180 .

[0110] The call details interface 870, in various implementations, can include multiple graphical elements. In some versions of those implementations, the graphical elements can be selectable, so that when a given one of the graphical elements is selected, the client device 110 can perform a corresponding action. As shown in Figures 8A and 8B, the call details interface 870 includes a first graphical element 871A titled "Assisted Call, Hypothetical Restaurant," a second graphical element 872A titled "Normal Call," and a third graphical element 873A titled "Save Contact 'Hypothetical Restaurant'." The graphical elements can also include sub-elements corresponding to candidate states associated with an IVR tree associated with the hypothetical restaurant. For example, and as shown in FIG. 8A , the first graphical element 871A may include a first sub-element 871A1 titled "Make Reservations" that is associated with a target state of navigating an IVR tree associated with a virtual restaurant to a portion corresponding to making a restaurant reservation, a second sub-element 871A2 titled "Order Carryout" that is associated with a target state of navigating an IVR tree associated with the virtual restaurant to a portion corresponding to ordering carryout from the restaurant, and a third sub-element 871A3 titled "Order Delivery" that is associated with a target state of navigating an IVR tree associated with the virtual restaurant to a portion corresponding to ordering delivery from the virtual restaurant.In contrast, and as shown in FIG. 8B, the first graphical element 871A may include a first sub-element 871B1, "Make a Reservation," associated with a target state of navigating an IVR tree associated with a virtual restaurant to a portion corresponding to making a restaurant reservation, and a second sub-element 871B2, "Hours of Operation," associated with a target state of navigating an IVR tree associated with the virtual restaurant to a portion corresponding to hearing the virtual restaurant's hours of operation.

[0111] 8A and 8B may be based on context information associated with a given user's client device 110. In particular, and with reference to FIG. 8A, time and date information 812A at the current location of client device 110 indicates a time of 9:30 AM, which corresponds to the virtual restaurant being open (e.g., based on business hours of 9:00 AM - 11:00 PM). As a result, because the virtual restaurant is open, candidate states associated with selectable sub-elements 871A2 and 871A3 may be available. In contrast, and with reference to FIG. 8B , time and date information 812B at the current location of client device 110 indicates a time of 8:30 AM, which corresponds to the virtual restaurant being closed (e.g., based on its business hours being 9:00 AM - 11:00 PM). As a result, although the candidate states associated with selectable sub-elements 871A2 and 871A3 may not be available because the virtual restaurant is closed, the automated assistant may still be able to make a restaurant reservation and / or inquire about business hours by navigating the IVR tree associated with the virtual restaurant even though it is closed. Thus, some of the candidate states may be available for a given context. An IVR tree may be dynamic in that some candidate states may be available in a given context, while other candidate states may not be available in that given context. As another example, in FIG. 8A , assume that a virtual restaurant is open, but the current location of client device 110 is outside the delivery area for the virtual restaurant. In this example, the candidate state associated with third sub-element 871A3, "Order Delivery," may be omitted. As yet another example, in FIG. 8A , assume that a virtual restaurant is open, but neither indoor nor outdoor dining is available at the virtual restaurant. In this example, the candidate state associated with first sub-element 871A1, "Make a Reservation," may be omitted.

[0112] 8C and 8D in detail, assume that a given user of client device 110 is viewing search results for hotel entities in a browser application accessible at client device 110 (e.g., as indicated by URL 811C of “www.exampleurl0.com / ”). Assume further that the search results include at least a first search result 820C of “Hypothetical Hotel” and a second search result 830C of “Example Hotel.” In some implementations, search result 820C and / or 830C may be associated with various selectable graphical elements that, when selected, cause client device 110 to perform a corresponding action. For example, when call graphical elements 821C and / or 831C, associated with search result 820A and / or 830C, respectively, are selected, the user input may indicate that a phone call action to the respective hotel entity should be performed. As another example, when directions graphical element 822C and / or 832C associated with search result 820A and / or 830C, respectively, is selected, the user input may indicate that a navigation action to the actual location of the respective hotel entity should be performed. As yet another example, when account element 823C and / or 833C associated with search result 820A and / or 830C, respectively, is selected, the user input may indicate that an account lookup action to the user account associated with the respective hotel entity should be performed.

[0113] In various implementations, call details interface 870 may be rendered at client device 110 before receiving user input to initiate a phone call with a respective one of the hotel entities, or may be rendered at client device 110 in response to receiving user input to initiate a phone call with a respective one of the hotel entities. In some versions of those implementations, call details interface 870 may be rendered at client device 110 as part of user interface 180. In some other versions of those implementations, call details interface 870 may be a separate interface that overlays user interface 180 and may include call details interface element 186 that allows a user to expand call details interface 870 to display additional call details (e.g., by swiping up on call details interface element 186) and / or dismiss call details interface 870 (e.g., by swiping down on call details interface element 186). It should be understood that while call details interface 870 is shown as being at the bottom of user interface 180, this is by way of example and not intended to be limiting. For example, call details interface 870 may be rendered at the top of user interface 180, to the side of user interface 180, or in an interface entirely separate from user interface 180.

[0114] The call details interface 870, in various implementations, can include multiple graphical elements. In some versions of those implementations, the graphical elements can be selectable, so that when a given one of the graphical elements is selected, the client device 110 can perform a corresponding action. As shown in FIG. 8C , the call details interface 870 includes a first graphical element 871C, "Assisted Call, Hypothetical Hotel," a second graphical element 872C, "Normal Call," and a third graphical element 873C, "Save Contact 'Hypothetical Hotel'." The graphical elements in FIG. 8C can also include sub-elements corresponding to candidate states associated with an IVR tree associated with the hypothetical hotel. For example, and as shown in FIG. 8C , the first graphical element 871C may include a first sub-element 871C1 entitled “Change Reservation” associated with a target state of navigating an IVR tree associated with the virtual hotel to a portion corresponding to changing a hotel reservation, a second sub-element 871C2 entitled “Request Free Upgrade” associated with a target state of navigating an IVR tree associated with the virtual hotel to a portion corresponding to requesting a free upgrade for an existing hotel reservation, and a third sub-element 871C3 entitled “Concierge Direct Line” associated with a target state of navigating an IVR tree associated with the virtual hotel to a portion corresponding to contacting the virtual hotel's concierge.

[0115] In contrast, and as shown in Figure 8D, call details interface 870 includes a first graphical element 871D titled "Assisted Call, Example Hotel" (rather than "Virtual Hotel" in Figure 8C), a second graphical element 872D titled "Regular Call," and a third graphical element 873D titled "Save Contact 'Example Hotel'." The graphical elements in Figure 8D may also include sub-elements corresponding to candidate states associated with the IVR tree associated with the example hotel. For example, and as shown in FIG. 8D , the first graphical element 871D may include a first sub-element 871D1 titled "Change Reservation" that is associated with a target state of navigating an IVR tree associated with an exemplary hotel to a portion corresponding to changing a hotel reservation, a second sub-element 871D2 titled "Cancel Reservation" that is associated with a target state of navigating an IVR tree associated with an exemplary hotel to a portion corresponding to canceling an existing hotel reservation, and a third sub-element 871D3 titled "Front Desk" that is associated with a target state of navigating an IVR tree associated with a virtual hotel to a portion corresponding to contacting the front desk at the exemplary hotel.

[0116] In particular, differences between the candidate states included in FIG. 8C and FIG. 8D may be based on context information associated with a given user of the given user's client device 110. In particular, and with reference to FIG. 8C , the automated assistant has access to a user account for a given user of the client device 110 associated with a virtual hotel. For example, the automated assistant may have access to a user account status (e.g., bronze member, silver member, gold member, etc.), a user account number, a user account history including stays at various locations of the virtual hotel, and the like. For purposes of example, assume that a given user of the client device 110 is a gold member of the virtual hotel. As a result, candidate states associated with selectable sub-elements 871C2 and 871C3 may be available based on the gold member status of a given user of the client device 110. In contrast, and with reference to FIG. 8D , the automated assistant may not have any record of a user account associated with the exemplary hotel. As a result, although candidate states associated with selectable sub-elements 871C2 and 871C3 may not be available, the automated assistant may still be able to navigate to the various candidate states shown in FIG. 8D . Thus, the IVR tree may be dynamic in that some of the candidate states may be available in a given context, while other candidate states may not be available in that given context.

[0117] 8A and 8B are described herein with respect to the availability of candidate states based on contextual information associated with a given user's client device 110, it should be understood that while FIGS. 8C and 8D are described herein with respect to the availability of candidate states based on contextual information associated with a given user of client device 110, this is by way of example and not intended to be limiting. For example, the availability of candidate states may be based on both contextual information associated with a given user's client device 110 and contextual information associated with a given user of client device 110. For example, and with respect to FIG. 8C, if a given user of client device 110 initiates an assisted phone call when the virtual hotel concierge is not working, the candidate states associated with the third sub-element 871C3, “Concierge Direct Line,” may be omitted, even though the user account status of the user account associated with the virtual hotel allows the given user of client device 110 to directly contact the virtual hotel concierge.

[0118] 8A-8D are described herein with respect to several entities, and several graphical elements are shown along with search results and call detail interfaces, it should be understood that this is for purposes of example and not intended to be limiting. It should be understood that an IVR tree associated with any entity employing an IVR system can be stored and subsequently utilized in conducting an assisted telephone call. It should further be understood that the graphical elements and information provided by the automated assistant can be based on an IVR tree, and that the configurations of IVR trees associated with entities are virtually limitless.

[0119] 9 is a block diagram of an example computing device 910 that may optionally be utilized to perform one or more aspects of the techniques described herein. In some implementations, one or more of the client device, the cloud-based automated assistant component, and / or other components may comprise one or more components of the example computing device 910.

[0120] Computing device 910 typically includes at least one processor 914 that communicates with several peripheral devices via a bus subsystem 912. These peripheral devices may include, for example, a storage subsystem 924 including a memory subsystem 925 and a file storage subsystem 926, a user interface output device 920, a user interface input device 922, and a network interface subsystem 916. The input and output devices enable user interaction with computing device 910. The network interface subsystem 916 provides an interface to external networks and is coupled to corresponding interface devices in other computing devices.

[0121] The user interface input devices 922 may include a keyboard, a pointing device such as a mouse, a trackball, a touchpad, or a graphics tablet, a scanner, a touchscreen integrated into a display, a voice recognition system, an audio input device such as a microphone, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and methods for inputting information into the computing device 910 or over a communications network.

[0122] The user interface output devices 920 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display, such as via an audio output device. In general, use of the term "output device" is intended to include all possible types of devices and methods for outputting information from the computing device 910 to a user or to another machine or computing device.

[0123] The storage subsystem 924 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 924 may include logic for performing selected aspects of the methods disclosed herein and for implementing the various components shown in FIG.

[0124] These software modules are generally executed by the processor 914 alone or in combination with other processors. The memory 925 used in the storage subsystem 924 can include several memories, including a main random access memory (RAM) 930 for storing instructions and data during program execution, and a read-only memory (ROM) 932 in which fixed instructions are stored. The file storage subsystem 926 can provide persistent storage for program and data files and may include a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of some implementations may be stored by the file storage subsystem 926 in the storage subsystem 924 or in other machines accessible by the processor 914.

[0125] The bus subsystem 912 provides a mechanism for allowing the various components and subsystems of the computing device 910 to communicate with each other as intended. Although the bus subsystem 912 is shown schematically as a single bus, alternative implementations of the bus subsystem 912 may use multiple buses.

[0126] Computing device 910 can be of different types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of computing device 910 shown in Figure 9 is intended only as a specific example to illustrate some implementations. Many other configurations of computing device 910 are possible, having more or fewer components than the computing device shown in Figure 9.

[0127] In situations where the systems described herein collect or possibly monitor personal information about a user or may utilize personal and / or monitored information, the user may be given the opportunity to control whether a program or feature collects user information (e.g., information about the user's social network, social actions or activities, occupation, user preferences, or the user's current geographic location) or to control whether and / or how content is received from content servers that may be more relevant to the user. Also, some data may be processed in one or more ways before being stored or used so that personally identifiable information is removed. For example, the user's identity may be processed so that personally identifiable information cannot be determined for the user, or the user's geographic location may be generalized (e.g., to the city level, zip code level, or state level) where the geographic location information is obtained so that the user's specific geographic location cannot be determined. Thus, the user may control how information is collected about and / or used.

[0128] In some implementations, a method implemented by one or more processors is provided, including receiving user input from a given user and via a client device associated with the given user to initiate an assisted telephone call; identifying an entity to be engaged on behalf of the given user during the assisted telephone call based on the user input; identifying an interactive voice response (IVR) tree associated with the identified entity, the IVR tree including a plurality of candidate states; determining a target state of the IVR tree from among the plurality of candidate states based on the user input or additional user input; and initiating execution of the assisted telephone call to navigate the IVR tree to the target state. Executing the assisted telephone call comprises automatically navigating the assisted telephone call to the target state of the IVR tree. The method further includes detecting at least one user interaction targeted at the client device and associated with the identified entity during execution of the assisted telephone call and while the IVR tree is navigated to the target state; and generating one or more prompts related to modifying navigation of the IVR tree based on the at least one user interaction targeted at the client device. The method further includes, in response to determining that further additional user input received in response to rendering one or more of the further prompts modifies navigation of the IVR tree, modifying execution of the assisted telephone call based on the further additional user input.

[0129] These and other implementations of the techniques disclosed herein can optionally include one or more of the following features.

[0130] In some implementations, the method may further include, after navigating the IVR tree to arrive at a particular state during the assisted telephone call and based on further additional user input, generating a notification requesting that a given user associated with the client device join the assisted telephone call, and causing the notification to be rendered at the client device associated with the given user. In some versions of these implementations, causing the notification to be rendered at the client device associated with the given user may include one or more of vibrating the client device, audibly rendering a notification requesting that the given user join the assisted telephone call at the client device, or visually rendering a notification requesting that the given user join the assisted telephone call at the client device. In some additional or alternative versions of these implementations, the method may further include, after navigating the IVR tree to a particular state, engaging in a conversation with a human agent associated with the identified entity to verify whether the particular state corresponds to the target state.

[0131] In some further versions of those implementations, engaging in a conversation with the human agent to verify whether a particular state corresponds to the target state may include processing audio data capturing the human agent's verbal utterances using a speech recognition model and determining whether the human agent verifies that the particular state corresponds to the target state based on processing the audio data. Generating a notification requesting that a given user associated with the client device join the assisted telephone call may be in response to a determination that the human agent verifies that the particular state corresponds to the target state. In still further versions of those implementations, the system may further include causing an additional client device associated with the human agent to render synthesized speech audio data including synthesized speech for verifying whether the IVR tree was successfully navigated. The audio data capturing the human agent's verbal utterances may be received in response to rendering the synthesized speech audio data on the additional client device associated with the human agent.

[0132] In some further additional or alternative versions of those implementations, the method may further include, in response to a determination by the human agent that the particular state does not correspond to the target state, causing further synthesized voice audio data to be rendered at an additional client device associated with the human agent, the further synthesized voice including further synthesized voice to request that the assisted telephone call be transferred to another human agent associated with the identified entity.

[0133] In some implementations, the method may further include identifying a task to be performed on behalf of the given user and during the assisted telephone call based on the user input or additional user input, and after navigating the IVR tree to the target state and during the assisted telephone call, engaging in a conversation with a human agent associated with the identified entity to complete the task on behalf of the given user. Engaging in a conversation with the human agent may include causing, at an additional client device associated with the human agent, synthetic speech audio data including synthetic speech for completing the task, capturing the oral utterance of the human agent and processing audio data received in response to the rendering of the synthetic speech using a speech recognition model, and determining a result associated with the performance of the task based on processing the audio data. In some versions of these implementations, the method may further include generating a notification based on the result associated with the performance of the task and causing the notification to be rendered at a client device associated with the given user.

[0134] In some implementations, the method may further include, in response to determining that further additional user input received in response to rendering one or more of the additional prompts does not modify navigation of the IVR tree, continuing execution of the assisted telephone call to navigate the IVR tree to the target state.

[0135] In some implementations, the method may further include, in response to identifying the IVR tree associated with the identified entity, obtaining one or more prompts related to navigating the IVR tree associated with the identified entity, and causing one or more of the prompts to be rendered at a client device associated with the given user. Additional user input may be received in response to the one or more of the prompts being rendered.

[0136] In some implementations, the at least one user interaction at a client device associated with a given user may include one or more of a search interaction at a client device associated with the identified entity, a browsing interaction at a client device associated with the identified entity, or a navigation interaction at a client device associated with the identified entity.

[0137] In some implementations, automatically navigating the assisted telephone call to a target state of the IVR tree may include one or more of determining one or more values ​​associated with corresponding parameters for navigating the IVR tree based on user input or additional user input. In some versions of those implementations, modifying the navigation of the IVR may include determining an alternative target state of the IVR tree based on user interaction or additional user input. The modified execution of the assisted telephone call may include automatically navigating the assisted telephone call to an alternative target state of the IVR tree based on one or more of the values ​​associated with corresponding parameters. In some further versions of those implementations, modifying the navigation of the IVR may include determining one or more alternative target states of the IVR tree based on user interaction or additional user input. The modified execution of the assisted telephone call may include automatically navigating the assisted telephone call to a target state of the IVR tree based on one or more alternative values ​​associated with corresponding parameters.

[0138] In some implementations, automatically navigating the assisted telephone call to a target state of the IVR tree may include one or more of emulating one or more button presses to be rendered at an additional client device associated with the identified entity to automatically navigate the IVR tree until the target state is reached, causing synthesized speech audio data including synthesized speech to be rendered at an additional client device associated with the identified entity to automatically navigate the IVR tree until the target state is reached, or causing audio data capturing verbal utterances of a given user associated with a client device to be rendered at an additional client device associated with the identified entity.

[0139] In some implementations, a method implemented by one or more processors is provided, the method including receiving user input from a given user via a search interface of a client device associated with the given user, issuing a search query in the search interface of the client device based on the user input, identifying given search results related to an entity to be engaged on behalf of the given user during the assisted telephone call based on the issued search query, the given search results related to a target state from among a plurality of candidate states for an interactive voice response (IVR) tree associated with the identified entity, causing at least the given search results to be rendered in the search interface of the client device, receiving a selection of the given search results from the given user via the search interface of the client device in response to the rendering of the given search results in the search interface of the client device, and initiating execution of the assisted telephone call in response to receiving the selection of the given search results. Executing the assisted telephone call includes automatically navigating the assisted telephone call to the target state of the IVR tree.

[0140] These and other implementations of the techniques disclosed herein can optionally include one or more of the following features.

[0141] In some implementations, the method may further include identifying additional given search results related to the entity based on the issued search query, the additional given search results being associated with an additional target state of the plurality of candidate states for the IVR tree. In some versions of those implementations, the method may further include causing the additional given search results to be rendered together with the given search result. In some further versions of those implementations, causing the additional given search results to be rendered together with the given search result may include rendering the given search result more prominently than the additional given search results.

[0142] In still further versions of these implementations, rendering the given search result more prominently than additional given search results may include rendering the given search result more prominently than additional given search results based on recent user interactions of the given user with the client device. In still further versions of these implementations, the recent user interactions of the given user with the client device may include the given user accessing content related to the given search result. In still further additional or alternative versions of these implementations, rendering the given search result more prominently than additional given search results may include rendering the given search result more prominently than additional given search results based on a reputation of the given search result. In still further versions of these implementations, the reputation of the given search result may be based on click-through rates of multiple users associated with the given search result and applied to the given user.

[0143] In some implementations, a method implemented by one or more processors is provided, including receiving user input from a given user and via a client device associated with the given user to initiate an assisted telephone call, identifying an entity to be engaged on behalf of the given user during the assisted telephone call based on the user input, identifying an interactive voice response (IVR) tree associated with the identified entity, the IVR tree including a plurality of candidate states, determining availability of the plurality of candidate states as target states of the IVR tree based on context information associated with the client device or a given user associated with the client device, determining a target state of the IVR tree from among the plurality of available candidate states based on the user input or additional user input, and initiating execution of the assisted telephone call to navigate the IVR tree to the target state. Executing the assisted telephone call includes automatically navigating the assisted telephone call to the target state of the IVR tree.

[0144] These and other implementations of the techniques disclosed herein can optionally include one or more of the following features.

[0145] In some implementations, the context information may be related to the client device, and the context information may include one or more of a current location of the client device, a current time at the client device's current location, or a current date at the client device's current location. In some versions of these implementations, a given IVR tree selected from among multiple heterogeneous IVR trees may be based on one or more of a current location of the client device or a current time at the client device's current location.

[0146] In some implementations, the context information may be related to a given user associated with the client device, and the context information may include one or more of an electronic communication created at or received at the client device, user account information associated with the given user's user account, or financial information associated with the given user. In some versions of these implementations, a given IVR tree selected from among a plurality of heterogeneous IVR trees may be based on one or more of an electronic communication created at or received at the client device, user account information associated with the given user's user account, or financial information associated with the given user.

[0147] Additionally, some implementations include one or more processors (e.g., central processing units (CPUs), graphics processing units (GPUs), and / or tensor processing units (TPUs)) of one or more computing devices, where the one or more processors are operable to execute instructions stored in associated memory, where the instructions are configured to cause performance of any of the above-described methods. Some implementations also include one or more non-transitory computer-readable storage media that store computer instructions executable by the one or more processors to perform any of the above-described methods. Some implementations also include computer program products including instructions executable by the one or more processors to perform any of the above-described methods.

[0148] It should be appreciated that all combinations of the above concepts and additional concepts described in more detail herein are contemplated as being part of the presently disclosed subject matter, for example, all combinations of claimed subject matter listed at the end of this disclosure are contemplated as being part of the presently disclosed subject matter. [Explanation of symbols]

[0149] 110 client devices 111 User Input Engine 112 Device State Engine 113 Rendering Engine 114 Search Engines 115 Automated Assistants 120A Voice Recognition Model 120A1, 120A2 voice recognition engine 130A Natural Language Understanding Model 130A1, 130A2 Natural Language Understanding Engine 140A speech synthesis model 140A1, 140A2 speech synthesis engine 150 Assisted Phone Call Engine 151 Entity Identification Engine 151A Entity Database 152 Interactive Voice Response (IVR) Engine 152A IVR Tree Database 153 Parameter Engine 153A User Profile Database 154 User Interaction Engine 155 Verification Engine 156 Notification Engine 157 Task Execution Engine 160 Assisted Telephone Call System 180 Graphical User Interface 181, 182, 183 System interface elements 184 Text-Responsive Interface Elements 185 Voice Response Interface Elements 186 Call Details Interface Elements 190 Network 200 IVR Tree 211 Root Node 221, 222, 223 internal nodes 231, 232, 233, 234, 235, 236 Leaf nodes 411 URL 420 first search result 421 Call Graphical Elements 422 Directions Graphical Elements 423 Division Graphical Elements 430 Second Search Result 431 Call Graphical Elements 432 Directions Graphical Elements 433 Division Graphical Elements 442C First selectable graphical element 443C Second selectable graphical element 444C Third selectable graphical element 452C1, 452C2, 452C3, 452C4 Oral Speech 454C1 Synthetic Speech 470 Call Detail Interface 471A First Graphical Element 471A1 First subelement 471A2 Second subelement 471A3 Third subelement 472A Second Graphical Element 473A Third Graphical Element 479 Notification 479B1 First proposal 479B2 Second proposal 611 URL 620 first search result 620A1 First IVR Deep Link 620A2 Secondary IVR Deep Link 620A3 Additional Graphical Elements 621 Call Graphical Elements 622 User Account Graphical Elements 623 Flight Graphical Elements 652D1 Speech response, oral speech 652D2 Oral speech 652E1 Speech response 654C1 Oral speech 670 Call Details Interface 671 First Graphical Element 671A First Information 671B Second Information 671C Graphical Elements 672A Second Graphical Element 673A Third Graphical Element 679C Notification 679C1, 679C2, 679C3, 679C4, 679C5 candidate states 679D, 679E Notification 684A Search Query 811A, 811C URL 812A, 812B Time and Date Information 820A Search Results 820C 1st search result 821A, 821C Call Graphical Elements 822A, 822C Directions Graphical Elements 823A Menu Graphical Elements 823C Account Elements 830C 2nd search result 831C Call Graphical Elements 832C Directions Graphical Elements 833C Account Elements 871A First Graphical Element 871A1 First subelement 871A2 Second subelement 871A3 Third subelement 871B1 First subelement 871B2 Second subelement 871C First Graphical Element 871C1 First subelement 871C2 Second subelement 871C3 Third Subelement 871D First Graphical Element 871D1 First subelement 871D2 Second subelement 871D3 Third Subelement 872A, 872C, 872D Secondary Graphical Elements 873A, 873C, 873D Third Graphical Element 870 Call Detail Interface 910 Computing Devices 912 Bus Subsystem 914 processor 916 Network Interface Subsystem 920 User Interface Output Device 922 User Interface Input Devices 924 Memory Subsystem 925 Memory Subsystem 926 File Storage Subsystem 930 Random Access Memory (RAM) 932 Read-Only Memory (ROM)

Claims

1. 1. A method implemented by one or more processors, comprising: receiving a user input from a given user and via a client device associated with said given user to initiate an assisted telephone call; identifying an entity to be engaged on behalf of the given user during the assisted telephone call based on the user input; identifying an interactive voice response (IVR) tree associated with the identified entity, the IVR tree including a plurality of candidate states; initiating execution of the assisted telephone call; During the assisted telephone call, determining a corresponding confidence level associated with each of the plurality of candidate states of the IVR tree, the confidence level being based on how frequently the IVR tree traversed during the assisted telephone call matches an IVR tree stored in an IVR tree database in association with the IVR tree, the corresponding confidence level associated with a first subset of the plurality of candidate states of the IVR tree meeting a first threshold confidence level, and the corresponding confidence level associated with a second subset of the plurality of candidate states of the IVR tree not meeting the first threshold confidence level but meeting a second threshold confidence level; determining that the set of candidate states of the IVR tree to be provided for presentation to the user includes the first subset of the plurality of candidate states of the IVR tree but does not include the second subset of the plurality of candidate states of the IVR tree. determining determining a set of candidate states of the IVR tree to be offered for presentation to the user based on the corresponding confidence level associated with each of the plurality of candidate states of the IVR tree; providing the set of candidate states of the IVR tree for presentation to the user; A method for providing the above.

2. the corresponding confidence level associated with each of the plurality of candidate states of the IVR tree meets a first threshold confidence level, and determining the set of candidate states of the IVR tree to be provided for presentation to the user based on the corresponding confidence level associated with each of the plurality of candidate states of the IVR tree comprises: determining that the set of candidate states of the IVR tree to be offered for presentation to the user includes each of the plurality of candidate states of the IVR tree. The method of claim 1.

3. causing the set of candidate states of the IVR tree to be provided for presentation to the user, causing an IVR system utilizing the IVR tree associated with the identified entity to provide the set of candidate states of the IVR tree for presentation to the user before rendering a synthesized speech identifying any of the candidate states of the IVR tree. The method of claim 2.

4. causing the set of candidate states of the IVR tree to be provided for presentation to the user, causing an IVR system utilizing the IVR tree associated with the identified entity to provide the first subset of the candidate states of the IVR tree for presentation to the user before rendering a synthesized speech identifying any of the candidate states of the IVR tree. The method of claim 3.

5. causing the set of candidate states of the IVR tree to be provided for presentation to the user, after the IVR system renders a synthesized speech identifying the first subset of the candidate states of the IVR tree; causing the second subset of the candidate states of the IVR tree to be provided for presentation to the user. The method of claim 4.

6. The method of claim 1 , wherein the first subset of the candidate states of the IVR tree and the second subset of the candidate states of the IVR tree are mutually exclusive.

7. determining the set of candidate states of the IVR tree to be provided for presentation to the user based on the corresponding confidence level associated with each of the plurality of candidate states of the IVR tree, wherein the corresponding confidence level associated with each of the plurality of candidate states of the IVR tree does not meet a first threshold confidence level and does not meet a second threshold confidence level; after the IVR system has rendered a corresponding synthetic speech identifying each of the candidate states of the IVR tree; determining that the set of candidate states of the IVR tree to be offered for presentation to the user includes each of the candidate states of the IVR tree. The method of claim 1.

8. causing the set of candidate states of the IVR tree to be provided for presentation to the user, causing each of the candidate states of the IVR tree to be offered for presentation to the user. The method of claim 7.

9. determining the set of candidate states of the IVR tree to be provided for presentation to the user based on the corresponding confidence level associated with each of the plurality of candidate states of the IVR tree, wherein the corresponding confidence level associated with each of the plurality of candidate states of the IVR tree does not meet a first threshold confidence level and does not meet a second threshold confidence level; an IVR system utilizing the IVR tree associated with the identified entity, after rendering a corresponding synthetic speech identifying each of the candidate states of the IVR tree; determining that the set of candidate states of the IVR tree to be offered for presentation to the user includes each of the candidate states of the IVR tree. The method of claim 1.

10. causing the set of candidate states of the IVR tree to be provided for presentation to the user, The method of claim 7 , further comprising the step of causing each of the candidate states of the IVR tree to be provided for presentation to the user.

11. causing the set of candidate states of the IVR tree to be provided for presentation to the user, causing a corresponding selectable graphical element associated with a corresponding one of the candidate states included in the set of candidate states of the IVR tree to be provided for presentation to the user. The method of claim 1.

12. receiving a user selection of a given corresponding selectable graphical element from among the corresponding selectable graphical elements, the given corresponding selectable graphical element being associated with a given candidate state of the IVR tree from among the plurality of candidate states of the IVR tree; automatically navigating the assisted telephone call to a given candidate state of the IVR tree associated with the given corresponding selectable graphical element; 12. The method of claim 11, further comprising:

13. 12. The method of claim 11, wherein each of the corresponding selectable graphical elements includes textual content that differs from synthesized speech generated by an IVR system utilizing the IVR tree associated with the identified entity.

14. 2. The method of claim 1, wherein the corresponding confidence level associated with each of the plurality of candidate states of the IVR tree is based on a quantity of assisted telephone calls with the identified entity, the assisted telephone calls including navigating an IVR tree associated with the identified entity but different from a stored IVR tree associated with the identified entity.

15. The method of claim 1 , further comprising providing the transcription of the assisted telephone call for presentation to the user in a first portion of a graphical user interface of the client device.

16. 16. The method of claim 15, wherein the transcription of the assisted telephone call is provided for presentation to the user in a first portion of a graphical user interface of the client device, and the set of candidate states of the IVR tree is provided for presentation to the user in a second portion of the graphical user interface.

17. The method of claim 1 , wherein the user of the client device is not a currently participating participant in the assisted telephone call.

18. 1. A system comprising: one or more processors; and a memory storing instructions that, when executed, cause the one or more processors to: receiving a user input from a given user and via a client device associated with the given user to initiate an assisted telephone call; identifying an entity to be engaged on behalf of the given user during the assisted telephone call based on the user input; identifying an interactive voice response (IVR) tree associated with the identified entity, the IVR tree including a plurality of candidate states; initiating execution of the assisted telephone call; During the assisted telephone call, determining a corresponding confidence level associated with each of the plurality of candidate states of the IVR tree, the confidence level being based on how frequently the IVR tree traversed during the assisted telephone call matches an IVR tree stored in association with the IVR tree in an IVR tree database; determining a set of candidate states of the IVR tree to be offered for presentation to the user based on the corresponding confidence level associated with each of the plurality of candidate states of the IVR tree; providing the set of candidate states of the IVR tree for presentation to the user; A system that enables

19. A computer-readable storage medium storing instructions that, when executed, cause one or more processors to: receiving a user input from a given user and via a client device associated with the given user to initiate an assisted telephone call; identifying an entity to be engaged on behalf of the given user during the assisted telephone call based on the user input; identifying an interactive voice response (IVR) tree associated with the identified entity, the IVR tree including a plurality of candidate states; initiating execution of the assisted telephone call; During the assisted telephone call, determining a corresponding confidence level associated with each of the plurality of candidate states of the IVR tree, the confidence level being based on how frequently the IVR tree traversed during the assisted telephone call matches an IVR tree stored in association with the IVR tree in an IVR tree database; determining a set of candidate states of the IVR tree to be offered for presentation to the user based on the corresponding confidence level associated with each of the plurality of candidate states of the IVR tree; providing the set of candidate states of the IVR tree for presentation to the user; 1. A computer-readable storage medium enabling operations including:

20. at least one processor; a memory storing instructions that, when executed, cause said at least one processor to perform a method according to any one of claims 1 to 17; At least one computing device comprising:

21. 18. A computer-readable storage medium storing instructions that, when executed, cause at least one processor to perform the method of any one of claims 1 to 17.

Images