Automatic navigation of interactive voice response (IVR) tree on behalf of human user

Abstract

The invention relates to automatic navigation of an interactive voice response (IVR) tree on behalf of a human user. Embodiments are directed to utilizing an assistant to automatically navigate an interactive voice response (IVR) tree to reach a target state during an auxiliary telephone call. An assistant can receive input for initiating an auxiliary telephone call, and based on the input, identify an entity to engage on behalf of a user during the auxiliary telephone call, and identify an IVR tree stored in association with the entity. In various implementations, navigation of the IVR tree can be modified based on interactions detected at the client device after initiating an auxiliary telephone call. In various implementations, a secondary telephone call can be initiated from a search interface, and a target state can be associated with a given search result. In various embodiments, the IVR tree can be dynamic because only a subset of candidate states of the IVR tree can be used as a target state.

Description

[0001] Division Explanation

[0002] This application is a divisional application of Chinese Patent Application No. 202080100936.5 with an application date of December 14, 2020. Background Art

[0003] Users can interact with an automated assistant via various computing devices, such as smart phones, tablet computers, wearable devices, automotive systems, stand-alone personal assistant devices, etc. The automated assistant receives input from the user (e.g., verbal, touch, and / or typed) and responds with a responsive output (e.g., visual and / or auditory).

[0004] Users can interact with the automated assistant so that the automated assistant performs actions on behalf of the user. As an example, the automated assistant can navigate an interactive voice response (IVR) system, can place a phone call on behalf of the user to perform a given action, and can participate in a conversation with other users to perform an action. For example, a user can provide user input that requests the automated assistant to make a restaurant reservation by phone and on behalf of the user. The automated assistant can initiate a phone call to a specific restaurant and can navigate the IVR system associated with the specific restaurant to make a reservation, and / or can provide reservation information to additional personnel associated with the specific restaurant to make a reservation. Then, when the person associated with the specific restaurant is an active participant in the phone call, the automated assistant can notify the user that a restaurant reservation has been successfully made on behalf of the user during the phone call, and / or can enable the user to join the phone call. Summary of the Invention

[0005] Embodiments disclosed herein relate to using an automated assistant to perform an assisted phone call on behalf of a given user with an entity to navigate an interactive voice response (IVR) tree associated with the entity. The IVR tree associated with the entity can be stored in a database accessible by a client device of the given user. Performing the assisted phone call to navigate the IVR tree can include automatically navigating the IVR tree to a target state of the IVR tree. Based on user input detected at the client device of the given user, the target state of the IVR tree can be determined from among a plurality. The automated assistant can automatically navigate the IVR tree to the target state of the IVR tree via an instance of synthesized voice, which can be rendered as part of the assisted phone call, which can include injecting the synthesized voice or human voice into the assisted phone such that the synthesized voice or human voice is audibly perceivable to the IVR system (but not necessarily to the given user), and / or an instance of an emulated button press can be rendered as part of the assisted phone call, which can include injecting the emulated button press into the call such that it is perceivable by the IVR system. The synthesized voice, human voice, and / or emulated button press can be determined based on corresponding values of parameters associated with navigating the IVR tree to the target state. The parameters can be stored in association with the IVR tree associated with the entity and can be retrieved from the database and / or requested from the given user of the client device for the corresponding values of the parameters. In some embodiments, the automated assistant can cause the client device to render a notification requesting that a user of the client device join the assisted phone call. In some additional or alternative embodiments, after automatically navigating the IVR tree to the target state, the automated assistant can optionally perform a task on behalf of the user. For example, the automated assistant can participate in a conversation with a human representative associated with the entity to perform a task on behalf of the user.

[0006] Some embodiments relate to modifying the navigation of an IVR tree based on user interactions detected at a client device after initiating an assisted telephone call. The automated assistant is capable of generating and rendering a prompt related to modifying the navigation of the IVR tree and is capable of determining whether to modify the navigation of the IVR tree based on user input received in response to the prompt. In some versions of those embodiments, modifying the navigation of the IVR tree can include modifying a target state to an alternative target state or adding an additional target state to navigate to after navigating to the target state. For example, assume that the automated assistant is navigating to a target state associated with the painting department of an IVR tree associated with a home improvement entity. In this example, if a given user interacts with the home and garden section of a web page associated with the home improvement entity, the user can be prompted to additionally or alternatively navigate to a target state associated with the home and garden department of the IVR tree associated with the home improvement entity. In some additional and / or alternative versions of those embodiments, modifying the navigation of the IVR tree can include modifying the corresponding value of a parameter associated with navigating the IVR tree to the target state. Continuing with the above example, if a given user interacts with a web page associated with a particular brand of paint, the user can be prompted to use the corresponding value associated with the particular brand of paint for the paint brand parameter. In these and other ways, client device resources and / or network resources can be conserved by preventing the restart of another assisted telephone call performed with an alternative target state and / or alternative value for navigating the IVR tree. Additionally, the assisted telephone call can end more quickly, thereby reducing the total duration of computer and / or network resources used to perform the assisted telephone call. Additionally, battery resources of the client device can also be conserved because a given user is not an active participant in the assisted telephone call and the client device can be placed in a locked or low power state during the assisted telephone call.

[0007] Some additional or alternative embodiments relate to providing an IVR deep link to a candidate state of an IVR tree and search results received in response to a search query determined based on user input. In some versions of those embodiments, a given IVR deep link can be provided based on a most recent user interaction at a client device prior to issuing the search query. For example, assume that a user of a client device is interacting with search results related to reporting lost luggage to an airline entity and then issues a search for the airline entity. In this example, the automated assistant can cause an IVR deep link associated with the target state of reporting lost luggage (and optionally more prominently than other IVR deep links that may also be provided) to be provided for selection by a given user of the client device based on the most recent user interaction. In some additional and / or alternative versions of those embodiments, a given IVR deep link can be provided based on terms of the search query. For example, assume that a given user of a client issues a search query of "Report lost luggage to Example Airlines". In this example, the automated assistant can cause an IVR deep link associated with the target state of reporting lost luggage to be provided for selection by a given user of the client device based on the terms of the search query that include "Report lost luggage" and "Example Airlines". In these and other ways, the amount of user input can be reduced, thereby conserving client device resources because a given user only needs to select the "IVR deep link". Additionally, the assisted phone call can end more quickly, thereby reducing the total duration of computer and / or network resources used to perform the assisted phone call. Additionally, battery resources of the client device can also be conserved because a given user is not an active participant in the assisted phone call and the client device can be placed in a locked or low-power state during the assisted phone call.

[0008] Some additional or alternative embodiments relate to determining the availability of candidate states based on context information associated with a client device of a given user and / or context information associated with a given user of the client device. Context information associated with the client device can include, for example, device status information (e.g., on state, off state, locked state, sleep state), date and / or time information associated with the current location of the client device, the current location of the client device, and / or other context information generated by the client device. For example, if a restaurant entity is open, candidate states associated with the restaurant entity can include a takeout candidate state and a delivery candidate state. However, if the restaurant entity is closed, the candidate states associated with the restaurant entity may not include the takeout candidate state or the delivery candidate state, or if the client device is outside the delivery radius, even if the restaurant is open, the candidate states associated with the restaurant entity may not include the delivery candidate state. Additionally, context information associated with a given user of the client device can include, for example, electronic communications created by or received by the user of the client device, user account information associated with various user accounts of the given user of the client device, money information of the given user of the client device, an album of the given user of the client device, a social media profile of the given user of the client device, user preferences of the user of the client device, personal information of the given user of the client device, and / or other information associated with the given user of the client device. For example, if a given user is a gold or higher member at a hotel entity, candidate states associated with the hotel entity can include a contact concierge state or a request free upgrade state. However, if the given user is not a gold or higher member at the hotel entity, the candidate states can omit these states. In these and other ways, client device resources and / or network resources can be conserved by preventing the restart of another assisted phone call that might be made for a candidate state that may not be available for a given user. Additionally, the assisted phone call can end more quickly, thereby reducing the total duration of computer and / or network resources used to perform the assisted phone call. Additionally, battery resources of the client device can also be conserved because the given user is not an active participant in the assisted phone call and the client device can be placed in a locked or low-power state during the assisted phone call.

[0009] Some additional or alternative embodiments relate to rendering candidate states in various ways based on a confidence level of an IVR tree associated with an entity. The confidence level of the IVR tree stored in association with the entity can be based on, for example, the number of secondary phone calls with the entity that navigate an IVR tree different from the IVR tree stored in association with the entity, and optionally based on a degree of difference between the navigated IVR tree and the stored IVR tree. For example, if the confidence level of the IVR tree meets a first threshold indicating that the automated assistant is highly confident in the IVR tree associated with the entity, the automated assistant can cause a sequence of button presses to be rendered at the client device to inform a given user of the client device how to quickly navigate the IVR tree to one or more candidate states (e.g., press "1", "3", "5" to report lost luggage) before or when initiating a secondary phone call, or the automated assistant can cause selectable graphical elements associated with the candidate states to be rendered on the client device (which may not match the sequence of button presses), and when selected, the selectable graphical elements cause the automated assistant to automatically navigate to a target state associated with the selected selectable graphical element (e.g., press "5" to report lost luggage) before or when initiating a secondary phone call. As another example, if the confidence level of the IVR tree fails to meet the first threshold but meets a second threshold, which indicates that the automated assistant is mildly confident in the IVR tree associated with the entity, then the automated assistant can cause the above-described selectable graphical elements associated with the candidate states to be rendered on the client device only when traversing a portion of the IVR tree in which the automated assistant is mildly confident (which may not match the sequence of button presses). As yet another example, if the confidence level of the IVR tree fails to meet the first threshold and the second threshold, which indicates that the automated assistant has no confidence in the IVR tree associated with the entity, then the automated assistant can cause the above-described selectable graphical elements associated with the candidate states to be rendered at the client device after the automated assistant perceives the options of the IVR tree (which may not match the sequence of button presses). In these and other ways, client device resources and / or network resources can be conserved by preventing the restart of another secondary phone call in the case where the automated assistant provides incorrect information to the user to navigate the IVR tree. Additionally, the secondary phone call can end more quickly, thereby reducing the total duration of the computer and / or network resources used to perform the secondary phone call by providing a sequence of button presses or selectable elements associated with the candidate states. Additionally, the battery resources of the client device can also be conserved because a given user is not an active participant in the secondary phone call and the client device can be placed in a locked or low-power state during the secondary phone call.

[0010] The above description is provided as an overview of some of the embodiments disclosed herein. These and other embodiments are described in additional detail herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 A block diagram depicting an example environment that illustrates aspects of the present disclosure and in which embodiments disclosed herein can be implemented.

[0012] Figure 2 An example interactive voice response (IVR) tree is depicted in accordance with various embodiments.

[0013] Figure 3 A flowchart depicting an example method of modifying navigation of an interactive voice response (IVR) tree based on user interactions at a client device associated with a given user, in accordance with various embodiments.

[0014] Figure 4A 、 Figure 4B and Figure 4C Various non-limiting examples of user interfaces associated with modifying navigation of an interactive voice response (IVR) tree based on user interactions at a client device associated with a given user are depicted in accordance with various embodiments.

[0015] Figure 5 A flowchart depicting an example method of navigating an interactive voice response (IVR) tree from a search interface at a client device associated with a given user, in accordance with various embodiments.

[0016] Figure 6A 、 Figure 6B 、 Figure 6C 、 Figure 6D and Figure 6E Various non-limiting examples of user interfaces associated with navigating an interactive voice response (IVR) tree from a search interface at a client device associated with a given user are depicted in accordance with various embodiments.

[0017] Figure 7 A flowchart depicting an example method of navigating a dynamic interactive voice response (IVR) tree, in accordance with various embodiments.

[0018] Figure 8A 、 Figure 8B 、 Figure 8C and Figure 8D Various non-limiting examples of user interfaces associated with navigating a dynamic interactive voice response (IVR) tree are depicted in accordance with various embodiments.

[0019] Figure 9 An example architecture of a computing device is depicted in accordance with various embodiments. Detailed implementation manners

[0020] Now turn to Figure 1 , which depicts a block diagram of an example environment that demonstrates various aspects of the present disclosure and in which the implementations disclosed herein can be implemented. In Figure 1 , a client device 110 is shown, and the client device 110 includes a user input engine 111, a device state engine 112, a rendering engine 113, a search engine 114, a speech recognition engine 120A1, a natural language understanding (“NLU”) engine 130A1, and a speech synthesis engine 140A1 in various implementations.

[0021] The user input engine 111 is capable of detecting various types of user input at the client device 110. The user input detected at the client device 110 can include: verbal input detected via the microphone of the client device 110 by an additional user associated with an entity during an assisted phone call and / or additional verbal input transmitted from an additional client device to the client device 110; touch input detected via a user interface input device (e.g., a touch screen) of the client device 110; and / or typed input detected via a user interface input device (e.g., via a virtual keyboard on the touch screen) of the client device 110. The additional user associated with the entity can be, for example, an additional human participant associated with an additional client device, an additional automated assistant associated with the additional client device of the additional user, and / or other additional users.

[0022] The described assisted telephone calls can be performed using various voice communication protocols (e.g., Voice over Internet Protocol (VoIP), Public Switched Telephone Network (PSTN), and / or other telephone communication protocols). As described herein, synthesized speech can be rendered as part of an assisted telephone call, which can include injecting the synthesized speech into the call such that it is perceivable by at least one participant in the assisted telephone call. Also as described herein, simulated button presses can be rendered as part of an assisted telephone call, which can include injecting the simulated button presses into the call such that it is perceivable by at least one participant in the assisted telephone call. The synthesized speech can be generated and / or injected by the client device 110, which is one of the endpoints of the call, and / or can be generated and / or injected by a server that communicates with the client device 110 and is also connected to the assisted telephone call. Also as described herein, audible output can also be rendered outside of the assisted telephone call, which does not include injecting the audible output into the call, but the audible output can be detected by a microphone of the client device 110 connected to the assisted telephone call and can thus be perceived during the call. In some embodiments, the assisted telephone call can optionally be muted and / or filtered to reduce the perception of audible output rendered outside of the assisted telephone call during the assisted telephone call.

[0023] In various embodiments, the automated assistant 115 (generally indicated by the dashed line in Figure 1 can perform an assisted telephone call at the client device 110 using the assisted telephone call system 160 over a network 190 (e.g., Wi-Fi, Bluetooth, Near Field Communication, Local Area Network, Wide Area Network, and / or other networks). The assisted telephone call system 160 includes a speech recognition engine 120A2, an NLU engine 130A2, a speech synthesis engine 140A2, and an assisted telephone call engine 150 in various embodiments. In some embodiments, the automated assistant 115 can utilize the assisted telephone call system 160 to navigate an Interactive Voice Response (IVR) tree of an IVR system associated with an entity on behalf of the user of the client device 110 during an assisted telephone call with an additional user. In some additional or alternative embodiments, the automated assistant 115 can utilize the assisted telephone call system 160 to perform tasks on behalf of the user of the client device 110 during an assisted telephone call with an additional user. In some additional or alternative embodiments, the automated assistant 115 can utilize the assisted telephone call system 160 to navigate non-interactive systems (e.g., a waiting-to-be-answered system, a voicemail system) and / or identify whether a failure has occurred in the IVR system or nothing has occurred during the assisted telephone call (and optionally terminate the assisted telephone call and notify the user).

[0024] In addition, in some embodiments, after navigating the IVR tree and before a user of the client device 110 performs any tasks, the automated assistant 115 can obtain the consent of an additional user to participate in a conversation with the automated assistant 115. For example, the automated assistant 115 can obtain consent to process audio data corresponding to the speech of a human participant when connecting with the human participant. If the automated assistant 115 obtains consent from the additional user, then the automated assistant 115 can verify that the target state of the IVR tree has been reached (e.g., as described below with respect to the verification engine 155), and / or can perform tasks on behalf of the user (e.g., as described below with respect to the task engine 157). However, if the automated assistant 115 does not obtain consent from the additional user, the automated assistant 115 can cause the client device 110 (e.g., using the rendering engine 113) to render to the user of the client device 110 a notification indicating that the user needs to take over the assisted phone call and / or terminate the assisted phone call and (e.g., using the rendering engine 113) render a notification to the user of the client device 110.

[0025] As described in detail below, the automated assistant 115 can use the assisted phone call system 160 to perform an assisted phone call in response to detecting user input from a user of the client device 110 to initiate a phone call using the automated assistant 115. In some embodiments, the automated assistant 115 can determine, based on user input received at the client device 110, a target state for navigation of an IVR tree associated with an entity to be contacted on behalf of the user of the client device 110 during the assisted phone call from among a plurality of candidate states. In some versions of those embodiments, the automated assistant 115 can receive a selection of the target state before or at the time of initiating the assisted phone call. In some additional or alternative versions of those embodiments, the automated assistant 115 can receive a sequence of selections resulting in the target state before or at the time of initiating the assisted phone call. The automated assistant 115 can navigate the IVR tree until the target state is reached, and can cause (e.g., using the rendering engine 113) a notification that the automated assistant 115 has reached the target state to be rendered at the client device 110. In various embodiments, the automated assistant 115 can additionally perform tasks on behalf of the user when the target state is reached, and can cause a notification of the result of the performance of the tasks to be rendered at the client device 110, including (e.g., using the rendering engine 113).

[0026] As Figure 1 shown, the assisted phone call system 160 can be remotely implemented (e.g., via a server and / or other remote client devices) and communicate with the client device 110. Although in Figure 1The assisted telephone call system 160 is shown as being implemented remotely via the network 190 and communicating with the client device 110, but it should be understood that this is for illustrative purposes only and is not meant to be limiting. For example, in various embodiments, the assisted telephone call system 160 can be implemented locally on the client device 110. As another example, in various embodiments, the assisted telephone call system 160 can be implemented remotely at the assisted telephone call system 160 without any connection to the client device 110 (e.g., cloud-based assistance telephone call), and the client device 110 can optionally be in a powered-off state or not connected to any network (e.g., determined using the device status engine 112). Additionally, although Figure 1 the automated assistant 115 is shown as being implemented locally on the client device 110 and remotely at the assisted telephone call system 160, it should be understood that this is also for illustrative purposes only and is not meant to be limiting. For example, in various embodiments, the automated assistant 115 can be implemented locally on the client device 110, while in other embodiments, the automated assistant 115 can be implemented locally on the client device 110 and interact with a separate cloud-based automated assistant (e.g., forming a logical instance of the automated assistant 115).

[0027] In an implementation where the user input engine 111 (e.g., during an assisted phone call) detects a user's verbal input via the microphone of the client device 110 and / or receives audio data captured from an additional user and transmitted from an additional client device to the client device 110, the speech recognition engine 120A1 of the client device 110 can use the speech recognition model 120A to process the audio data capturing the verbal input and / or the additional verbal input. The speech recognition engine 120A1 can generate recognized text corresponding to the verbal input and / or the additional verbal input based on the processing of the audio data. Additionally, the NLU engine 130A1 of the client device 110 can use the NLU model 130A to process the recognized text generated by the speech recognition engine 120A1 to determine the intent included in the verbal input and / or the additional verbal input. For example, if the client device 110 detects a verbal input from the user of "Call HypotheticalAirline to report lost luggage", the client device 110 can use the speech recognition model 120A to process the audio data capturing the verbal input to generate recognized text corresponding to the verbal input of "Call HypotheticalAirline to report lost luggage", and can use the NLU model 130A to process the recognized text to at least determine a first intent to initiate a call and a second intent to report lost luggage. As another example, if the client device 110 detects an additional verbal input of "Simply tell us why you're calling", the client device 110 can use the speech recognition model 120A to process the audio data capturing the additional verbal input to generate recognized text corresponding to the additional verbal input of "Simply tell us why you're calling", and can use the NLU model 130A to process the recognized text to determine an intent of a request natural language input including a request for information associated with why the assisted phone call was initiated (e.g., reporting lost luggage). In some versions of those implementations, the client device 110 can transmit the audio data, the recognized text, and / or the intent to the assisted phone call system 160.

[0028] In other embodiments, when the user input engine 111 detects the user's verbal input via the microphone of the client device 110 and / or receives audio data that captures additional verbal input from an additional client device and is transmitted to the client device 110 (e.g., during an assisted phone call and / or during an ongoing call), the automated assistant 115 can cause the client device 110 to send the audio data that captures the verbal input and / or the additional verbal input to the assisted phone call system 160. The speech recognition engine 120A2 and / or the NLU engine 130A2 of the assisted phone call system 160 can process the audio data that captures the verbal input and / or the audio data that captures the additional verbal utterance in a similar manner as described above for the speech recognition engine 120A1 and / or the NLU engine 130A1 of the client device 110. In some additional and / or alternative embodiments, the speech recognition engine 120A1 and / or the NLU engine 130A1 of the client device 110 can be used in combination with the speech recognition engine 120A2 and / or the NLU engine 130A2 of the assisted phone call system 160 in a distributed manner. Additionally, the speech recognition model 120A and / or the NLU model 130A can be stored locally on the client device 110 and / or remotely on a server that communicates with the client device 110 and / or the assisted phone call system 160 via the network 190.

[0029] In various embodiments, the speech recognition model 120A is an end-to-end speech recognition model such that the speech recognition engine 120A1 and / or 120A2 can use the model to directly generate the recognized text corresponding to the verbal input. For example, the speech recognition model 120A can be an end-to-end model for generating the recognized text on a character-by-character basis (or on some other token-by-token basis). A non-limiting example of such an end-to-end model for generating the recognized text on a character-by-character basis is the Recurrent Neural Network Transducer (RNN-T) model. The RNN-T model is a form of sequence-to-sequence model that does not employ an attention mechanism. In other embodiments, the speech recognition model 120A is not an end-to-end speech recognition model such that the speech recognition engine 120A1 and / or 120A2 can alternatively generate predicted phonemes (and / or other representations). For example, the predicted phonemes (and / or other representations) can then be used by the speech recognition engine 120A1 and / or 120A2 to determine the recognized text that matches the predicted phonemes. In doing so, the speech recognition engine 120A1 and / or 120A2 can optionally employ a decoding graph, a dictionary, and / or other resources.

[0030] In embodiments where the user input engine 111 detects a touch and / or typed input via the user interface input device of the client device 110, the automated assistant 115 can cause an indication of the touch input and / or an indication of the typed input to be transmitted from the client device 110 to the assisted telephone call system 160. In some versions of those embodiments, the indication of the touch input and / or the indication of the typed input can include the underlying text or representation of the touch input and / or the text of the typed input, and the speech recognition model 120A is an end-to-end speech recognition model, and the underlying text can be processed using the NLU model 130A to determine the underlying text and / or the intent of the text.

[0031] As described herein, the assisted telephone call engine 150 of the assisted telephone call system 160 can further process the recognized text generated by the speech recognition engines 120A1 and / or 120A2, the underlying text or representation of the touch input detected at the client device 110, the underlying text of the typed input detected at the client device 110, and / or the intent determined by the NLU engines 130A1 and / or 130A2. In various embodiments, the assisted telephone call engine 150 includes an entity recognition engine 151, an interactive voice response (IVR) engine 152, a parameter engine 153, a user interaction engine 154, a verification engine 155, a notification engine 156, and a task execution engine 157.

[0032] The entity recognition engine 151 can recognize an entity that the user of the client device 110 wants to contact. The entity can be, for example, a personal entity, a business entity, a location entity, and / or other entities. For example, a personal entity can be a friend entity, a family member entity, a co-worker entity, and / or other specific types of personal entities. In addition, a business entity can be a restaurant entity, an airline entity, a hotel entity, a salon entity, a doctor's office entity, and / or other specific types of business entities. In addition, a location entity can be a school entity, a museum entity, a library entity, a park entity, and / or other specific types of location entities. In some embodiments, the entity recognition engine 151 can also determine the specific entity of the recognized entity. For example, the specific entity of a personal entity can be the name of the individual (e.g., Jane Doe, John Doe, etc.), the specific entity of a business entity can be the name of the business (e.g., Hypothetical Airlines, Example Airlines, etc.), and the specific entity of a location entity can be the name of the location (e.g., Hypothetical University, Example National Park, etc.). Although the entities described herein can be defined at various levels of granularity, for simplicity, they are collectively referred to herein as "entities".

[0033] In some embodiments, the entity recognition engine 151 is capable of identifying an entity on behalf of a user of the client device 110 to be contacted based on a user interaction with the client device 110 before initiating an assisted telephone call using the automated assistant 115. In some versions of those embodiments, the entity can be identified in response to receiving user input for initiating an assisted telephone call. For example, if a user of the client device 110 directs an input (e.g., verbal or touch) to a call interface element of a software application (e.g., associated with a contact in a contacts application, associated with a search result in a browser application, and / or associated with other callable entities included in other software applications), the entity recognition engine 151 can identify the entity associated with the call interface element. For example, if the user input is directed to a call interface element in a browser application associated with "HypotheticalAirlines", the entity recognition engine 151 can identify "Hypothetical Airlines" (or more generally, a business entity or a restaurant entity) as the entity to be contacted on behalf of the user of the client device 110 during the assisted telephone call.

[0034] In addition, the entity recognition engine 151 is capable of identifying an entity from among a plurality of entities stored in the entity database 151A. In some embodiments, the plurality of entities stored in the entity database 151A can be indexed by an entity and / or a particular type of entity. For example, if the entity recognition engine 151 identifies the entity "HypotheticalAirlines", "HypotheticalAirlines" can be indexed in the entity database 151A as a business entity and can optionally be further indexed as an airline entity. By storing and indexing the identified entities in the entity database 151A, the entity recognition engine 151 can easily identify and retrieve entities, thereby reducing subsequent processing for identifying entities when encountering entities in future assisted telephone calls. In addition, in various embodiments, each entity can be associated with a task in the entity database 151A.

[0035] The IVR engine 152 is capable of identifying an IVR tree (and optionally, audio data associated with the IVR tree) associated with an entity identified via the entity recognition engine 151. In some embodiments, the IVR tree associated with the identified entity can be stored in the IVR tree database 152A in association with the identified entity. For example, the IVR tree stored in the IVR tree database 152A can be mapped to the corresponding entity stored in the entity database 151A. In some additional or alternative embodiments, the IVR tree can be stored in the entity database 151A in association with the corresponding entity, and the IVR tree database 152A can be omitted. Each IVR tree can include a plurality of corresponding candidate states, and the automated assistant 115 can determine the target state of a given IVR tree from among the plurality of corresponding candidate states based on user input detected via the user input engine 111. During an assisted phone call, the automated assistant can automatically navigate the IVR tree to the target state.

[0036] In some embodiments, and prior to receiving user input to initiate an assisted phone call, the automated assistant 115 (and / or additional instances of the automated assistant) can initiate an automated phone call to a plurality of entities stored in the entity database 151A. During these automated phone calls, the automated assistant 115 (and / or additional instances of the automated assistant) can navigate the IVR system associated with the entity to map out the corresponding IVR tree associated with each of the plurality of entities, and the mapping of the corresponding IVR tree can be stored in the IVR tree database 152A. In some versions of those embodiments, the automated assistant 115 (and / or additional instances of the automated assistant) can initiate multiple automated phone calls to a given entity among the plurality of entities to map variants of the given IVR tree associated with the given entity. For example, the automated assistant 115 can initiate a first automated phone call to a restaurant entity when the restaurant entity is open for business, and can initiate a second automated phone call to the restaurant entity when the restaurant is closed. In other words, the IVR engine 152 can enable the automated assistant 115 to map the given IVR tree associated with the restaurant entity in different contexts (e.g., the context associated with a given client device or the context associated with a user) to determine whether the given IVR tree is dynamic. Thus, when the IVR tree 152 identifies an IVR tree associated with an entity, it can identify the IVR tree that matches the current context of the client device 110 and / or the user of the client device 110 (e.g., as described with reference to Figure 7 and Figures 8A to 8D . In some additional or alternative embodiments, and prior to receiving user input to initiate an assisted phone call, the entity can provide a representation of the IVR tree and the IVR tree can be stored in the IVR tree database 152A.

[0037] In various embodiments, the automated assistant 115 can track changes in the IVR tree over time and optionally the degree of change based on an assisted phone call initiated in response to user input for initiating an assisted phone call. These changes can be identified based on the state difference between the stored IVR tree and the IVR tree currently being navigated by the automated assistant 115 and / or the state difference between the stored audio data associated with the IVR tree and the audio data associated with the IVR tree currently being navigated by the automated assistant 115. For example, if the automated assistant 115 determines that the IVR tree associated with a given entity and traversed during an assisted phone call is different from the IVR tree stored in the IVR tree database 152A (e.g., the currently navigated IVR tree includes a new introduction message, different internal nodes, or leading nodes, etc.), then the automated assistant 115 can store the different state of the currently navigated IVR tree together with the IVR tree that has been stored in the IVR tree database 152A associated with the given entity. Additionally, the inclusion of a new introduction message can serve as a signal to the automated assistant 115 that the IVR tree may include other downstream changes, and those changes are monitored rather than immediately attempting to navigate the IVR tree (if possible). As a result, the automated assistant 115 can... As another example, if the automated assistant 115 determines that the audio data received during navigation of the IVR is different from the audio data stored in association with the IVR tree (e.g., based on comparing the acoustic characteristics of the received audio data and the stored audio data and / or comparing the recognized text corresponding to the received audio data and the stored audio data), then the automated assistant 115 can store the different audio data of the currently navigated IVR tree together with the IVR tree that has been stored in the IVR tree database 152A associated with the given entity. If the difference between the stored IVR tree and the navigated IVR tree persists for a threshold number of assisted phone calls with a given entity, the different IVR tree can replace the IVR tree associated with the given entity. In some versions of those embodiments, if a different IVR tree is encountered in the same context (e.g., determined based on context information associated with the client device 110 and / or the user of the client device 110), then only the different IVR tree can be considered for replacing the IVR tree. In some versions of those embodiments, the confidence level associated with the IVR tree associated with a given entity can be based on the frequency of matching between the IVR tree traversed during an assisted phone call and the IVR tree stored in the IVR tree database 152A in association with the IVR tree. In some further versions of those embodiments, and as described with respect to Figures 6B to 6E the automated assistant 115 can render multiple candidate states in different ways based on the confidence level and whether the confidence level meets one or more thresholds.

[0038] In some embodiments, the parameter engine 153 can identify parameters associated with navigating an IVR tree identified via the IVR engine 152 based on the entities identified via the entity recognition engine 151. The parameters of the IVR tree can be stored in association with the IVR tree identified via the IVR engine 152 and can vary based on the navigation path for navigating the IVR tree to a target state. The automated assistant 115 can use the corresponding values of the parameters to navigate the IVR tree. In some embodiments, the corresponding values of the parameters can be stored in the user profile database 153A, and the automated assistant 115 can retrieve the corresponding values of the parameters from the user profile database 153A in response to identifying the parameters associated with navigating the IVR tree and without requesting any values from the user of the client device 110. In some additional or alternative versions of those embodiments, the automated assistant 115 can cause the corresponding values of the parameters to be requested from the user in response to identifying the parameters associated with navigating the IVR tree and in response to determining that the corresponding values cannot be retrieved from the user profile database 153A. For example, the automated assistant 115 can initiate a conversation with the user of the client device and cause one or more prompts to be rendered at the client device 110 via the rendering engine 113 to request the corresponding values of any parameters that cannot be resolved using information from the user profile database 153A.

[0039] The user profile database 153A can include information about a user from various sources, including, for example, electronic communications associated with the user of the client device 110 (e.g., e-mails, text messages, voice messages, etc.), user account information associated with various user accounts of the user of the client device 110 (e.g., account status information, account numbers, account usernames, account passwords, etc.), application information from applications installed on and / or accessible by the client device 110 (e.g., smart device applications, note applications, reminder applications, phone applications, calendar applications, and / or other software applications), payment information of the user of the client device 110 (e.g., bank account numbers, credit card numbers, payment application information, etc.), photo albums of the user of the client device 110, social media profiles of the user of the client device 110, user preferences of the user of the client device 110, personal information of the user of the client device 110 (e.g., name, phone number, physical address, e-mail address, social security number, date of birth, etc.), and / or other access-restricted information. For example, to navigate an IVR tree associated with an airline entity to a target state associated with canceling a flight reservation, the parameter engine 153 can identify the parameters for canceling a flight reservation and can determine the values of these parameters. For example, the flight confirmation number parameter can be identified and the flight confirmation number can be retrieved from a flight confirmation e-mail, the frequent flyer number parameter can be identified and the frequent flyer number of the user of the client device 110 associated with the airline whose flight is being canceled can be retrieved from an application associated with the airline. In contrast, to navigate an IVR tree associated with an airline entity to a different target state associated with changing a flight reservation, the parameter engine 153 can identify the above parameters and corresponding values, and can also identify the date or time parameter of the new flight and the destination parameter of the new flight, and can request the corresponding values of these parameters from the user of the client device 110.

[0040] For example, briefly referring to Figure 2 , an example IVR tree 200 is depicted. Figure 2 The IVR tree 200 depicted in Figure 2As shown, the IVR tree includes a root node 211, a plurality of internal nodes 221, 222, and 223, and a plurality of leaf nodes 231, 232, 233, 234, 235, and 236. The leaf nodes 231, 232, 233, 234, 235, and 236 can correspond to a plurality of candidate states, and a given one of the leaf nodes 231, 232, 233, 234, 235, and 236 can correspond to a target state. It is noted that the plurality of nodes are connected by a plurality of corresponding edges. These nodes and / or edges can be stored in the IVR tree database 152A as the IVR tree (and optionally together with the audio data associated with each node and / or edge). Each of the corresponding edges can be traversed based on an input generated by the automated assistant 115 during an assisted telephone call. For example, a corresponding edge can be traversed based on an input corresponding to a digit indicator that can be simulated as a button press by the automated assistant 115 (e.g., pressing "1" to traverse from the root node 211 to the internal node 1 221, pressing "2" to traverse from the root node 211 to the internal node 2222, pressing "3" to traverse from the root node 211 to the leaf node 1 231, etc., as indicated by the edges in Figure 2 ), or a corresponding edge can be traversed based on an input corresponding to a free-form natural language input (e.g., saying "book a flight" to traverse from the root node 211 to the internal node 1 221, etc.).

[0041] In some embodiments, the free-form natural language input can be synthetic speech audio data generated by the automated assistant 115 that includes synthetic speech to be rendered at an additional client device to traverse along the edges of the IVR tree. For example, the automated assistant can generate synthetic speech audio data that includes an indication of the digit indicator described above and / or a brief overview of why the assisted telephone call is being made. In some additional or alternative embodiments, the free-form natural language input can be audio data that captures the verbal input of a user of the client device 110 and is provided before and / or during the initiation of the assisted telephone call. For example, if the IVR tree requires the user to provide a verbal utterance to verify the identity of the user of the client device 110 (e.g., using speech recognition), then the audio data can capture the verbal input of the user.

[0042] In various embodiments, the automated assistant 115 is able to track the navigation of the IVR tree 200. By tracking the navigation of the IVR tree 200, the automated assistant 115 can determine when to provide input (e.g., synthesized speech, simulated button presses, and / or audio data) for navigating the IVR tree 200. For example, the automated assistant 115 can track the initiation of an assisted phone call starting from the root node 211 and begin traversing along the edges of the IVR tree 200 to internal nodes 221, 222, and 223 and / or leaf nodes 231, 232, 233, 234, 235, and 236 to navigate the IVR tree 200 to a target state determined based on user input detected at the client device 110. As the automated assistant 115 traverses along the edges to different nodes to reach the target state, the automated assistant 115 can track the traversed edges and / or nodes. In this example, if the IVR tree 200 includes an introductory message before presenting the main menu associated with the root node 211, the automated assistant 115 can determine that the introductory message is new and suppress any input for navigating the IVR tree until the main menu associated with the root node 211 is presented (or any other change is presented). As another example, the automated assistant 115 can determine that no input that is to be provided for navigating the IVR tree to the target state can be provided until the IVR system presents all options for navigating the IVR tree (e.g., the IVR tree does not support providing input at any time). In this example, the automated assistant 115 can suppress any input for navigating the IVR tree until all options for navigating the IVR tree are presented. In some embodiments, the automated assistant 115 can return to a previous node traversed during the navigation of the IVR tree 200.

[0043] As a non - limiting example, assume that the IVR tree 200 corresponds to an IVR tree associated with a hypothetical entity - Example Airlines. In this example, the root node 211 may correspond to the main menu of the IVR system associated with Example Airlines, the internal nodes may correspond to intermediate states 221, 222, and 223, and the leaf nodes 231, 232, 233, 234, 235, and 236 may correspond to the target states of the IVR tree. Further assume that leaf node 1 231 is associated with the target state of "Speak to a representative", leaf node 2 232 is associated with the target state of "Upcoming flight information", leaf node 3 233 is associated with the target state of "Report lost baggage", leaf node 4 234 is associated with the target state of "Book a flight", leaf node 5 235 is associated with the target state of "Change flight reservation", and leaf node 6 236 is associated with the target state of "Cancel flight reservation". In some embodiments, the target state to which the automated assistant 115 will navigate the IVR tree during an assisted phone call can be determined based on user input detected via the user input engine 111 of the client device 110 before and / or during the assisted phone call.

[0044] For example, assume that user input indicating that the user wants to report lost baggage is received at the client device 110. In this example, the automated assistant 115 can determine to navigate to leaf node 3 233 associated with the target state of "Report lost baggage". In this example, the automated assistant 115 may know to traverse from the root node 211 to internal node 1 221, and from internal node 1 221 to leaf node 3 233 to report lost baggage because the IVR tree 200 was previously mapped by the automated assistant 115 (and / or an additional automated assistant) as described above with respect to the IVR engine 152. When navigating from the root node 211 to leaf node 3 233 and / or when reaching leaf node 3 233, the automated assistant 115 can cause the corresponding values of the parameters requested when navigating the IVR tree 200 to be rendered at an additional client device associated with the entity. In this example, the automated assistant 115 can render the corresponding values of flight information associated with the most recent flight (e.g., flight confirmation number, departure destination, arrival destination), the corresponding values of the personal information of the user of the client device 110 (e.g., name and phone number), and / or other corresponding values of the parameters in response to the corresponding values of the parameters being requested at an additional client device associated with the entity. When reaching leaf node 3 233, the automated assistant 115 can optionally (e.g., via the notification engine 156) generate a notification to be rendered (e.g., via the rendering engine 113) at the client device 110 that requests the user to join the assisted phone call and / or reminds the user of the client device 110 of the result of reporting lost baggage.

[0045] As another example, assume that user input indicating that the user wants to cancel a lost reservation is received at the client device 110. In this example, the automated assistant 115 is able to determine to navigate to the leaf node 6 236 associated with the target state of "report lost luggage". In this example, the automated assistant 115 may know to traverse from the root node 211 to the internal node 1 221, from the internal node 1 221 to the internal node 3 223, and from the internal node 3 223 to the leaf node 6 236 to cancel the flight reservation because the IVR tree 200 was previously mapped by the automated assistant 115 (and / or an additional automated assistant), as described above with respect to the IVR engine 152. When navigating from the root node 211 to the leaf node 6 236 and / or when reaching the leaf node 6 236, the automated assistant 115 is able to cause the corresponding values of the parameters requested when navigating the IVR tree 200 to be rendered at an additional client device associated with the entity. In this example, the automated assistant 115 is able to render the corresponding values of flight information associated with the upcoming flight (e.g., flight confirmation number, departure destination, arrival destination), account information associated with the Example Airlines account (e.g., frequent flyer number), and / or other corresponding values of the parameters at an additional client device associated with the entity in response to the corresponding values of the parameters being requested. When reaching the leaf node 6 236, the automated assistant 115 is able to optionally (e.g., via the notification engine 156) generate a notification to be rendered (e.g., via the rendering engine 113) at the client device 110 that requests the user to join an assisted phone call and / or reminds the user of the client device 110 of the result of canceling the flight.

[0046] Although Figure 2 the IVR tree 200 is depicted as having specific nodes configured in a particular manner based on the corresponding edges, it should be understood that this is for example purposes only and is not meant to be limiting, and it should be understood that the IVR tree can be configured in almost infinitely different ways. Additionally, based on certain contexts, some nodes of the IVR tree 200 may be available or may not be available to the user of the client device 110 (e.g., as described with respect to Figure 7 and Figures 8A to 8D ). For example, assume that a given IVR tree is associated with a restaurant entity. In this example, the target state associated with placing a takeout order may be available when the restaurant entity is open. Conversely, the target state associated with placing a takeout order may not be available when the restaurant entity is closed.

[0047] Return reference Figure 1, the user interaction engine 154 is capable of detecting user interactions at the client device 110. User interactions can include, for example, search interactions with a web browser or web browser application, browsing interactions with a web browser or web browser application, navigation interactions with a web browser or navigation application, assistant interactions with the automated assistant 115, and / or other user interactions. In some embodiments, the user interaction engine 154 can detect user interactions of a user of the client device 110 at the client device 110 or at an additional client device that communicates with the client device 110 (e.g., via the network 190) after the automated assistant 115 initiates the execution of an assisted phone call. In some versions of those embodiments, the automated assistant 115 can determine whether the detected user interaction is associated with an entity that engages with the user of the client device 110 during the assisted phone call. If the automated assistant 115 determines that the detected user interaction is associated with the entity, the automated assistant 115 can generate one or more prompts (e.g., via the notification engine 156) asking the user of the client device 110 whether the navigation of the IVR tree should be modified. In some further versions of those embodiments, modifying the navigation of the IVR tree can include navigating to a different target state. For example, if the automated assistant 115 initiates an assisted phone call to navigate an IVR tree associated with a home improvement entity to a target state associated with the painting department of the home improvement entity, and the user of the client device 110 subsequently interacts with a web page associated with the home and garden department of the home improvement entity, the automated assistant 115 can generate one or more prompts asking the user whether to additionally or alternatively navigate to a target state associated with the home and garden department. In other further versions of those embodiments, modifying the navigation of the IVR tree can include navigating to the same IVR tree with different parameters. For example, if the automated assistant 115 initiates an assisted phone call to navigate an IVR tree associated with a home improvement entity to a target state associated with the painting department of the home improvement entity, and the user device 110 of the client subsequently interacts with a web page associated with a specific paint manufacturer, the automated assistant 115 can generate one or more prompts asking the user whether to use the specific paint manufacturer as the corresponding value of a parameter when navigating to the target state associated with the painting department. Modifying the navigation of the IVR tree is described in detail below (e.g., with reference to Figure 3 and Figures 4A to 4C ).

[0048] In some additional or alternative embodiments, the user interaction engine 154 is capable of detecting user interaction by a user of the client device 110 at the client device 110 or at an additional client device communicating with the client device 110 (e.g., via the network 190) before the automated assistant 115 initiates execution of an assisted telephone call. In some versions of those embodiments, the automated assistant 115 may identify an entity associated with the user interaction and, if an assisted telephone call is subsequently initiated with the identified entity, may cause a candidate state of the IVR tree to be rendered based on the user interaction. For example, if the user is browsing a web page associated with a hypothetical airline entity, Example Airlines, to research how to report lost luggage before initiating an assisted telephone call with Example Airlines, the automated assistant 115 may (alone or more prominently than other selectable elements) render a selectable graphical element associated with a target state associated with reporting lost luggage in the IVR tree associated with Example Airlines. In other words, based on the user researching how to report lost luggage before initiating an assisted telephone call, the automated assistant 115 is capable of causing a selectable graphical element to be rendered as an “IVR deep link” to the target state of the IVR tree (e.g., as described with respect to Figure 5 and Figure 6A ).

[0049] In some embodiments, the verification engine 155 can verify whether the state that the automated assistant 115 navigates to during an assisted phone call is actually the target state. In some versions of these embodiments, when a human representative associated with the entity joins the assisted phone call, the automated assistant 115 can process the received audio data that captures the verbal input of the human representative to verify that the state that the automated assistant 115 navigates to is actually the target state. For example, assume that the automated assistant 115 is navigating an IVR tree associated with a home improvement entity to a target state associated with the painting department of the home improvement entity. Further assume that a human representative takes over the assisted phone call from the IVR system associated with the home improvement entity and provides the verbal input "Thank you for calling the painting department. How can I help you today?". In this example, the verification engine 155 can cause the automated assistant 115 to process the audio data that captures the verbal input using the speech recognition engines 120A1 and / or 120A2, and can determine based on processing the audio data that captures the verbal input that the verbal input includes "painting department". In some additional or alternative versions of these embodiments, a verbal input can be received in response to synthetic speech audio data that includes synthetic speech, to verify the target state of navigating the IVR tree. Continuing with the above example, assume that a human representative takes over the assisted phone call from the IVR system associated with the home improvement entity and provides the verbal input "Hello". In this example, the verification engine 155 can cause the automated assistant 115 to render the synthetic speech "Hello, is this the painting department?" at an additional client device associated with the human representative. The verification engine 155 can cause the automated assistant 115 to process the audio data that captures the verbal input in response to the synthetic speech using the speech recognition engines 120A1 and / or 120A2, and can determine whether the verbal input verifies reaching the target state (e.g., "yes", "this is the painting department", or some other confirmation).

[0050] In various embodiments, if the state reached when navigating the IVR does not correspond to the target state, the verification engine 155 can cause the automated assistant 115 to render at an additional client device associated with the human representative a synthetic speech requesting to transfer the assisted phone call to the target state. Continuing with the above example, if it is determined that the verbal input indicates that the automated assistant 115 has reached a state associated with the home and garden department (instead of the painting department), the verification engine 155 can cause the automated assistant 115 to render at an additional client device associated with the human representative the synthetic speech "Can you transfer me to the painting department?", and can verify again that the state after the transfer corresponds to the desired target state of the user of the client device 110.

[0051] The notification engine 156 can generate various notifications based on the navigation of the IVR tree associated with the identified entity and / or based on tasks performed by the automated assistant 115 on behalf of the user of the client device 110 during an assisted phone call without pausing the assisted phone call. For example, the automated assistant 115 can cause the notification engine 156 to generate a notification requesting the user of the client device 110 to join the assisted phone call when a target state is reached. In some embodiments, the notification engine 156 may cause the automated assistant 115 to generate a notification requesting the user of the client device 110 to join the assisted phone call only after the verification engine 155, as described above with respect to the verification engine 155, verifies that the automated assistant has actually reached the target state. As another example, if the automated assistant 115 cannot resolve the corresponding value of a parameter during an assisted phone call, the automated assistant 115 can cause the notification engine 156 to generate one or more prompts to request the corresponding value of any unresolved parameter required to continue navigating the IVR tree associated with the entity. As yet another example, if the automated assistant performs a task on behalf of the user of the client device 110 during an assisted phone call, the automated assistant 115 can cause the notification engine 156 to generate a notification including the result of the execution of the task.

[0052] In some embodiments, the type of the notification generated by the notification engine 157 and rendered at the client device 110 via the rendering engine 113 and / or one or more attributes of the rendered notification (e.g., volume, brightness, size) can be based on the state of the client device 110 and / or the state of an ongoing secondary phone call (e.g., determined using the device state engine 112). The state of the ongoing secondary phone call can be based on, for example, software applications running in the foreground of the client device 110, software applications running in the background of the client device 110, whether the client device 110 is in a locked state, whether the client device 110 is in a sleep state, whether the client device 110 is in an off state, sensor data generated by sensors of the client device 110, and / or other data associated with the client device 110. For example, if the state of the client device 110 indicates that a software application that displays a transcription of the secondary phone call (e.g., an automated assistant application, a phone application, a secondary phone call application, and / or other software applications) is running in the foreground of the client device 110, the type of the notification can be a banner notification, a pop-up notification, and / or other types of visual notifications. As another example, if the state of the client device 110 indicates that the client device 110 is in a sleep or locked state, the type of the notification can be an audible indication via the speaker and / or a vibration via the speaker or other hardware components of the client device 110 that mimic a phone call. As yet another example, if sensor data from a presence sensor, an accelerometer, and / or other sensors of the client device indicates that the user is not currently nearby and / or currently holding the client device, a more intrusive notification (e.g., visual and audible at a first volume level) can be provided. On the other hand, if such sensor data indicates that the user is currently nearby and / or currently holding the client device, a less intrusive notification (e.g., only visual, or visual and audible at a second volume level that is less than the first volume level) can be provided. As yet another example, when the state of the conversation indicates that the conversation is approaching completion, a more intrusive notification can be provided, and when the state of the conversation indicates that the conversation is not approaching completion, a less intrusive notification can be provided.

[0053] In addition, in implementations where the automated assistant 115 causes a notification requesting the user to join an assisted phone call to be rendered at the client device 110 but the user does not join the assisted phone call within a threshold duration (e.g., 15 seconds, 30 seconds, 60 seconds, and / or other durations), the automated assistant is able to cause the assisted phone call to terminate. In some versions of these implementations, the threshold duration can start when the notification requesting the user to join the assisted phone call is rendered on the client device 110. In some additional or alternative versions of those implementations, and before terminating the assisted phone call, the state of the assisted phone call can be stored in one or more databases (e.g., the random access memory of the client device) of the client device 110. If the user of the client device 110 subsequently interacts with the notification to join the terminated assisted phone call, the automated assistant 115 can re-initiate the execution of the assisted phone call by loading the stored state of the assisted phone call, such that the assisted phone call resumes in a given state that corresponds to the stored state of the assisted phone call. In this way, the automated assistant 115 does not need to navigate parts of the IVR tree that have already been navigated by the automated assistant 115.

[0054] In various implementations, and after the automated assistant 115 has navigated to a target state of the IVR tree associated with an entity, the automated assistant is able to perform a task on behalf of the user of the client device 110. In some versions of these implementations, the task engine 157 can determine the task to be performed on behalf of the client device 110 user based on user input detected at the client device 110 for initiating an assisted phone call or based on other user input detected at the client device 110 before the automated assistant 115 initiates the assisted phone call. For example, if the user of the client device 110 provides an oral input of "Call Example Café for a reservation tonight", the task engine 157 can utilize the intent to initiate a call and make a restaurant reservation (e.g., determined using the NLU model 130A) to determine the task of making a restaurant reservation based on the oral input. The parameters described herein determined by the parameter engine 153 can also identify the parameters of the task identified by the task engine 157 and determine the corresponding values of the parameters of the task in a similar manner as described above for the parameter engine 153.

[0055] The task engine 157 can enable the automated assistant 115 to participate in a conversation with a human representative associated with the identified entity during an assisted phone call to perform a task. For example, the task engine 157 can provide text and / or phonemes based on corresponding values of parameters to the text-to-speech engines 140A1 and / or 140A2 to generate synthesized speech audio data. The synthesized speech audio data can be transmitted to an additional client device of the human representative for audible rendering at the additional client device. For example, the text-to-speech engines 140A1 and / or 140A2 can determine a sequence of phonemes determined to correspond to the value of a parameter requested by the human representative and can use the text-to-speech model 140a to process the sequence of phonemes to generate synthesized speech audio data. The synthesized speech audio data can be, for example, in the form of an audio waveform. When determining the sequence of phonemes corresponding to the value of a parameter, the text-to-speech engines 140A1 and / or 140A2 can access a token-to-phoneme mapping stored locally on the client device 110 or (e.g., via the network 190) at the server. The task engine 157 can enable the speech recognition engines 120A1 and / or 120A2 to process any audio data that captures oral input provided by the human representative, and then the automated assistant 115 can use the text-to-speech engines 140A1 and / or 140A2 to generate further synthesized speech audio data.

[0056] As described herein, the rendering engine 113 can render various notifications or other outputs at the client device 110. The rendering engine 113 can render various notifications described herein audibly and / or visually. In addition, the rendering engine 113 can enable the transcription of a conversation to be rendered on the user interface of the client device 110. In some embodiments, the transcription can correspond to a conversation between the user of the client device 110 and the automated assistant 115. In some additional or alternative embodiments, the transcription can correspond to a conversation between an IVR system associated with the identified entity and the automated assistant 115. In other embodiments, the transcription can correspond to a conversation between a human representative associated with the identified entity and the automated assistant 115.

[0057] In some embodiments, the user input detected at the client device 110 via the user input engine 111 can be a search query received via the search interface of the client device 110. For example, the search interface can be implemented as part of a phone or contacts application, a browser application, an automated assistant application, a web-based browser, and / or any other interface enabled with search functionality. The search engine 114 can identify and retrieve search results rendered at the client device 110 via the rendering engine 113. In some versions of these embodiments, the search results can include content responsive to the search query and can optionally include one or more selectable elements. The one or more selectable elements can include, for example, deep links to one or more candidate states of an IVR tree associated with an entity, and when selected based on additional user input (e.g., touch, typing, and / or verbal), cause the automated assistant 115 to navigate to a target state of the IVR tree corresponding to the selected candidate state of the IVR tree included in the search results. For example, if a user of the client device 110 submits a search query of "home improvement store" via the search interface of the phone application, selectable elements associated with various departments corresponding to one or more candidate states of an IVR tree associated with the home improvement store can be included in the search results responsive to the search query. Additionally, if the user has recently interacted with other search results associated with "paint" (e.g., as described above with respect to the user interaction engine 154), a particular selectable element (e.g., "painting department") can be rendered more prominently than other selectable elements.

[0058] By using the techniques described herein, various technical advantages can be achieved. As a non-limiting example, the automated assistant 115 can end an assisted phone call more quickly because the automated assistant does not need to hear the options of the IVR system before navigating the IVR tree as it is stored associated with the entity. Network and computing resources can be saved because the duration of the assisted phone call can be reduced by using the techniques disclosed herein. Additionally, the number of user inputs can be reduced, thereby conserving client device resources because a given user only needs to select an "IVR deep link" in various embodiments to initiate the execution of an assisted phone call. Also, the battery resources of the client device 110 can be conserved because the user is not an active participant in the assisted phone call and the client device 110 can be placed in a locked or low-power state during the assisted phone call.

[0059] Now turning to Figure 3 , a flowchart depicting an example method 300 for modifying navigation of an IVR tree based on user interactions at a client device associated with a given user is shown. For convenience, the operations of method 300 are described with reference to the system performing the operations. The system of method 300 includes a computing device (e.g.,Figure 1 , Figures 4A to 4C , Figures 6A to 6E and / or Figures 8A to 8D client device 110 and / or Figure 9 one or more processors and / or other components of computing device 910, one or more servers, and / or other computing devices). Further, although the operations of method 300 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, and / or added.

[0060] At block 352, the system receives, from a given user via a client device associated with the given user, user input for initiating an assisted phone call. The user input for initiating an assisted phone call can be one or more of a verbal input for initiating an assisted phone call, a touch input for initiating an assisted phone call, or a typed input for initiating an assisted phone call. Further, the user input for initiating an assisted phone call can be received at various interfaces (e.g., a search interface, a voice interface, an automated assistant interface, and / or other interfaces from which an assisted phone call can be initiated).

[0061] At block 354, the system identifies an entity to be contacted on behalf of the given user during the assisted phone call based on the user input. The system can identify the entity to be contacted during the assisted phone call based on the user input received at block 352 and / or user interactions with the client device immediately prior to initiating the assisted phone call. For example, the system can identify the entity "Example Airlines" based on processing the verbal input "Report my lost luggage to Example Airlines" received at the client device. As another example, the system can identify the entity "Example Airlines" based on determining that a given user of the client device selected a call interface element associated with Example Airlines or a particular graphical element associated with a candidate state of an IVR tree associated with Example Airlines.

[0062] At block 356, the system identifies an IVR tree associated with the identified entity, the IVR tree including a plurality of candidate states. The IVR tree associated with the identified entity can be stored in association with the entity identified at block 354 (e.g., in Figure 1 entity database 151A and / or in Figure 1in the IVR tree database 152A). The stored IVR tree associated with an entity may have been previously stored in one or more databases based on the system (and / or other instances of the system) previously crawling an IVR system associated with the identified entity, and / or the identified entity may be able to provide an IVR tree for the IVR system associated with the identified entity. Additionally, the system is able to continuously update the IVR tree of the IVR system associated with the identified entity based on multiple assisted phone calls navigating the IVR tree. Multiple candidate states may correspond to specific nodes of the IVR tree that the system is able to navigate to during an assisted phone call.

[0063] At block 358, the system determines a target state of the IVR tree from among the multiple candidate states based on the user input or additional user input. In some embodiments, the target state may be determined based on the user input received at block 352. For example, assume that the IVR tree is associated with the entity of Example Airlines, and assume that a given candidate state of the IVR corresponds to the "Report lost luggage" candidate state of the IVR tree associated with the entity of Example Airlines. Further assume that the verbal input "Report my lost luggage to Example Airlines" is received at the client device at block 352. In this example, the target state of "Report lost luggage" may be identified based on the verbal input. In some additional or alternative embodiments, the target state may be determined based on additional user input other than the user input received at block 352. For example, assume that the IVR tree is associated with the entity of Example Airlines, and assume that a given candidate state of the IVR corresponds to the "Lost luggage" candidate state of the IVR tree associated with the entity of Example Airlines. Further assume that the verbal input "Call Example Airlines" is received at the client device at block 352, or a selection of an alternative graphical element associated with calling Example Airlines is received at the client device at block 352. In this example, one or more of the multiple candidate states may be presented as an output at the client device in response to receiving the user input, and the target state may be the target state of "Report lost luggage" based on a selection from among the one or more presented multiple candidate states. The output may be presented before initiating the assisted phone call or at the time of initiating the assisted phone call.

[0064] At block 360, the system initiates the execution of an assisted phone call to navigate the IVR tree to a target state. The system is able to initiate the execution of the assisted phone call by establishing a communication session between the client device associated with a given user and an additional client device associated with the entity identified at block 354. The communication session can be established using various voice communication protocols including, for example, VoIP, PSTN, and / or other telephone communication protocols. In some embodiments, the system is able to navigate the IVR tree to the target state by simulating button presses to traverse the nodes of the IVR tree, by rendering synthetic speech audio data including synthetic speech to traverse the nodes of the IVR tree, and / or by rendering audio data including the verbal input of the given user associated with the client device (e.g., as described with respect to Figure 2 ). In some versions of those embodiments, the system is also able to utilize the corresponding values of the parameters associated with navigating the IVR tree. The corresponding values can be retrieved from one or more databases (e.g., Figure 1 the user profile database 153A) and / or can be retrieved from the corresponding values of the given user request parameters associated with the client device.

[0065] At block 362, the system determines whether a user interaction is detected at the client device. User interactions can include, for example, a search interaction with a web browser or web browser application, a browsing interaction with a web browser or web browser application, a navigation interaction with a web browser or navigation application, an assistant interaction with the automated assistant 115, and / or other user interactions with the client device. Notably, when the system executes the assisted phone call, the given user associated with the client device can still interact with the client device such that the assisted phone call is executed in the background of the client device. If at an iteration of block 362 the system determines that no user interaction is detected at the client device, the system can proceed to block 370. Block 370 is described in detail below. If at an iteration of block 362 the system determines that a user interaction is detected at the client device, the system can proceed to block 364.

[0066] At block 364, the system determines whether the user interaction detected at block 362 is associated with the entity identified at block 354. For example, the system is able to determine whether the user interaction is directed to a web page associated with the entity identified at block 354, is directed to obtaining directions to a physical store associated with the entity identified at block 354, and / or other user interactions directed to the entity. If at an iteration of block 364 the system determines that the user interaction detected at block 362 is not associated with the entity identified at block 354, the system can proceed to block 370. Block 370 is described in detail below. If at an iteration of block 364 the system determines that the user interaction detected at block 362 is associated with the entity identified at block 354, the system can proceed to block 366.

[0067] At block 366, the system generates one or more prompts related to modifying the navigation of the IVR tree based on the user interaction detected at block 362. In some embodiments, modifying the navigation of the IVR tree can include modifying the target state for navigating the IVR tree to an alternative target state. For example, assume an assisted phone call is initiated to navigate to a target state associated with the flooring department of an IVR tree associated with a home improvement entity, and further assume a user interaction is detected that points to a portion of a web page associated with the lighting department of the home improvement entity. In this example, the system can prompt a given user to change the target state from being associated with the flooring department to an alternative target state associated with the lighting department.

[0068] In some additional or alternative embodiments, modifying the navigation of the IVR tree can include adding an additional target state for navigating the IVR tree. Continuing the above example, instead of prompting a given user to change the target state from being associated with the flooring department to an alternative target state associated with the lighting department, the system can prompt a given user to add the lighting department as an additional target state in addition to the target state associated with the flooring department. In this example, the user can be notified to join the assisted phone call when reaching the target state associated with the flooring department. After the given user interacts with the flooring department, the given user can terminate the assisted phone call. However, the system can keep the assisted phone call connected and can request a human representative associated with the flooring department to transfer the assisted phone call to the lighting department, or the system can re-navigate the IVR tree to the additional target state associated with the lighting department.

[0069] In some additional or alternative embodiments, modifying the navigation of the IVR tree can include modifying the corresponding values of the parameters for navigating the IVR tree. For example, assume an assisted phone call is initiated to navigate to a target state associated with the lighting department of an IVR tree associated with a home improvement entity, and further assume a user interaction is detected that points to a portion of a web page associated with a specific brand of smart bulbs sold at the home improvement entity. In this example, the system can prompt a given user to include the corresponding values associated with the specific brand of smart bulbs for the parameters associated with navigating to the target state associated with the lighting department. Additionally, the system can cause one or more prompts to be visually and / or audibly rendered at the client device associated with the given user.

[0070] At block 368, the system determines whether to modify the navigation of the IVR tree based on additional user input received in response to one or more prompts. For example, if the user affirmatively responds to one or more prompts rendered at the client device, the navigation of the IVR tree can be modified based on the user interaction detected at block 362. However, if the user does not affirmatively respond (or negatively responds) to one or more prompts, the navigation of the IVR tree may not be modified. If at an iteration of block 368 the system determines based on the additional user input not to modify the navigation of the IVR tree, the system can proceed to block 370.

[0071] At block 370, the system navigates the IVR tree to a target state. In embodiments where the system advances from blocks 362, 364, and / or 368 to block 370, the system can continue to navigate the IVR tree based on the target state determined at block 358. The system can navigate the IVR tree based on simulated button presses to navigate the IVR tree such that synthetic speech audio data including synthetic speech is rendered at an additional client device associated with the entity identified at block 354, and / or such that audio data including verbal input is rendered at an additional client device associated with the entity identified at block 354. If at an iteration of block 368 the system determines based on the additional user input to modify the navigation of the IVR tree, the system can proceed to block 372. At block 372, the system modifies the corresponding value of a parameter used to navigate the IVR tree or modifies the target state to additionally include an additional target state or alternatively include an alternative target state, as described above with respect to block 366. As described above with respect to Figure 2 navigating the IVR tree.

[0072] From block 370 or block 372, the system can proceed to block 374. At block 374, the system can verify whether the assisted phone call has reached a target state. The system can process verbal input received at the client device from a human representative associated with the entity who has joined the call or an additional automated assistant of the IVR system associated with the entity to determine whether the system has reached the target state. For example, assume that the target state of the IVR tree is associated with the lighting department of a home improvement store entity. Further assume that the human representative or additional automated assistant provides the verbal input "Thank you for calling the lighting department. How can I help you?" In this example, the system can determine that the verbal input includes the term "lighting department" to verify that the system has actually reached the desired target state for a given user associated with the client device. In some embodiments, the verbal input received at the client device can be provided in response to a synthesized voice request for verbal input rendered at an additional client device associated with the human representative or additional automated assistant (e.g., "Hello, is this the lighting department?" or other synthesized voice requesting verification of the target state). If at an iteration of block 374 the system determines that the assisted phone call has reached a state other than the target state, the system can proceed to block 376. At block 376, the system requests to transfer the assisted phone call to a representative associated with the target state. The request to transfer the assisted phone call can be included in a synthesized voice (e.g., "Can you transfer me to the lighting department?" or other synthesized voice requesting transfer of the assisted phone call) rendered at an additional client device associated with the human representative or additional automated assistant. The system can then return to block 374 to verify whether the representative associated with the target state corresponds to the target state after the transfer. If at an iteration of block 374 the system determines that the assisted phone call has reached the target state, the system can proceed to block 378.

[0073] At block 378, the system renders a notification via the client device indicating the result of the execution of navigating the IVR tree. For example, the notification can include a request for a given user associated with the client device to join the assisted phone call. In the embodiments described herein where the system also represents a given user associated with the client device and performs tasks regarding the entity, the notification can additionally or alternatively include the result of the execution of the task.

[0074] Now referring to Figures 4A to 4C , various non-limiting examples of user interfaces associated with modifying the navigation of an IVR tree based on user interactions at a client device 110 associated with a given user are depicted. Figures 4A to 4C Each depicts a client device 110 having a graphical user interface 180 and can include Figure 1 one or more components of the client device. The automated assistant associated with the client device 110 (e.g.,Figure 1 One or more aspects of the automated assistant 115) can be implemented locally on the client device 110 and / or in a distributed manner (e.g., via Figure 1 the network 190) on other client devices that are in network communication with the client device 110. For simplicity, Figures 4A to 4C the operations are described herein as being performed by the automated assistant. Although Figures 4A to 4C the client device 110 is depicted as a mobile phone, it should be understood that this is not intended to be limiting. The client device 110 can be, for example, a stand-alone assistant device (e.g., having a speaker and / or a display), a laptop computer, a desktop computer, an in-vehicle computing device, and / or any other client device capable of making a telephone call.

[0075] Figures 4A to 4C The graphical user interface 180 also includes: a text response interface element 184 through which a user can select to generate a user input in the form of a text response via a virtual keyboard or other touch and / or typing input; and a voice response interface element 185 through which a user can select to generate a user input in the form of a voice response via the microphone of the client device 110. In some embodiments, a user can generate a user input via the microphone without selecting the voice response interface element 185. For example, active monitoring of audible user input via the microphone can be performed to obviate the need for the user to select the voice response interface element 185. In some and / or other embodiments of these, the voice response interface element 185 can be omitted. Additionally, in some embodiments, the text response interface element 184 can be additionally and / or alternatively omitted (e.g., the user can provide only audible user input). Figures 4A to 4C The graphical user interface 180 also includes system interface elements 181, 182, 183 with which a user can interact to cause the computing device 110 to perform one or more actions.

[0076] In various embodiments described herein, a user input can be received to initiate a telephone call with an entity (e.g., an assisted telephone call) using the automated assistant. The user input can be an oral input, a touch input, and / or a typing input that includes an indication for initiating the assisted telephone call. Additionally, the automated assistant can navigate an IVR tree associated with the entity to a target state from among multiple candidate states on behalf of a given user of the client device 110, and / or perform tasks regarding the entity. As Figure 4AAs shown, the user interface 180 includes search results for grocery store entities from a browser application accessible at the client device 110 (e.g., as indicated by the URL 411 of "www.exampleurl0.com / "). Additionally, the search results include a first search result 420 for "Hypothetical Grocer" located at a first position and a second search result 430 for "Hypothetical Grocer" located at a second position.

[0077] In some embodiments, the search results 420 and / or 430 can be associated with various selectable graphical elements that, when selected, cause the client device 110 to perform corresponding actions. For example, when a call graphical element 421 and / or 431 associated with a given one of the search results 420 and / or 430 is selected, the user input can indicate that a phone call action should be performed to the grocery store entity associated with the search results 430 and 420. As another example, when a directions graphical element 422 and / or 432 associated with a given one of the search results 420 and / or 430 is selected, the user input can indicate that a navigation action should be performed to the grocery store entity associated with the search results 430 and 420. As another example, when a departments graphical element 423 and / or 433 associated with a given one of the search results 420 and / or 430 is selected, the user input can indicate that a browser-based operation should be performed to display the departments of the grocer entity associated with the search results 420 and / or 430. Although the assisted phone call is Figure 4A initiated from the browser application, it should be understood that this is for illustrative purposes only and is not intended to be limiting. For example, if the assisted phone call is initiated using voice input from the home screen of the client device 110, from the lock screen of the client device 110, and / or other states of the client device 110, the assisted phone call can be initiated from various software applications accessible at the client device 110 (e.g., an automated assistant application, a phone application, a contacts application, an email application, a text or SMS messaging application, and / or other software applications).

[0078] As an example, assume that at the client device 110, a user input is detected to initiate a phone call using the first search result 420 for "Hypothetical Grocer" at the first location. For example, the user input can be an oral input of "Call Hypothetical Grocer" (and selected based on the proximity of the first location to the client device 110) or a touch input pointing to the call graphical element 421, as indicated by 490A1. In some embodiments, the call details interface 470 can be rendered at the client device 110 before receiving the user input for initiating a phone call to "Hypothetical Grocer", or can be rendered at the client device 110 in response to receiving the user input for initiating a phone call to "Hypothetical Grocer" (as indicated by 490A1). In some versions of those embodiments, the call details interface 470 can be rendered at the client device 110 as part of the user interface 180. In some other versions of those embodiments, the call details interface 470 can be an interface separate from the user interface 180 that overlays the user interface 180, and can include a call details interface element 186 that allows the user to expand the call details interface 470 to display additional call details (e.g., by swiping up on the call details interface element 186) and / or close the call details interface 470 (e.g., by swiping down on the call details interface element 186). Although the call details interface 470 is depicted as being located at the bottom of the user interface 180, it should be understood that this is for example purposes only and is not intended to be limiting. For example, the call details interface 470 can be rendered at the top of the user interface 180, on one side of the user interface 180, or as an interface completely separate from the user interface 180.

[0079] In various embodiments, the call details interface 470 can include a plurality of graphical elements. In some versions of these embodiments, the graphical elements can be selectable such that when a given one of the graphical elements is selected, the client device 110 can perform a corresponding action. As Figure 4AAs shown, the call details interface 470 includes a first graphical element 471A of "Assisted Call, Hypothetical Grocer, First Location", a second graphical element 472A of "Regular Call", and a third graphical element 473A of "Save Contact ‘Hypothetical Grocer’". It is worth noting that in some versions of those embodiments, the graphical elements can include sub-elements to provide an indication of candidate states for navigating the IVR tree associated with "Hypothetical Grocer". For example, the first graphical element 471A can include: a first sub-element 471A1 of "Produce Department" associated with a target state of navigating the IVR tree associated with Hypothetical Grocer to a portion corresponding to the produce department; a second sub-element 471A2 of "Bakery Department" associated with a target state of navigating the IVR tree associated with Hypothetical Grocer to a portion corresponding to the bakery department; and a third sub-element 471A3 of "Deli Department" associated with a target state of navigating the IVR tree associated with Hypothetical Grocer to a portion corresponding to the deli department.

[0080] For example, assume that a user input is detected at the client device 110 to initiate the execution of an assisted phone call to Hypothetical Grocer to navigate the IVR tree associated with Hypothetical Grocer to a target state associated with the produce department of Hypothetical Grocer. The user input can be, for example, a verbal input of "call the produce department of Hypothetical Grocer" or a touch input pointing to the first sub-element 471A1, as indicated by 490A2. In response to detecting the user input, the automated assistant can determine the target state of "Produce Department" for navigating the IVR tree associated with Hypothetical Grocer. In some embodiments, the automated assistant can identify the corresponding values of the parameters associated with navigating the IVR tree to the target state and / or the corresponding values of the tasks to be performed when reaching the target state associated with the produce department, as described herein (e.g., with respect to Figure 1 the parameter engine 153 and with respect to Figure 2 ).

[0081] In some embodiments, and as Figure 4B shown, while the automated assistant is navigating the IVR tree to a target state during an assisted phone call, a given user of the client device 110 can still be able to interact with the client device 110. In some versions of those embodiments, user interactions detected at the client device 110 while the automated assistant is navigating the IVR tree can be used to modify the navigation of the IVR tree. For example, and as Figure 4B shown, assume that a given user associated with the client device 110 views a web page associated with a first search result 420 for "Hypothetical Grocer" (e.g., as indicated by "www.exampleurl1.com / ") at a browser application accessible to the client device 110. In particular, the web page can include selectable graphical elements corresponding to information related to the various departments of Hypothetical Grocer. For example, a given user associated with the client device 110 can select a first graphical selectable element 471B1 to view produce availability at the produce department of Hypothetical Grocer, select a second graphical selectable element 471B2 to view cake availability at the bakery department of Hypothetical Grocer, or select a third graphical selectable element 471B3 to view meat availability at the deli department of Hypothetical Grocer, or view any other graphical selectable element that can be presented.

[0082] For purposes of example, while the automated assistant is navigating the IVR tree associated with the Hypothetical Grocer to a target state associated with the produce department, assume that the user selects third graphical selectable element 471B3 to view meat availability at the deli department of the Hypothetical Grocer, as indicated by 490B1. In response to determining that a given user associated with client device 110 interacted with a search result related to the deli department of the Hypothetical Grocer while the automated assistant is navigating the IVR tree associated with the Hypothetical Grocer to a target state not associated with the meat department, the automated assistant can generate one or more prompts and render the one or more prompts as notifications at user interface 180 of client device 110. For example, the automated assistant can cause notification 479 to be visually rendered in call detail interface 470. The notification 479 includes an indication of "Would you rather connect with the deli department during the ongoing assisted telephone call?" and also includes a first suggestion 479B1 of "yes" and a second suggestion 479B2 of "no". Figure 4B As shown, and in response to rendering notification 479, additional user input is received at client device 110 indicating that a given user of client device 110 would actually rather be contacted with the deli department rather than the produce department as originally intended (e.g., selecting first suggestion 479B1 of "yes" as indicated by 490B2). In this example, navigation of the IVR tree associated with HypotheticalGrocer can be modified based on user interaction with a web page at client device 110. For example, rather than navigating the IVR to a target state associated with the produce department as originally intended by a given user associated with client device 110, the automated assistant can instead navigate to an alternate target state associated with the deli department.

[0083] In various embodiments, if a user interaction is associated with an entity engaged during an assisted telephone call, the automated assistant can prompt only a given user of the client device 110 to modify the navigation of the IVR tree. For example, only a given user of the client device 110 can be prompted to modify the navigation of the IVR tree based on a user interaction associated with Hypothetical Grocer. In some additional or alternative embodiments, even if the user interaction is not associated with an entity engaged during the assisted telephone call, the automated assistant can still prompt a given user of the client device 110 to modify the navigation of the IVR tree. For example, a given user of the client device 110 can be prompted to modify the navigation of the IVR tree based on a user interaction associated with a web page of a particular brand of deli meat different from Hypothetical Grocer. In this example, one or more of the prompts can also include a coupon or advertisement for the particular brand of deli meat.

[0084] In some embodiments, modifying the navigation of the IVR system can be based on the state of navigating the IVR tree. For example, if the automated assistant has not reached a target state when additional user input is received in response to one or more prompts to modify the navigation of the IVR tree, the automated assistant can modify the corresponding values of the parameters utilized when navigating the IVR tree to navigate to the target state, modify the internal nodes traversed when navigating the IVR tree (e.g., as described with respect to Figure 2 ), to navigate to an alternative target state, and / or restart the navigation of the IVR tree. As another example, if the automated assistant has reached a target state when additional user input is received in response to one or more prompts to modify the navigation of the IVR tree, the automated assistant can request that the assisted telephone call be transferred to an alternative target state determined based on the additional user input, and / or restart the navigation of the IVR tree.

[0085] For example, and with specific reference to Figure 4C , assume that the automated assistant has navigated the IVR tree to a target state indicated by the spoken utterance 452C1 of "Now connecting you with the produce department" from the Hypothetical Grocer IVR and the subsequent spoken utterance 452C2 of "Hello, produce department" from the Hypothetical Grocer produce representative. Further assume that the automated assistant receives, in response to Figure 4BBefore reaching the target state, the user input indicated by 490B1 for modifying the navigation of the IVR to an alternative target state associated with the deli department. Typically, at this stage of navigating the IVR tree associated with Hypothetical Grocer, the automated assistant can verify that it has reached the target state (e.g., based on subsequent verbal utterance 452C2), generate a notification requesting the user to join an assisted phone call and / or perform a task on behalf of a given user of the client device 110. However, since the given user provided the Figure 4B user input for modifying the navigation of the IVR tree, the automated assistant is able to cause the synthetic speech 454C1 of "Hello, can you transfer me to the deli department", and can receive the verbal utterance 452C3 of "One moment please" in response to the synthetic speech 452C3 when the Hypothetical Grocer produce representative holds the automated assistant online to transfer the call to the deli department, and the automated assistant can remain online until it receives the verbal utterance 452C4 of "Hello, deli department". Thus, at this stage of navigating the IVR tree associated with Hypothetical Grocer, the automated assistant can verify that it has reached the target state (e.g., based on the verbal utterance 452C4), and generate a notification requesting the user to join an assisted phone call (e.g., as indicated by 454C2), and / or perform a task on behalf of a given user of the client device 110.

[0086] In addition, in various embodiments, the automated assistant is able to cause the transcriptions of various conversations to be visually rendered at the user interface 180 of the client device 110 (e.g., as Figure 4Cas shown). For example, the transcription can be displayed on the home screen of the client device 110, at various software applications (e.g., an automated assistant application, a calling application, and / or other applications). In some embodiments, the transcription can include a conversation between an automated assistant, an IVR system, and / or a human representative associated with an entity. Additionally, in various embodiments, the automated assistant can also cause various graphical elements to be rendered in the call details interface 470. For example, a first selectable graphical element 442C associated with ending an assisted phone call can be provided, a second selectable graphical element 443C associated with ending a given user's joining an assisted phone call can be provided, a third selectable graphical element 444C associated with an assisted phone call that is audibly perceivable via the speaker of the client device 110 can be provided, and / or other selectable graphical elements associated with the assisted phone call can be provided. In various embodiments, in response to generating a notification as indicated by 452C2, these selectable graphical elements can be rendered only at the call details interface 470.

[0087] Although the navigation of the IVR tree associated with Hypothetical Grocer is described herein as being modified from an original target state associated with the produce department to an alternative target state associated with the deli department Figures 4A to 4C , it should be understood that this is for illustrative purposes and is not meant to be limiting. For example, and as described above (e.g., with respect to Figure 1 the user interaction engine 154, with respect to Figure 2 the IVR tree 200, and with respect to Figure 3 the method 300), modifying the navigation of the IVR tree can include adding the deli department as an additional target state that can be navigated to after navigating to the original target state associated with the produce department. As another example, and also as described above (e.g., with respect to Figure 1 the user interaction engine 154, with respect to Figure 2 the IVR tree 200, and with respect to Figure 3 the method 300), modifying the navigation of the IVR tree can include modifying the corresponding values of the parameters used to navigate the IVR tree to the original target state associated with the produce department. For example, if, when navigating to the target state associated with the produce department, a given user indicates that the automated assistant should ask about avocados but starts looking at tomatoes in the produce section of the web page associated with Hypothetical Grocer, the user can be prompted as to whether the automated assistant should additionally or alternatively ask about tomatoes.

[0088] In addition, although Figure 4CDepicted is a transcription of an automated assistant that includes navigating an IVR tree associated with HypotheticalGrocer, but it should be understood that this is for example purposes and is not meant to be limiting. For example, it should be understood that the above-described assisted phone call can be performed while the client device 110 is in a dormant state, a locked state, while other software applications are running in the foreground and / or in other states. Additionally, in embodiments where the automated assistant causes a notification to be rendered at the client device 110, the type of notification rendered at the client device is based on the state of the client device 110, as described herein. Additionally, although described herein with respect to navigating an IVR tree associated with Hypothetical Grocer Figures 4A to 4C , it should be understood that this is also not meant to be limiting, and the techniques described herein can be used to navigate any previously stored IVR tree and be utilized with respect to multiple different entities.

[0089] Turning now to Figure 5 , a flowchart depicting an example method 500 for navigating an IVR tree from a search interface at a client device associated with a given user is shown. For convenience, the operations of method 500 are described with reference to the system performing the operations. The system of method 500 includes one or more processors and / or other components of a computing device (e.g., Figure 1 , Figures 4A to 4C , Figures 6A to 6E and / or Figures 8A to 8D the client device 110 and / or Figure 9 the computing device 910, one or more servers, and / or other computing devices). Additionally, although the operations of method 500 are shown in a particular order, this is not meant to be limiting. One or more operations can be reordered, omitted, and / or added.

[0090] At block 552, the system receives user input from a given user via a search interface of a client device associated with the given user. The user input for initiating an assisted phone call can be one or more of an oral input for initiating an assisted phone call, a touch input for initiating an assisted phone call, or a typed input for initiating an assisted phone call. Additionally, the user input for initiating an assisted phone call can be received at various interfaces (e.g., a search interface, a voice interface, an automated assistant interface, and / or other interfaces from which an assisted phone call can be initiated). Additionally, for example, the search interface can be implemented as part of a phone or contacts application, a browser application, an automated assistant application, a web-based browser, and / or any other interface that enables a search function.

[0091] At block 554, the system issues a search query at a search interface of a client device based on user input. The search query can be posted to one or more databases accessible to the client device (e.g., a restricted access database and / or a public database), a search engine, and / or any other search-based system.

[0092] At block 556, the system identifies a given search result associated with an entity to be contacted on behalf of a given user during an assisted telephone call based on the issued search query, the given search result being associated with a target state of an IVR tree for the identified entity from among a plurality of candidate states. At block 558, the system causes at least the given search result to be rendered at the client device. At block 560, the system receives a selection of the given search result from the given user via the search interface of the client device. The given search result can include content responsive to the search query issued at block 554. In some embodiments, additional given search results corresponding to other candidate states can also be rendered with the given search result. In some versions of those embodiments, the given search result can be presented more prominently than the additional given search results associated with other candidate states based on a most recent user interaction at the client device. For example, if the user is interacting with other search results related to various brands of paint or painting equipment and the issued search query is for a home improvement entity, the given search result can be associated with the home improvement entity or, more specifically, with the painting department of the home improvement entity. Thus, upon selection of the given search result, the system can determine the painting department as the target state for navigating an IVR tree associated with the home improvement entity. The selection of the given search result can be based on a touch input, a verbal input, and / or a typed input.

[0093] At block 562, the system initiates execution of the assisted telephone call in response to the selection to navigate the IVR tree to the target state. The system can initiate execution of the assisted telephone call by establishing a communication session between the client device associated with the given user and an additional client device associated with the entity identified at block 556. Various voice communication protocols including, for example, VoIP, PSTN, and / or other telephone communication protocols can be used to establish the communication session. In some embodiments, the system can navigate the IVR tree to the target state by simulating button presses to traverse nodes of the IVR tree, by rendering synthetic voice audio data including a synthetic voice to traverse nodes of the IVR tree, and / or by rendering audio data including a verbal input of the given user associated with the client device (e.g., as described with respect to Figure 2 ). In some versions of those embodiments, the system can also utilize corresponding values of parameters associated with navigating the IVR tree. The values can be retrieved from one or more databases (e.g., Figure 1Retrieve from the user profile database 153A) and / or be able to retrieve corresponding values of given user request parameters associated with the client device.

[0094] At block 564, the system renders, via the client device, a notification indicating the result of the execution of the navigational IVR tree. For example, the notification can include a request for a given user associated with the client device to join an assisted telephone call. In the embodiments described herein where the system also performs tasks on behalf of a given user associated with the client device and with respect to an entity, the notification can additionally or alternatively include the result of the execution of the task. In various embodiments, the navigation of the IVR tree described with respect to Figure 5 can be modified based on user interaction at the client device, as described above with respect to Figure 3 and Figures 4A to 4C as described.

[0095] Now referring to Figures 6A to 6E , various non - limiting examples of user interfaces associated with navigating an IVR tree from a search interface at a client device 110 associated with a given user are depicted. Figures 6A to 6E Each depicts a client device 110 having a graphical user interface 180 and can include Figure 1 one or more components of the client device. The client device 110 can be substantially similar to Figures 4A to 4C the client device and includes many of the same components described above with respect to Figures 4A to 4C , including an instance of an automated assistant (e.g., Figure 1 the automated assistant 115).

[0096] In the various embodiments described herein, user input can be received and a search query can be issued based on the user input. The user input can be verbal input, touch input, and / or typed input that includes the search query. For example, and as Figure 6AAs shown, assume that the user provides a search query 684A for "Lost Baggage Hypothetical Airlines (Hypothetical Airlines lost baggage)" at a browser application accessible at the client device 110 (e.g., as indicated by the URL 611 of "www.exampleurl0.com / "), such as included in the text response interface element 184. Additionally, the search results include a first search result 620 for "Hypothetical Airline" that offers flights serving North America. In some embodiments, the search result 620 can be associated with various selectable graphical elements that, when selected, cause the client device 110 to perform corresponding actions. For example, when the call graphical element 621 associated with the search result 620 is selected, the user input can indicate that a phone call action to Hypothetical Airline should be performed. As another example, when the user account graphical element 622 associated with the search result 620 is selected, the user input can indicate that an account lookup action for the user account associated with Hypothetical Airline should be performed. As yet another example, when the flights graphical element 623 associated with the search result 620 is selected, the user input can indicate that a flight lookup action for the flights associated with Hypothetical Airline should be performed.

[0097] In various embodiments, the search result 620 can also include one or more IVR deep links to candidate states of an IVR tree associated with an entity. For example, and as Figure 6AAs shown, a first IVR deep link 620A1 for "reportlastbaggage (Report lost baggage)" corresponding to a candidate state for reporting lost baggage when navigating an IVR tree associated with Hypothetical Airline can be provided, and a second IVR deep link 620A2 for "speak with a representative (Speak with a representative)" corresponding to a candidate state for speaking with a Hypothetical Airline representative when navigating an IVR tree associated with Hypothetical Airline can be provided. An additional graphical element 620A3 for "see more (See more)" can be optionally provided, and when the additional graphical element 620A3 is selected, candidate states associated with one or more additional IVR deep links can be rendered together with the first deep link 620A1 and the second deep link 620A2. In some versions of those embodiments, one or more of the rendered IVR deep links 620A1 and / or 620A2 can be based on a most recent user interaction at the client device 110. For example, the first IVR deep link 620A1 for "report lost baggage (Report lost baggage)" can be rendered at the client device based on a search query 684A including "lost baggage (Lost baggage)" or based on a given user of the client device 110 previously interacting with other search results related to reporting lost baggage. As another example, the second IVR deep link 620A2 for "speak with arepresentative (Speak with a representative)" can be rendered at the client device 110 based on it being the most popular target state among the candidate states. In these examples, the first IVR deep link 620A1 can be rendered more prominently based on determining that it is more relevant to a given user of the client device 110 (e.g., determined based on the search query 684A and / or based on the given user of the client device 110 previously interacting with other search results) than the second IVR deep link 620A2.

[0098] A secondary phone call can be initiated based on user input pointing to the first IVR deep link 620A1 to navigate the IVR tree associated with Hypothetical Airlines to a target state associated with reporting lost luggage, and an execution of the secondary phone call for navigating the IVR tree to the target state can be initiated. In some embodiments, the automated assistant can utilize the resources of the client device 110 to initiate the execution of the secondary phone call. In some additional or alternative embodiments, the automated assistant can utilize the resources of a remote system to initiate the execution of the secondary phone call (e.g., a cloud-based secondary phone call). For example, assume that the user selects the first IVR deep link 620A1, but the client device 110 cannot execute a phone call using VoIP, PSTN, or other phone communication protocols (e.g., via a laptop or desktop computer). In this example, the automated assistant 115 can cause a remote system (which may or may not communicate with the client device 110) to initiate and execute the secondary phone call on behalf of the user.

[0099] In various embodiments, the call details interface 670 can be rendered at the client device 110 before receiving user input for initiating a phone call with "HypotheticalAirline", or can be rendered at the client device 110 in response to receiving user input for initiating a phone call with HypotheticalAirline. In some versions of those embodiments, the call details interface 670 can be rendered at the client device 110 as part of the user interface 180. In some other versions of those embodiments, the call details interface 670 can be an interface separate from the user interface 180 that overlays the user interface 180, and can include call details interface elements 186 that allow the user to expand the call details interface 670 to display additional call details (e.g., by swiping up on the call details interface element 186) and / or close the call details interface 670 (e.g., by swiping down on the call details interface element 186). Although the call details interface 670 is depicted as being located at the bottom of the user interface 180, it should be understood that this is for illustrative purposes and is not intended to be limiting. For example, the call details interface 670 can be rendered at the top of the user interface 180, on one side of the user interface 180, or as an interface completely separate from the user interface 180.

[0100] In various embodiments, the call details interface 670 can include a plurality of graphical elements. In some versions of these embodiments, the graphical elements can be selectable such that when a given one of the graphical elements is selected, the client device 110 can perform a corresponding action. As Figure 6AAs shown, the call details interface 670 includes a first graphical element 671 for "Assisted Call, Hypothetical Airline", a second graphical element 672A for "Regular Call", and a third graphical element 673A for "Save Contact 'Hypothetical Airlines'". An assisted phone call can be initiated from the call details interface 670 additionally or alternatively based on user input pointing to the first selectable graphical element 671, and the execution of the assisted phone call is based on the selection of the first selectable graphical element 671.

[0101] In various embodiments, one or more IVR deep links associated with a candidate state may not be rendered. In some versions of those embodiments, the automated assistant may visually render various nodes (e.g., internal nodes and / or leaf nodes as described with respect to Figure 2 for navigating an IVR tree associated with an entity). The visual rendering of the various nodes may be based on the confidence level of the IVR tree stored in association with the entity. The confidence level of the IVR tree stored in association with the entity may be based on the number of assisted phone calls to the entity that navigate an IVR tree different from the IVR tree stored in association with the entity, and optionally based on the degree of difference between the navigated IVR tree and the stored IVR tree. In particular, Figures 6B to 6E illustrates various non-limiting examples of how these various nodes may be presented to the user based on different confidence levels of the IVR tree associated with Hypothetical Airline.

[0102] For an example with respect to Figures 6B to 6E assume that the entity to be engaged during an assisted phone call is Hypothetical Airline as described above with respect to Figure 6A and the user is viewing a first search result 610 associated with Hypothetical Airline, but has not received user input including a search query 684A. Specifically referring to Figure 6B, further assume that the automated assistant has confidence in the IVR tree height stored in association with Hypothetical Airline. The automated assistant can have confidence in the IVR tree height stored in association with Hypothetical Airline based on the confidence level determined based on the nodes of the IVR tree remaining the same for most, if not all, of the assisted phone calls that include the same IVR tree. In this example, instead of providing one or more IVR deep links, the automated assistant is able to cause information including an indication of how to navigate the IVR tree associated with Hypothetical Airline to be presented before receiving user input for initiating the execution of an assisted phone call. For example, the call details interface 670 can include: first information 671A of "Press '1', '3', '5' to report lost baggage", which indicates the button press sequence that a given user of the client device 110 can provide to reach the target state of the IVR tree associated with reporting lost baggage; and second information 671B of "Press '5' to speak with a representative", which indicates the button press sequence that a given user of the client device 110 can provide to reach the target state of the IVR tree associated with speaking with a representative. A graphical element 671C of "see more" can be optionally provided, and when the graphical element 671C is selected, it can cause additional information associated with other candidate states to be rendered together with the first information 671A and the second information 671B. Similar to Figure 6A , the information associated with the candidate states rendered in the call details interface 670 can be optionally rendered based on the most recent user interaction with the client device 110 and / or the overall popularity of the candidate states.

[0103] Specific reference Figure 6C And similar to Figure 6B, further assume that the automated assistant has confidence in the IVR tree height stored in association with HypotheticalAirline. In this example, instead of providing one or more IVR deep links or information related to directly navigating to the desired target state via a button press sequence before initiating the assisted phone call, the automated assistant is able to cause the candidate state to be rendered in notification 679C in the call details interface 670 before receiving the spoken utterance 654C1 from the Hypothetical Airline IVR that includes the candidate state, as indicated by 652C1. In particular, the candidate state can include those indicated by 679C1, 679C2, 679C3, 679C4, and 679C5. Each candidate state included in notification 679C can be selectable such that a given user associated with the client device 110 can simply select a given one of the candidate states as the target state, and the automated assistant will automatically navigate the assisted phone call to the target state.

[0104] Specific reference Figure 6D And associated with Figure 6C Compared to, further assume that the automated assistant has moderate confidence in the IVR tree stored in association with HypotheticalAirlines. Based on the confidence level determined based on the nodes of the IVR tree including some differences in some assisted phone calls that include the same IVR tree, the automated assistant can have moderate confidence in the IVR tree stored in association with HypotheticalAirline. In this example, instead of providing notification 679D in the call details interface 670 before receiving the spoken utterance including the candidate state from the Hypothetical Airline IVR, the automated assistant can wait until the first part 652D1 of the spoken response including a subset of the candidate states is provided by the HypotheticalAirline IVR to render notification 679C including the candidate state. For example, the degree of difference between the stored IVR trees associated with Hypothetical Airline can have only moderate confidence in the subset of candidate states included in the first part 652D1 of the spoken utterance, but high confidence in the remaining candidate states. Thus, after the subset of candidate states is included in the first part 652D1 of the spoken utterance, the automated assistant can generate notification 679D before receiving the second part 652D2 of the spoken utterance, as indicated by 654D1. Similar to Figure 6C , each candidate state included in notification 679D can be selectable such that a given user associated with the client device 110 can simply select a given one of the candidate states as the target state, and the automated assistant will automatically navigate the assisted phone call to the target state.

[0105] Specific reference Figure 6E And in relation to Figure 6C and Figure 6D In contrast, further assume that the automated assistant has no confidence in the IVR tree stored in association with HypotheticalAirline. Based on the confidence level determined for the nodes of the IVR tree based on differences in a threshold number of auxiliary phone calls including different IVR trees, the automated assistant may have no confidence in the IVR tree stored in association with HypotheticalAirline. In this example, rather than providing the notification 679E in the call details interface 670 prior to receiving the spoken utterance, the automated assistant may wait until the HypotheticalAirlineIVR provides a spoken response 652E1 including a candidate status to render a notification 679D including the candidate status, as indicated by 654E1. Similar to Figure 6C , each candidate status included in the notification 679E can be selectable such that a given user associated with the client device 110 can simply select a given one of the candidate statuses as the target status, and the automated assistant will automatically navigate the auxiliary phone call to the target status.

[0106] Although the present disclosure has been described herein with respect to the entity Hypothetical Airline Figures 6A to 6E , and certain graphical elements and information are depicted in conjunction with the search results 620 and the call details interface, it should be understood that this is for illustrative purposes only and is not intended to be limiting. It should be understood that the IVR tree associated with any entity employing an IVR system can be stored and subsequently used to perform an auxiliary phone call. Additionally, it should be understood that the graphical elements and information provided by the automated assistant can be based on the IVR tree, and the configurations of the IVR trees associated with entities are virtually infinite.

[0107] Turning now to Figure 7 , a flowchart depicting an example method 700 of navigating a dynamic interactive voice response (IVR) tree is shown. For convenience, the operations of method 700 are described with reference to the system performing the operations. The system of method 700 includes one or more processors and / or other components of a computing device (e.g., Figure 1 , Figures 4A to 4C , Figures 6A to 6E and / or Figures 8A to 8D the client device 110 and / or Figure 9 the computing device 910, one or more servers, and / or other computing devices). Additionally, although the operations of method 700 are shown in a particular order, this is not intended to be limiting. One or more operations can be reordered, omitted, and / or added.

[0108] At block 752, the system receives, from a given user via a client device associated with the given user, user input for initiating an assisted telephone call. The user input for initiating an assisted telephone call can be one or more of an oral input for initiating an assisted telephone call, a touch input for initiating an assisted telephone call, or a typed input for initiating an assisted telephone call. Additionally, the user input for initiating an assisted telephone call can be received at various interfaces (e.g., a search interface, a voice interface, an automated assistant interface, and / or other interfaces from which an assisted telephone call can be initiated).

[0109] At block 754, the system identifies an entity to be contacted on behalf of the given user during the assisted telephone call based on the user input. The system can identify the entity to be contacted during the assisted telephone call based on the user input received at block 752 and / or user interactions with the client device immediately prior to initiating the assisted telephone call. For example, the system can identify the entity "Example Airlines" based on processing the oral input "Report my lost luggage to Example Airlines" received at the client device. As another example, the system can identify the entity "Example Airlines" based on determining that a given user of the client device selected a call interface element associated with Example Airlines or a particular graphical element associated with a candidate state of an IVR tree associated with Example Airlines.

[0110] At block 756, the system identifies an IVR tree associated with the entity identified at block 754, the IVR tree including a plurality of candidate states. The IVR tree associated with the identified entity can be stored in association with the entity identified at block 754 (e.g., in the Figure 1 entity database 151A and / or Figure 1 the IVR tree database 152A). The stored IVR tree associated with the entity can have been previously stored in one or more databases based on the system (and / or additional instances of the system) previously crawling an IVR system associated with the identified entity, and / or the identified entity can provide the IVR tree for the IVR system associated with the identified entity. Additionally, the system can continuously update the IVR tree of the IVR system associated with the identified entity based on multiple assisted telephone calls navigating the IVR tree. The plurality of candidate states can correspond to particular nodes of the IVR tree that the system can navigate to during the assisted telephone call.

[0111] At block 758, the system receives context information. In some embodiments, block 756 can include optional sub-blocks 758A and / or optional sub-block 758B. If included, at optional sub-block 758A, the system receives context information associated with the client device. Context information associated with the client device can include, for example, device status information (e.g., determined by device status engine 112), date and / or time information associated with the current location of the client device, the current location of the client device, and / or other context information generated by the client device. If included, at optional sub-block 758B, the system receives context information associated with a given user of the client device. Context information associated with a given user of the client device can include, for example, electronic communications created by or received by the user of the client device (e.g., emails, text messages, voice messages, etc.), user account information associated with various user accounts of the given user of the client device (e.g., account status information, account numbers, account usernames, account passwords, etc.), financial information of the given user of the client device (e.g., bank account numbers, credit card numbers, payment app information, etc.), photo albums of the given user of the client device, social media profiles of the given user of the client device, user preferences of the user of the client device, personal information of the given user of the client device (e.g., name, phone number, physical address, email address, social security number, date of birth, etc.), and / or other information associated with the given user of the client device.

[0112] At block 760, the system determines the availability of multiple candidate states as target states of the IVR tree based on context information associated with the client device and / or a given user of the client device. In various embodiments, based on the context information, some candidate states of the IVR tree may be available or unavailable to a given user of the client device. For example, assume that the entity identified at block 754 is a restaurant entity. In this example, some candidate states may be available only during the business hours of the restaurant (e.g., candidate states associated with placing a takeout order). As another example, assume that the entity identified at block 754 is an airline entity. In this example, some candidate states may be available only when the user account status is above a threshold membership status (e.g., only for gold status and above members). In other words, the IVR tree associated with the identified entity can be dynamic because candidate states can be available or unavailable in certain contexts.

[0113] At block 762, the system initiates the execution of an assisted telephone call to navigate the IVR tree to a target state. The system is capable of initiating the execution of an assisted telephone call in response to additional user input selecting the target state. Additionally, the system is capable of initiating the execution of an assisted telephone call by establishing a communication session between a client device associated with a given user and an additional client device associated with the entity identified at block 754. A variety of voice communication protocols, including for example VoIP, PSTN, and / or other telephone communication protocols, can be used to establish the communication session. In some embodiments, the system is capable of navigating the IVR tree to the target state by simulating button presses to traverse nodes of the IVR tree, by rendering synthetic speech audio data including synthetic speech to traverse nodes of the IVR tree, and / or by rendering audio data including the spoken input of a given user associated with the client device (e.g., as described with respect to Figure 2 ). In some versions of those embodiments, the system is also capable of leveraging corresponding values of parameters associated with navigating the IVR tree. The corresponding values can be retrieved from one or more databases (e.g., Figure 1 's user profile database 153A) and / or can be requested from a given user associated with the client device.

[0114] At block 764, the system renders, via the client device, a notification indicating the result of the execution of navigating the IVR tree. For example, the notification can include a request for a given user associated with the client device to join an assisted telephone call. In the embodiments described herein where the system also performs a task on behalf of a given user associated with the client device and with respect to the entity, the notification can additionally or alternatively include the result of the execution of the task. In various embodiments, the navigation of the IVR tree described with respect to Figure 5 can be modified based on user interaction at the client device, as seen above with reference to Figure 3 and Figures 4A to 4C .

[0115] Now referring to Figures 8A to 8D , various non-limiting examples of user interfaces associated with navigating a dynamic IVR tree are depicted. Figures 8A to 8D Each depicts a client device 110 having a graphical user interface 180 and can include Figure 1 one or more components of the client device. The client device 110 can be substantially similar to Figures 4A to 4C 's client device and includes many of the same components described above with respect to Figures 4A to 4C , including an instance of an automated assistant (e.g., Figure 1 's automated assistant 115).

[0116] In various embodiments described herein, the IVR tree associated with an entity can be a dynamic IVR tree because one or more candidate states associated with the IVR tree can be available or unavailable. The availability of one or more candidate states can be determined based on context information. In some embodiments, the context information can be context information associated with the client device 110 and can include, for example, device status information of the client device 110 (e.g., determined via the device status engine 112), date and / or time associated with the current location of the client device 110, the current location of the client device, and / or other context information generated by the client device 110. In some additional or alternative embodiments, the context information can be context information associated with a given user of the client device 110 and can include, for example, electronic communications created or received by the user of the client device (e.g., email, text message, voice message, etc.), user account information associated with various user accounts of the given user of the client device 110 (e.g., account status information, account number, account username, account password, etc.), money information of the given user of the client device 110 (e.g., bank account number, credit card number, payment application information, etc.), photo album of the given user of the client device 110, social media profile of the given user of the client device 110, user preferences of the user of the client device 110, personal information of the given user of the client device (e.g., name, phone number, physical address, email address, social security number, date of birth, etc.), and / or other information associated with the given user of the client device.

[0117] Specific reference Figure 8A and Figure 8B, assume that a given user of client device 110 is viewing search results for a restaurant entity at a browser application accessible at client device 110 (e.g., as indicated by URL 811A such as "www.exampleurl0.com / "). Further assume that the search results include at least search results 820A for "Hypothetical Restaurant" that is open from 9:00 a.m. to 11:00 p.m. In some embodiments, search results 820A can be associated with various selectable graphical elements that, when selected, cause client device 110 to perform corresponding actions. For example, when the call graphical element 821A associated with search results 820A is selected, the user input can indicate that a phone call action to Hypothetical Restaurant should be performed. As another example, when the directions graphical element 822A associated with search results 820A is selected, the user input can indicate that a navigation action to the physical location of Hypothetical Restaurant should be performed. As yet another example, when the menu graphical element 823A associated with search results 820A is selected, the user input can indicate that a menu lookup action for the menu associated with Hypothetical Airline should be performed.

[0118] In various embodiments, call details interface 870 can be rendered at client device 110 before receiving user input for initiating a phone call to "Hypothetical Restaurant", or can be rendered at client device 110 in response to receiving user input for initiating a phone call to Hypothetical Restaurant. In some versions of those embodiments, call details interface 870 can be rendered at client device 110 as part of user interface 180. In some other versions of those embodiments, call details interface 870 can be an interface separate from user interface 180 that overlays user interface 180, and can include call details interface element 186 that allows the user to expand call details interface 870 to display additional call details (e.g., by swiping up on call details interface element 186) and / or close call details interface 870 (e.g., by swiping down on call details interface element 186). Although call details interface 870 is depicted as being located at the bottom of user interface 180, it should be understood that this is for illustrative purposes and is not intended to be limiting. For example, call details interface 870 can be rendered at the top of user interface 180, on one side of user interface 180, or as an interface completely separate from user interface 180.

[0119] In various embodiments, the call details interface 870 can include a plurality of graphical elements. In some versions of these embodiments, the graphical elements can be selectable such that when a given one of the graphical elements is selected, the client device 110 can perform a corresponding action. As Figure 8A and Figure 8B shown, the call details interface 870 includes a first graphical element 871A of "Assisted Call, Hypothetical Restaurant", a second graphical element 872A of "Regular Call", and a third graphical element 873A of "Save Contact ‘HypotheticalRestaurant’". The graphical elements can also include sub-elements corresponding to candidate states associated with an IVR tree associated with HypotheticalRestaurant. For example, and as Figure 8A shown, the first graphical element 871A can include: a first sub-element 871A1 of "Make Reservations" associated with navigating the IVR tree associated with Hypothetical Restaurant to a target state corresponding to the part for making a restaurant reservation; a second sub-element 871A2 of "Order Carryout" associated with navigating the IVR tree associated with Hypothetical Restaurant to a target state corresponding to the part for ordering takeout from the restaurant; and a third sub-element 871A3 of "Order Delivery" associated with navigating the IVR tree associated with HypotheticalRestaurant to a target state corresponding to the part for ordering delivery from Hypothetical Restaurant. In contrast, and as Figure 8B shown, the first graphical element 871A can include: a first sub-element 871B1 of "MakeReservations" associated with navigating the IVR tree associated with Hypothetical Restaurant to a target state corresponding to the part for making a restaurant reservation; and a second sub-element 871B2 of "Hours of Operation" associated with navigating the IVR tree associated with Hypothetical Restaurant to a target state corresponding to the part for listening to the business hours of Hypothetical Restaurant.

[0120] It is worth noting that included in Figure 8A and8B The differences in candidate states in Figure 8A can be based on context information associated with the client device 110 of a given user. Notably, and referring to Figure 8B , the time and date information 812A at the current location of the client device 110 indicates a time of 9:30 am, and this corresponds to HypotheticalRestaurant being open (e.g., based on business hours being 9:00 am to 11:00 pm). Thus, the candidate states associated with the alternative sub-elements 871A2 and 871A3 can be available because Hypothetical Restaurant is open. Conversely, and referring to Figure 8A , the time and date information 812B at the current location of the client device 110 indicates a time of 8:30 am, and this corresponds to Hypothetical Restaurant being closed (e.g., based on business hours being 9:00 am to 11:00 pm). Thus, the candidate states associated with the alternative sub-elements 871A2 and 871A3 can be unavailable because Hypothetical Restaurant is closed, but the automated assistant can still be able to make restaurant reservations and / or inquire about business hours by navigating the IVR tree associated with HypotheticalRestaurant even when it is closed. Thus, the IVR tree can be dynamic because some candidate states can be available in a given context while other candidate states can be unavailable in a given context. As another example, assume that in Figure 8A Hypothetical Restaurant is open, but the current location of the client device 110 is outside the delivery radius of Hypothetical Restaurant. In this example, the candidate state associated with the third sub-element 871A3 of "Order Delivery" can be omitted. As yet another example, assume that in Figure 8A Hypothetical Restaurant is open, but indoor and outdoor dining services are not offered at Hypothetical Restaurant. In this example, the candidate state associated with the first sub-element 871A1 of "Make Reservations" can be omitted.

[0121] Specifically referring to Figure 8C and Figure 8D, assume that a given user of client device 110 is viewing search results for a hotel entity on a browser application accessible at client device 110 (e.g., as indicated by URL 811C such as "www.exampleurl0.com / "). Further assume that the search results include at least a first search result 820C for "HypotheticalHotel" and a second search result 830C for "Example Hotel". In some embodiments, search results 820C and / or 830C can be associated with various selectable graphical elements that, when selected, cause client device 110 to perform corresponding actions. For example, when call graphical elements 821C and / or 831C associated with search results 820A and / or 830C respectively are selected, user input can indicate that a phone call action should be performed on the corresponding hotel entity. As another example, when directions graphical elements 822C and / or 832C associated with search results 820A and / or 830C respectively are selected, user input can indicate that a navigation action to the physical location of the corresponding hotel entity should be performed. As yet another example, when account elements 823C and / or 833C associated with search results 820A and / or 830C respectively are selected, user input can indicate that an account lookup action on the user account associated with the corresponding hotel entity should be performed.

[0122] In various embodiments, call details interface 870 can be rendered at client device 110 before receiving user input for initiating a phone call with a corresponding one of the hotel entities, or can be rendered at client device 110 in response to receiving user input for initiating a phone call with a corresponding one of the hotel entities. In some versions of those embodiments, call details interface 870 can be rendered at client device 110 as part of user interface 180. In some other versions of those embodiments, call details interface 870 can be a separate interface figure that overlays user interface 180 and can include call details interface element 186, which allows the user to expand call details interface 870 to display additional call details (e.g., by swiping up on call details interface element 186) and / or close call details interface 870 (e.g., by swiping down on call details interface element 186). Although call details interface 870 is depicted as being located at the bottom of user interface 180, it should be understood that this is for illustrative purposes and is not intended to be limiting. For example, call details interface 870 can be rendered at the top of user interface 180, on one side of user interface 180, or as a completely separate interface from user interface 180.

[0123] In various embodiments, the call details interface 870 can include a plurality of graphical elements. In some versions of these embodiments, the graphical elements can be selectable graphical elements such that when a given one of the graphical elements is selected, the client device 110 can perform a corresponding action. As Figure 8C shown, the call details interface 870 includes a first graphical element 871C of "Assisted Call, HypotheticalHotel", a second graphical element 872C of "Regular Call", and a third graphical element 873C of "Save Contact ‘Hypothetical Hotel’". Figure 8C The graphical elements can also include sub-elements corresponding to candidate states associated with the IVR tree associated with HypotheticalHotel. For example, and as Figure 8C shown, the first graphical element 871C can include: a first sub-element 871C1 of "Change Reservation" associated with navigating the IVR tree associated with HypotheticalHotel to a target state corresponding to the part for changing hotel reservations; a second sub-element 871C2 of "Request Free Upgrade" associated with navigating the IVR tree associated with HypotheticalHotel to a target state corresponding to the part for requesting a free upgrade of an existing hotel reservation; and a third sub-element 871C3 of "Concierge Direct Line" associated with navigating the IVR tree associated with Hypothetical Hotel to a target state corresponding to the part for contacting the concierge of Hypothetical Hotel.

[0124] In contrast, and as Figure 8D shown, the call details interface 870 includes: a first graphical element 871D of "Assisted Call, ExampleHotel" (as contrasted with "Hypothetical Hotel" in Figure 8C ); a second graphical element 872D of "Regular Call"; and a third graphical element 873D of "Save Contact ‘ExampleHotel’". Figure 8D The graphical elements can also include sub-elements corresponding to candidate states associated with the IVR tree associated with Example Hotel. For example, and asFigure 8D As shown, the first graphical element 871D can include: a first sub - element 871D1 of "Change Reservation" associated with navigating the IVR tree associated with Example Hotel to a target state corresponding to the part for changing a hotel reservation; a second sub - element 871D2 of "Cancel Reservation" associated with navigating the IVR tree associated with Example Hotel to a target state corresponding to the part for canceling an existing hotel reservation; and a third sub - element 871D3 of "Front Desk" associated with navigating the IVR tree associated with Hypothetical Hotel to a target state corresponding to the part for contacting the front desk at Example Hotel.

[0125] It is worth noting that Figure 8C and Figure 8D the differences in the candidate states included in can be based on context information associated with a given user of the client device 110. It is worth noting that, and referring to Figure 8C , the automated assistant can access the user account of a given user of the client device 110 associated with HypotheticalHotel. For example, the automated assistant can access the user account status (e.g., bronze member, silver member, gold member, etc.), the user account number, the user account history including stays at various locations in Hypothetical Hotel, etc. For the sake of example, assume that a given user of the client device 110 is a gold member of Hypothetical Hotel. As a result, the candidate states associated with the selectable sub - elements 871C2 and 871C3 can be available based on the gold member status of the given user of the client device 110. In contrast, and referring to Figure 8D , the automated assistant may not have any record of the user account associated with Example Hotel. As a result, the candidate states associated with the selectable sub - elements 871C2 and 871C3 may not be available, but the automated assistant can still be able to navigate to Figure 8D the various candidate states depicted in. Thus, the IVR tree can be dynamic because some candidate states can be available in a given context while other candidate states can be unavailable in a given context.

[0126] Although described herein in terms of the availability of candidate states based on context information associated with a given user of the client device 110 Figure 8A and Figure 8B , and described herein in terms of the availability of candidate states based on context information associated with a given user of the client device 110Figure 8C and Figure 8D , however, it should be understood that this is for illustrative purposes and not intended to be limiting. For example, the availability of a candidate state can be based on context information associated with the client device 110 of a given user and context information associated with the given user of the client device 110. For example, and with reference to 8C, if a given user of the client device 110 initiates an assisted phone call when the concierge at the Hypothetical Hotel is not working, the candidate state associated with the third subelement 871C3 of "Concierge Direct Line" can be omitted, even though the user account status of the user account associated with the Hypothetical Hotel enables the given user of the client device 110 to directly contact the concierge at the Hypothetical Hotel.

[0127] In addition, although certain entities are described herein Figures 8A to 8D , and certain graphical elements are depicted with the search results and call detail interfaces, it should be understood that this is for illustrative purposes and not intended to be limiting. It should be understood that the IVR tree associated with any entity employing an IVR system can be stored and subsequently used to perform assisted phone calls. In addition, it should be understood that the graphical elements and information provided by the automated assistant can be based on the IVR tree, and the configuration of the IVR tree associated with an entity is virtually infinite.

[0128] Figure 9 is a block diagram of an example computing device 910 that can optionally be used to perform one or more aspects of the techniques described herein. In some embodiments, one or more of the client device, the cloud-based automated assistant component, and / or other components can include one or more components of the example computing device 910.

[0129] The computing device 910 generally includes at least one processor 914 that communicates with a plurality of peripheral devices via a bus subsystem 912. These peripheral devices can include: a storage subsystem 924 (including, for example, a memory subsystem 925 and a file storage subsystem 926), a user interface output device 920, a user interface input device 922, and a network interface subsystem 916. The input and output devices allow a user to interact with the computing device 910. The network interface subsystem 916 provides an interface to an external network and is coupled to a corresponding interface device in other computing devices.

[0130] The user interface input device 922 can include a keyboard, a pointing device such as a mouse, trackball, touchpad, or graphics tablet, a scanner, a touchscreen incorporated into a display, an audio input device such as a voice recognition system, microphone, and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways of inputting information into the computing device 910 or onto a communication network.

[0131] The user interface output device 920 can include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem can include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem can also provide a non-visual display via, for example, an audio output device. In general, the use of the term "output device" is intended to include all possible types of devices and ways of outputting information from the computing device 910 to a user or another machine or computing device.

[0132] The storage subsystem 924 stores programs and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 924 can include logic for performing selected aspects of the methods disclosed herein and implementing Figure 1 the various components depicted in

[0133] These software modules are typically executed by the processor 914 alone or in combination with other processors. The memory 925 used in the storage subsystem 924 can include multiple memories, including a main random access memory (RAM) 930 for storing instructions and data during program execution and a read-only memory (ROM) 932 in which fixed instructions are stored. The file storage subsystem 926 can provide permanent storage for program and data files and can include a hard disk drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules implementing the functionality of certain embodiments can be stored in the storage subsystem 924 by the file storage subsystem 926 or in other machines accessible to the processor 914.

[0134] The bus subsystem 912 provides a mechanism for enabling the various components and subsystems of the computing device 910 to communicate with each other as intended. Although the bus subsystem 912 is schematically shown as a single bus, alternative embodiments of the bus subsystem can use multiple buses.

[0135] The computing device 910 can be of various types, including workstations, servers, computing clusters, blade servers, server farms, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, for purposes of illustrating some embodiments, Figure 9The description of computing device 910 depicted is only intended as a specific example. Many other configurations of computing device 910 may have more or fewer components than the computing device depicted in Figure 9 the figures.

[0136] In cases where the systems described herein collect or otherwise monitor personal information about a user or may utilize personal and / or monitored information, the user may be provided with an opportunity to control whether the program or functionality collects user information (e.g., information about the user's social network, social behavior or activities, profession, user preferences, or user's current geographic location), or to control whether and / or how content may be received from a content server that may be more relevant to the user. Additionally, before storing or using certain data, it may be processed in one or more ways such that personally identifiable information is removed. For example, a user's identity may be processed so that the user's personally identifiable information cannot be determined, or a user's geographic location may be generalized (such as to a city, zip code, or state level) when geographic location information is obtained so that the user's specific geographic location cannot be determined. Thus, the user can control how information about the user is collected and / or used.

[0137] In some embodiments, a method implemented by one or more processors is provided, and the method includes: receiving, via a client device associated with a given user, user input for initiating an assisted telephone call; identifying, based on the user input, an entity to be contacted on behalf of the given user during the assisted telephone call; identifying an interactive voice response (IVR) tree associated with the identified entity, the IVR tree including a plurality of candidate states; determining, based on the user input or additional user input, a target state of the IVR tree from among the plurality of candidate states; and initiating execution of the assisted telephone call to navigate the IVR tree to the target state. Execution of the assisted telephone call includes automatically navigating the assisted telephone call to the target state of the IVR tree. The method further includes: during execution of the assisted telephone call and when the IVR tree is navigated to the target state, detecting at least one user interaction directed to the client device and associated with the identified entity, and generating one or more prompts related to modifying the navigation of the IVR tree based on the at least one user interaction directed to the client device. The method further includes: in response to determining that another additional user input is received in response to rendering one or more additional prompts, modifying the navigation of the IVR tree and modifying the execution of the assisted telephone call based on the another additional user input.

[0138] These and other embodiments of the techniques disclosed herein can optionally include one or more of the following features.

[0139] In some embodiments, the method may further include: after navigating the IVR tree to a particular state during the assisted phone call and based on the additional user input, generating a notification requesting the given user associated with the client device to join the assisted phone call, and causing the notification to be rendered at the client device associated with the given user. In some versions of those embodiments, causing the notification to be rendered at the client device associated with the given user may include one or more of the following: causing the client device to vibrate, causing the client device to audibly render the notification requesting the given user to join the assisted phone call, or causing the client device to visually render the notification requesting the given user to join the assisted phone call. In some additional or alternative versions of those embodiments, the method may further include, after navigating the IVR tree to the particular state, engaging in a conversation with a human representative associated with the identified entity to verify whether the particular state corresponds to the target state.

[0140] In some further versions of those embodiments, engaging in the conversation with the human representative to verify whether the particular state corresponds to the target state may include: using a speech recognition model to process audio data capturing the oral utterances of the human representative; and determining, based on processing the audio data, whether the human representative verifies that the particular state corresponds to the target state. Generating the notification requesting the given user associated with the client device to join the assisted phone call may be in response to determining that the human representative verifies that the particular state corresponds to the target state. In other versions of those embodiments, it may further include causing synthetic speech audio data including synthetic speech for verifying whether the IVR tree has been successfully navigated to be rendered at an additional client device associated with the human representative. The audio data capturing the oral utterances of the human representative may be received in response to rendering the synthetic speech audio data at the additional client device associated with the human representative.

[0141] In some further additional or alternative versions of those embodiments, the method may further include: in response to determining that the human representative indicates that the particular state does not correspond to the target state, causing another synthetic speech audio data including another synthetic speech to be rendered at an additional client device associated with the human representative requesting that the assisted phone call be transferred to another human representative associated with the identified entity.

[0142] In some embodiments, the method may further include: identifying, based on the user input or the additional user input, a task to be performed on behalf of the given user during the assisted telephone call; after navigating the IVR tree to the target state and during the assisted telephone call, participating in a conversation with a human representative associated with the identified entity to complete the task on behalf of the given user. Participating in the conversation with the human representative may include: causing synthetic speech audio data including synthetic speech for completing the task to be rendered at an additional client device associated with the human representative, processing audio data capturing the spoken words of the human representative using a speech recognition model, and the audio data is received in response to the rendering of the synthetic speech, and determining a result associated with the execution of the task based on processing the audio data. In some versions of those embodiments, the method may further include: generating a notification based on the result associated with the execution of the task; and causing the notification to be rendered at the client device associated with the given user.

[0143] In some embodiments, the method may further include: in response to determining that the another additional user input received in response to rendering one or more additional prompts in the additional prompts does not modify the navigation of the IVR tree, continuing the execution of the assisted telephone call to navigate the IVR tree to the target state.

[0144] In some embodiments, the method may further include: in response to identifying the IVR tree associated with the identified entity, obtaining one or more prompts related to navigating the IVR tree associated with the identified entity, and causing one or more of the prompts to be rendered at the client device associated with the given user. The additional user input may be received in response to rendering one or more of the prompts.

[0145] In some embodiments, the at least one user interaction at the client device associated with the given user may include one or more of the following: a search interaction at the client device associated with the identified entity, a browsing interaction at the client device associated with the identified entity, or a navigation interaction at the client device associated with the identified entity.

[0146] In some embodiments, automatically navigating the assisted telephone call to the target state of the IVR tree may include one or more of the following: determining one or more values associated with corresponding parameters for navigating the IVR tree based on the user input or the additional user input. In some versions of those embodiments, modifying the navigation of the IVR may include determining an alternative target state of the IVR tree based on the user interaction or the other additional user input. Performing the modification of the assisted telephone call may include automatically navigating the assisted telephone call to the alternative target state of the IVR tree based on one or more of the values associated with the corresponding parameters. In some further versions of those embodiments, modifying the navigation of the IVR may include determining one or more alternative target states of the IVR tree based on the user interaction or the other additional user input. Performing the modification of the assisted telephone call may include automatically navigating the assisted telephone call to the target state of the IVR tree based on one or more alternative values associated with the corresponding parameters.

[0147] In some embodiments, automatically navigating the assisted telephone call to the target state of the IVR tree may include one or more of the following: simulating rendering one or more button presses at an additional client device associated with the identified entity to automatically navigate the IVR tree until the target state is reached; causing synthetic voice audio data including synthetic voice to be rendered at an additional client device associated with the identified entity to automatically navigate the IVR tree until the target state is reached; or causing audio data capturing the spoken words of the given user associated with the client device to be rendered at an additional client device associated with the identified entity.

[0148] In some embodiments, a method implemented by one or more processors is provided, and the method includes: receiving user input from a given user via a search interface of a client device associated with the given user; issuing a search query at the search interface of the client device based on the user input; identifying a given search result associated with an entity to be contacted on behalf of the given user during an assisted telephone call based on the issued search query, the given search result being associated with a target state of an interactive voice response (IVR) tree from among a plurality of candidate states associated with the identified entity; causing at least the given search result to be rendered at the search interface of the client device; receiving a selection of the given search result from the given user via the search interface of the client device in response to the given search result being rendered at the search interface; and initiating execution of the assisted telephone call in response to receiving the selection of the given search result. Execution of the assisted telephone call includes automatically navigating the assisted telephone call to the target state of the IVR tree.

[0149] These and other embodiments of the techniques disclosed herein can optionally include one or more of the following features.

[0150] In some embodiments, the method may further include: identifying an additional given search result associated with the entity based on the issued search query, the additional given search result being associated with an additional target state of the IVR tree from among the plurality of candidate states. In some versions of those embodiments, the method may further include causing the additional given search result to be rendered together with the given search result. In some further versions of those embodiments, causing the additional given search result to be rendered together with the given search result may include rendering the given search result more prominently than the additional given search result.

[0151] In still further versions of those embodiments, rendering the given search result more prominently than the additional given search result can include rendering the given search result more prominently than the additional given search result based on a most recent user interaction of the given user with the client device. In yet further versions of those embodiments, the most recent user interaction of the given user with the client device can include the given user accessing content associated with the given search result. In yet another additional or alternative version of those embodiments, rendering the given search result more prominently than the additional given search result can include rendering the given search result more prominently than the additional given search result based on the popularity of the given search result. In yet further versions of those embodiments, the popularity of the given search result can be based on click-through rates of a plurality of users other than the given user associated with the given search result.

[0152] In some embodiments, a method implemented by one or more processors is provided, and the method includes: receiving, via a client device associated with a given user, user input for initiating an assisted telephone call from the given user; identifying, based on the user input, an entity to be contacted on behalf of the given user during the assisted telephone call; identifying an interactive voice response (IVR) tree associated with the identified entity, the IVR tree including a plurality of candidate states; determining, based on context information associated with the client device or the given user associated with the client device, the availability of the plurality of candidate states as target states of the IVR tree; and determining, based on the user input or additional user input, a target state of the IVR tree from among the available plurality of candidate states; and initiating execution of the assisted telephone call to navigate the IVR tree to the target state. Execution of the assisted telephone call includes automatically navigating the assisted telephone call to the target state of the IVR tree.

[0153] These and other embodiments of the techniques disclosed herein can optionally include one or more of the following features.

[0154] In some embodiments, the context information can be associated with the client device, and the context information can include one or more of the following: a current location of the client device, a current time at the current location of the client device, or a current date at the current location of the client device. In some versions of those embodiments, the given IVR tree selected from among the plurality of different IVR trees can be based on one or more of the following: the current location of the client device, or the current time at the current location of the client device.

[0155] In some embodiments, the context information may be associated with the given user associated with the client device, and wherein the context information may include one or more of the following: an electronic communication created at or received at the client device, user account information associated with the user account of the given user, or money information associated with the given user. In some versions of those embodiments, the given IVR tree selected from among the plurality of different IVR trees may be based on one or more of the following: an electronic communication created at or received at the client device, user account information associated with the user account of the given user, or money information associated with the given user.

[0156] Additionally, some embodiments include one or more processors of one or more computing devices (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU)), wherein the one or more processors are operable to execute instructions stored in an associated memory, and wherein the instructions are configured to cause the execution of any of the above methods. Some embodiments also include one or more non-transitory computer-readable storage media that store computer instructions executable by the one or more processors to execute any of the above methods. Some embodiments also include a computer program product that includes instructions executable by the one or more processors to execute any of the above methods.

[0157] It should be understood that all combinations of the foregoing concepts and additional concepts described in greater detail herein are contemplated as being part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter that appear at the beginning of this disclosure are contemplated as being part of the subject matter disclosed herein.

Claims

1. A method implemented by one or more processors, the method comprising: receiving, via a client device associated with a given user, user input from the given user to initiate an assisted telephone call; identifying, based on the user input, an entity to be contacted on behalf of the given user during the assisted telephone call; identifying an Interactive Voice Response (IVR) tree associated with the identified entity, the IVR tree including a plurality of candidate states; initiating execution of the assisted telephone call; and during execution of the assisted telephone call: determining a corresponding confidence level associated with the IVR tree; determining, based on the corresponding confidence level associated with the IVR tree, a set of candidate states of the IVR tree to be provided for presentation to the user; and causing the set of candidate states of the IVR tree to be provided for presentation to the user.

2. The method according to claim 1, wherein the corresponding confidence level associated with the IVR tree meets a first threshold confidence level, and wherein determining the set of candidate states of the IVR tree to be provided for presentation to the user based on the corresponding confidence level associated with the IVR tree includes: determining that the set of candidate states of the IVR tree to be provided for presentation to the user includes each candidate state of the plurality of candidate states of the IVR tree.

3. The method according to claim 2, wherein causing the set of candidate states of the IVR tree to be provided for presentation to the user includes: causing the set of candidate states of the IVR tree to be provided for presentation to the user before any synthesized speech identifying any candidate state of the IVR tree is rendered by an IVR system using the IVR tree associated with the identified entity.

4. The method according to claim 1, wherein the corresponding confidence level associated with a first subset of the plurality of candidate states of the IVR tree meets a first threshold confidence level, wherein the corresponding confidence level associated with a second subset of the plurality of candidate states of the IVR tree does not meet the first threshold confidence level but meets a second threshold confidence level, and wherein determining the set of candidate states of the IVR tree to be provided for presentation to the user based on the corresponding confidence level associated with the IVR tree includes: determining that the set of candidate states of the IVR tree to be provided for presentation to the user includes the first subset of the plurality of candidate states of the IVR tree but does not include the second subset of the plurality of candidate states of the IVR tree.

5. The method according to claim 4, wherein causing the set of candidate states of the IVR tree to be provided for presentation to the user includes: Before any synthesized speech that identifies any candidate state of the IVR tree associated with the identified entity is rendered by the IVR system using the IVR tree, causing the first subset of the candidate states of the IVR tree to be provided for presentation to the user.

6. The method according to claim 5, wherein, causing the set of candidate states of the IVR tree to be provided for presentation to the user further comprises: after the IVR system renders synthesized speech that identifies the first subset of the candidate states of the IVR tree: causing the second subset of the candidate states of the IVR tree to be provided for presentation to the user.

7. The method according to claim 4, wherein, the first subset of the candidate states of the IVR tree and the second subset of the candidate states of the IVR tree are mutually exclusive.

8. The method according to claim 1, wherein, the corresponding confidence level associated with the IVR tree does not meet a first threshold confidence level and also does not meet a second threshold confidence level, and wherein determining the set of candidate states of the IVR tree to be provided for presentation to the user based on the corresponding confidence level associated with the IVR tree comprises: after the IVR system iteratively renders corresponding synthesized speech that iteratively identifies each candidate state of the IVR tree: iteratively determining that the set of candidate states of the IVR tree to be provided for presentation to the user includes each candidate state of the IVR tree.

9. The method according to claim 8, wherein, causing the set of candidate states of the IVR tree to be provided for presentation to the user further comprises: iteratively causing each candidate state of the IVR tree to be provided for presentation to the user.

10. The method according to claim 1, wherein, the corresponding confidence level associated with the IVR tree does not meet a first threshold confidence level and also does not meet a second threshold confidence level, and wherein determining the set of candidate states of the IVR tree to be provided for presentation to the user based on the corresponding confidence level associated with the IVR tree comprises: after the IVR system renders corresponding synthesized speech that identifies each candidate state of the IVR tree associated with the identified entity: determining that the set of candidate states of the IVR tree to be provided for presentation to the user includes each candidate state of the IVR tree.

11. The method according to claim 10, wherein, causing the set of candidate states of the IVR tree to be provided for presentation to the user further comprises: causing each candidate state of the IVR tree to be provided for presentation to the user.

12. The method according to claim 1, wherein, causing the set of candidate states of the IVR tree to be provided for presentation to the user includes: Cause a corresponding optional graphical element associated with a corresponding candidate state included in the set of candidate states of the IVR tree to be provided for presentation to the user.

13. The method according to claim 12, further comprising: Receiving a user selection of a given corresponding optional graphical element from the corresponding optional graphical elements, the given corresponding optional graphical element being associated with a given candidate state of the IVR tree from the plurality of candidate states of the IVR tree; and Automatically navigating an assisted phone call to the given candidate state of the IVR tree associated with the given corresponding optional graphical element.

14. The method according to claim 12, wherein, Each of the corresponding optional graphical elements includes text content different from the synthesized speech generated by the IVR system that utilizes the IVR tree associated with the identified entity.

15. The method according to claim 1, wherein, The corresponding confidence level associated with the IVR tree is based on the number of assisted phone calls to the identified entity, the assisted phone calls including navigating an IVR tree associated with the identified entity but different from the stored IVR tree associated with the identified entity.

16. The method according to claim 1, further comprising: Causing a transcription of the assisted phone call to be provided for presentation to the user.

17. The method according to claim 16, wherein, The transcription of the assisted phone call is provided for presentation to the user at a first portion of the graphical user interface of the client device, and wherein the set of candidate states of the IVR tree is provided for presentation to the user at a second portion of the graphical user interface of the client device.

18. A method implemented by one or more processors, the method comprising: Receiving user input from a given user via a search interface of a client device associated with the given user; Issuing a search query at the search interface of the client device based on the user input; Identifying, based on the issued search query: A given search result associated with an entity to be subsequently contacted on behalf of the given user during an assisted phone call, the given search result being associated with a deep link pointing to an interactive voice response (IVR) target state, the IVR target state being from a plurality of candidate IVR states of a phone IVR tree associated with the identified entity, and An additional given search result associated with the entity, the additional given search result being associated with an additional deep link pointing to an additional IVR target state among the plurality of candidate IVR states of the phone IVR tree; Causing at least the given search result and the additional given search result to be rendered at the search interface of the client device, wherein causing at least the given search result and the additional given search result to be rendered at the search interface of the client device includes: Determining that the given search result is more relevant to the search query compared to the additional given search result based on one or more terms of the search query; and Based on determining that the given search result is more relevant to the search query than the additional given search result, rendering the given search result more prominently than the additional given search result; In response to rendering the given search result at the search interface of the client device, receiving, via the search interface of the client device, a selection from the given user of a deep link pointing to the IVR target state associated with the given search result; and In response to receiving the selection of the deep link pointing to the IVR target state associated with the given search result, initiating execution of the assisted phone call to the identified entity, wherein execution of the assisted phone call includes automatically navigating the assisted phone call on behalf of the given user to the IVR target state of the phone IVR tree.

19. The method according to claim 18, wherein, Determining that the given search result is more relevant to the search query is further based on the most recent user interaction of the given user with the client device.

20. The method according to claim 19, wherein, The most recent user interaction of the given user with the client device includes the given user accessing content associated with the given search result.

21. The method according to claim 18, wherein, Determining that the given search result is more relevant to the search query is further based on the popularity of the given search result.

22. The method according to claim 21, wherein, The popularity of the given search result is based on the click-through rate of multiple users other than the given user associated with the given search result.

23. The method according to claim 18, wherein, Automatically navigating the assisted phone call to the IVR target state of the phone IVR tree includes: Representing one or more corresponding instances of button presses on behalf of the given user to automatically navigate the assisted phone call to the IVR target state of the phone IVR tree; or Representing one or more corresponding instances of synthesized speech on behalf of the given user to automatically navigate the assisted phone call to the IVR target state of the phone IVR tree.

24. A method implemented by one or more processors, the method comprising: Receiving, via a client device associated with a given user, user input for initiating an assisted phone call from the given user; Identifying, based on the user input, an entity to be contacted on behalf of the given user during the assisted phone call; Identifying an interactive voice response IVR tree associated with the identified entity, the IVR tree including a plurality of candidate states; Determining, based on context information associated with the client device or the given user associated with the client device, the availability of the plurality of candidate states as target states of the IVR tree; and Determining, based on the user input or additional user input, the target state of the IVR tree from among the available plurality of candidate states; and Initiate the execution of the auxiliary telephone call to navigate the IVR tree to the target state, where the execution of the auxiliary telephone call includes automatically navigating the auxiliary telephone call to the target state of the IVR tree.

25. The method according to claim 24, wherein, the context information is associated with the client device, and wherein the context information includes one or more of the following: the current location of the client device, the current time at the current location of the client device, and the current date at the current location of the client device.

26. The method according to claim 24, wherein, a given IVR tree selected from among a plurality of different IVR trees is based on one or more of the following: the current location of the client device, and the current time at the current location of the client device.

27. The method according to claim 24, wherein, the context information is associated with the given user associated with the client device, and wherein the context information includes one or more of the following: an electronic communication created or received at the client device, user account information associated with the user account of the given user, and money information associated with the given user.

28. The method according to claim 24, wherein, a given IVR tree selected from among a plurality of different IVR trees is based on one or more of the following: an electronic communication created or received at the client device, user account information associated with the user account of the given user, and money information associated with the given user.

29. A system, comprising: at least one processor; and a memory storing instructions that, when executed, cause the at least one processor to perform the method according to any one of claims 1 to 28.

30. A non-transitory computer-readable storage medium storing computer-readable instructions that, when executed, cause at least one processor to execute the computer-readable instructions to perform the method according to any one of claims 1 to 28.

Images