Method for Automated Navigation of an Interactive Voice Response (IVR) Tree Representing a Human User

Abstract

Embodiments are directed to utilizing an assistant to automatically navigate an interactive voice response (IVR) tree to reach a target state during an assisted telephone call. The assistant can receive input for initiating an assisted telephone call, and based on the input, identify an entity to be engaged on behalf of a user during the assisted telephone call, and identify an IVR tree stored in association with the entity. In various embodiments, navigation of the IVR tree can be modified based on interactions detected at a client device after initiating the assisted telephone call. In various embodiments, the assisted telephone call can be initiated from a search interface, and a target state can be associated with a given search result. In various embodiments, the IVR tree can be dynamic in that only a subset of candidate states of the IVR tree can be available as target states.

Description

Background Art

[0001] A user can interact with an automated assistant via a variety of computing devices, such as smartphones, tablet computers, wearable devices, automotive systems, standalone personal assistant devices, etc. The automated assistant receives input (e.g., verbal, touch, and / or typed) from the user and responds with responsive output (e.g., visual and / or auditory).

[0002] The user is able to interact with the automated assistant to cause the automated assistant to perform actions on behalf of the user. As an example, the automated assistant is able to navigate an interactive voice response (IVR) system, is able to make a telephone call on behalf of the user to perform a given action, and is able to participate in a conversation with other users to perform actions. For example, the user is able to provide user input that requests the automated assistant to make a restaurant reservation over the phone and on behalf of the user. The automated assistant is able to initiate a telephone call with a specific restaurant, and is able to navigate the IVR system associated with the specific restaurant to make the reservation, and / or is able to provide reservation information to additional people associated with the specific restaurant to make the reservation. Then, when the person associated with the specific restaurant is an active participant in the telephone call, the automated assistant is able to notify the user that a restaurant reservation was successfully made on behalf of the user during the telephone call, and / or causes the user to join the telephone call. Summary of the invention

[0003] Embodiments disclosed herein relate to using an automated assistant to perform an assisted telephone call on behalf of a given user with an entity to navigate an interactive voice response (IVR) tree of an IVR system associated with the entity. The IVR tree of the IVR system associated with the entity can be stored in a database accessible to a client device of the given user. Performing an assisted telephone call to navigate the IVR tree can include automatically navigating the IVR tree to a target state of the IVR tree. Based on user input detected at a client device of the given user, the target state of the IVR tree can be determined from a plurality of. The automated assistant can automatically navigate the IVR tree to the target state of the IVR tree through an instance of synthesized speech, which can be rendered as part of the assisted telephone call, which can include injecting synthesized speech or human speech into the assisted telephone so that the synthesized speech or human speech is audibly perceptible to the IVR system (but not necessarily the given user), and / or an instance of a simulated button press can be rendered as part of the assisted telephone call, which can include injecting a simulated button press into the call so that the IVR system can perceive it. Synthesized speech, human speech, and / or simulated button presses can be determined based on corresponding values ​​of parameters associated with navigating the IVR tree to a target state. The parameters can be stored in association with the IVR tree associated with the entity, and the corresponding values ​​of the parameters can be retrieved from a database and / or requested from a given user of a client device. In some embodiments, the automated assistant can cause the client device to render a notification requesting the user of the client device to join an assisted telephone call. In some additional or alternative embodiments, after automatically navigating the IVR tree to the target state, the automated assistant can optionally perform tasks on behalf of the user. For example, the automated assistant can engage in a conversation with a human representative associated with the entity to perform tasks on behalf of the user.

[0004] Some embodiments relate to modifying the navigation of an IVR tree based on user interactions detected at a client device after initiating an auxiliary telephone call. The automated assistant is capable of generating and rendering prompts related to modifying the navigation of the IVR tree, and is capable of determining whether to modify the navigation of the IVR tree based on user input received in response to the prompts. In some versions of those embodiments, modifying the navigation of the IVR tree may include modifying a target state to an alternative target state or adding an additional target state to be navigated to after navigating to the target state. For example, assume that the automated assistant is navigating to a target state associated with the painting department of an IVR tree associated with a home improvement entity. In this example, if a given user interacts with the home and garden section of a web page associated with the home improvement entity, the user may be prompted to additionally or alternatively navigate to a target state associated with the home and garden department of the IVR tree associated with the home improvement entity. In some additional and / or alternative versions of those embodiments, modifying the navigation of the IVR tree may include modifying the corresponding value of a parameter associated with navigating the IVR tree to the target state. Continuing with the example above, if a given user interacts with a web page associated with a specific brand of paint, the user can be prompted to use the corresponding value associated with the specific brand of paint for the paint brand parameter. In these and other ways, client device resources and / or network resources can be preserved by preventing the restart of another auxiliary phone call performed with respect to an alternative target state and / or alternative value for navigating the IVR tree. In addition, the auxiliary phone call can be ended faster, thereby reducing the total duration of computer and / or network resources for executing the auxiliary phone call. In addition, the battery resources of the client device can also be preserved because the given user is not an active participant in the auxiliary phone call, and the client device can be placed in a locked or low-power state during the auxiliary phone call.

[0005] Some additional or alternative embodiments relate to providing IVR deep links to candidate states of an IVR tree and search results received in response to a search query determined based on user input. In some versions of those embodiments, a given IVR deep link can be provided based on the most recent user interaction at a client device before issuing a search query. For example, assuming that a user of a client device is interacting with search results related to reporting lost luggage to an airline entity, and then issuing a search for an airline entity. In this example, an automated assistant can cause an IVR deep link associated with a target state for reporting lost luggage (and optionally more significantly than other IVR deep links that may also be provided) to be provided for a given user of a client device to select based on recent user interactions. In some additional and / or alternative versions of those embodiments, a given IVR deep link can be provided based on the terms of a search query. For example, assuming that a given user of a client issues a search query for "reporting lost luggage to Example Airlines". In this example, an automated assistant can cause an IVR deep link associated with a target state for reporting lost luggage to be provided for a given user of a client device to select based on the terms of a search query including "reporting lost luggage" and "Example Airlines". In these and other ways, the amount of user input can be reduced, thereby conserving client device resources because a given user need only select an "IVR deep link." Additionally, the assisted phone call can end more quickly, thereby reducing the overall duration that computer and / or network resources are used to execute the assisted phone call. Additionally, battery resources of the client device can also be conserved because a given user is not an active participant in the assisted phone call, and the client device can be placed in a locked or low-power state during the assisted phone call.

[0006] Some additional or alternative embodiments relate to determining the availability of candidate states based on context information associated with a given user's client device and / or context information associated with a given user of the client device. The context information associated with the client device can include, for example, device state information (e.g., on state, off state, locked state, sleep state), date and / or time information associated with the current location of the client device, the current location of the client device, and / or other context information generated by the client device. For example, if a restaurant entity is open, the candidate states associated with the restaurant entity may include a takeout candidate state and a delivery candidate state. However, if the restaurant entity is closed, the candidate states associated with the restaurant entity may not include a takeout candidate state or a delivery candidate state, or if the client device is located outside the delivery radius, the candidate states associated with the restaurant entity may not include a delivery candidate state even if the restaurant is open. In addition, the context information associated with a given user of a client device can include, for example, electronic communications created by a user of the client device or received by a user of the client device, user account information associated with various user accounts of a given user of the client device, money information of a given user of the client device, a photo album of a given user of the client device, a social media profile of a given user of the client device, user preferences of a user of the client device, personal information of a given user of the client device, and / or other information associated with a given user of the client device. For example, if a given user is a gold or above member at a hotel entity, the candidate state associated with the hotel entity can include a contact concierge state or a request for a free upgrade state. However, if a given user is not a gold or above member at a hotel entity, the candidate state can omit these states. In these and other ways, client device resources and / or network resources can be reserved by preventing the restart of another auxiliary phone call executed with respect to a given user selecting a candidate state that may not be available. In addition, the auxiliary phone call can end faster, thereby reducing the total duration of computer and / or network resources used to execute the auxiliary phone call. In addition, the battery resources of the client device can also be reserved because the given user is not an active participant in the auxiliary phone call, and the client device can be placed in a locked or low power state during the auxiliary phone call.

[0007] Some additional or alternative embodiments involve rendering candidate states in various ways based on a confidence level of an IVR tree associated with an entity. The confidence level of the IVR tree stored associated with the entity can be based on a number of assisted telephone calls with the entity that included navigating an IVR tree different from the IVR tree stored associated with the entity, and optionally based on a degree of difference between the navigated IVR tree and the stored IVR tree. For example, if the confidence level of the IVR tree satisfies a first threshold indicating that the automated assistant is highly confident in the IVR tree associated with the entity, the automated assistant can cause a sequence of button presses to be rendered at the client device to inform a given user of the client device how to quickly navigate the IVR tree to one or more candidate states prior to or upon initiating an assisted telephone call (e.g., pressing "1", "3", "5" to report lost luggage), or the automated assistant can cause a selectable graphical element associated with the candidate states to be rendered on the client device (which may not match the sequence of button presses), and, when selected, the selectable graphical element causes the automated assistant to automatically navigate to a target state associated with the selected selectable graphical element prior to or upon initiating an assisted telephone call (e.g., pressing "5" to report lost luggage). As another example, if the confidence level of the IVR tree fails to meet the first threshold but meets the second threshold, indicating that the automated assistant is slightly confident in the IVR tree associated with the entity, then the automated assistant can cause the above-mentioned selectable graphical elements associated with the candidate state (which may not match the sequence of button presses) to be rendered on the client device only when traversing the portion of the IVR tree in which the automated assistant is slightly confident. As yet another example, if the confidence level of the IVR tree fails to meet the first threshold and the second threshold, indicating that the automated assistant is not confident in the IVR tree associated with the entity, then the automated assistant can cause the above-mentioned selectable graphical elements associated with the candidate state (which may not match the sequence of button presses) to be rendered at the client device after the automated assistant perceives the options of the IVR tree. In these and other ways, client device resources and / or network resources can be preserved by preventing the re-initiation of another assisted telephone call in the event that the automated assistant provides incorrect information to the user to navigate the IVR tree. In addition, the assisted telephone call can end more quickly, thereby reducing the total duration of computer and / or network resources used to perform the assisted telephone call by providing a sequence of button presses or selectable elements associated with the candidate state. Additionally, battery resources of the client device may also be preserved since a given user is not an active participant in the assisted phone call, and the client device may be placed in a locked or low power state during the assisted phone call.

[0008] The above description is provided as an overview of only some of the embodiments disclosed herein. These and other embodiments are described herein in additional detail. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 A block diagram of an example environment is depicted that illustrates various aspects of the present disclosure and in which implementations disclosed herein can be implemented.

[0010] Figure 2 Depicted are example interactive voice response (IVR) trees in accordance with various implementations.

[0011] Figure 3 Depicted is a flow diagram illustrating an example method of modifying navigation of an interactive voice response (IVR) tree based on user interaction at a client device associated with a given user in accordance with various implementations.

[0012] Figure 4A , Figure 4B and Figure 4C Depicted are various non-limiting examples of a user interface associated with modifying navigation of an interactive voice response (IVR) tree based on user interaction at a client device associated with a given user in accordance with various implementations.

[0013] Figure 5 Depicted is a flow diagram illustrating an example method of navigating an interactive voice response (IVR) tree from a search interface at a client device associated with a given user in accordance with various implementations.

[0014] Fig. 6A , Figure 6B , Figure 6C , Fig.6D and Fig. 6E Depicted are various non-limiting examples of user interfaces associated with navigating an interactive voice response (IVR) tree from a search interface at a client device associated with a given user, according to various implementations.

[0015] Figure 7 Depicted is a flow chart illustrating an example method of navigating a dynamic interactive voice response (IVR) tree in accordance with various implementations.

[0016] Fig. 8A , Figure 8B , Figure 8C and Fig.8D Depicted are various non-limiting examples of user interfaces associated with navigating a dynamic interactive voice response (IVR) tree in accordance with various implementations.

[0017] Fig. 9 Depicted is an example architecture of a computing device in accordance with various implementations. DETAILED DESCRIPTION

[0018] Now turn to Figure 1 , depicts a block diagram of an example environment that illustrates various aspects of the present disclosure and in which the embodiments disclosed herein may be implemented. Figure 1 A client device 110 is shown in the figure, and in various embodiments includes a user input engine 111, a device state engine 112, a rendering engine 113, a search engine 114, a speech recognition engine 120A1, a natural language understanding (“NLU”) engine 130A1, and a speech synthesis engine 140A1.

[0019] The user input engine 111 can detect various types of user input at the client device 110. The user input detected at the client device 110 can include: verbal input of additional users associated with the entity during the assisted phone call detected via a microphone of the client device 110 and / or additional verbal input transmitted to the client device 110 from additional client devices; touch input detected via a user interface input device (e.g., a touch screen) of the client device 110; and / or typed input detected by a user interface input device (e.g., via a virtual keyboard on the touch screen) of the client device 110. The additional users associated with the entity can be, for example, additional human participants associated with the additional client devices, additional automated assistants associated with the additional client devices of the additional users, and / or other additional users.

[0020] The assisted telephone calls described herein can be performed using various voice communication protocols (e.g., Voice over Internet Protocol (VoIP), Public Switched Telephone Network (PSTN), and / or other telephone communication protocols). As described herein, synthesized speech can be rendered as part of the assisted telephone call, which can include injecting the synthesized speech into the call so that it is perceptible to at least one participant of the assisted telephone call. Also as described herein, simulated button presses can be rendered as part of the assisted telephone call, which can include injecting simulated button presses into the call so that it is perceptible to at least one participant of the assisted telephone call. The synthesized speech can be generated and / or injected by a client device 110 that is one of the endpoints of the call and / or can be generated and / or injected by a server that communicates with the client device 110 and is also connected to the assisted telephone call. Also as described herein, audible output can also be rendered outside of the assisted telephone call, which does not include injecting the audible output into the call, but the audible output can be detected by a microphone of the client device 110 connected to the assisted telephone call and can therefore be perceptible during the call. In some implementations, the secondary phone call can optionally be muted and / or filtering can be used to mitigate the perception of audible output rendered outside of the secondary phone call during the secondary phone call.

[0021] In various embodiments, automated assistant 115 (typically provided by Figure 1 10 (indicated by dashed lines in FIG. 10 ) is capable of performing an assisted telephone call at the client device 110 using the assisted telephone call system 160 over a network 190 (e.g., Wi-Fi, Bluetooth, near field communication, local area network, wide area network, and / or other network). The assisted telephone call system 160 includes a speech recognition engine 120A2, an NLU engine 130A2, a speech synthesis engine 140A2, and an assisted telephone call engine 150 in various embodiments. In some embodiments, the automated assistant 115 is capable of navigating an interactive voice response (IVR) tree of an IVR system associated with an entity on behalf of a user of the client device 110 during an assisted telephone call with an additional user using the assisted telephone call system 160. In some additional or alternative embodiments, the automated assistant 115 is capable of performing tasks on behalf of a user of the client device 110 during an assisted telephone call with an additional user using the assisted telephone call system 160. In some additional or alternative embodiments, the automated assistant 115 can utilize the assisted telephone call system 160 to navigate a non-interactive system (e.g., an on-hold system, a voicemail system) and / or identify whether the IVR system fails or nothing happens during the assisted telephone call (and optionally terminate the assisted telephone call and notify the user).

[0022] In addition, in some embodiments, after navigating the IVR tree and before performing any tasks on behalf of the user of the client device 110, the automated assistant 115 can obtain consent from the additional user to participate in the conversation with the automated assistant 115. For example, the automated assistant 115 can obtain consent to process audio data corresponding to the voice of the human participant when connecting with the human participant. If the automated assistant 115 obtains consent from the additional user, then the automated assistant 115 can verify that the target state of the IVR tree has been reached (e.g., as described below with respect to the verification engine 155), and / or can perform tasks on behalf of the user (e.g., as described below with respect to the task engine 157). However, if the automated assistant 115 does not obtain consent from the additional user, the automated assistant 115 can cause the client device 110 to render (e.g., using the rendering engine 113) to the user of the client device 110 a notification indicating that the user needs to take over the auxiliary telephone call and / or terminate the auxiliary telephone call and render the notification to the user of the client device 110 (e.g., using the rendering engine 113).

[0023] As described in detail below, the automated assistant 115 can perform an assisted telephone call using the assisted telephone call system 160 in response to detecting a user input from a user of the client device 110 to initiate a telephone call using the automated assistant 115. In some embodiments, the automated assistant 115 can determine a target state for navigation of an IVR tree associated with an entity to be approached on behalf of the user of the client device 110 during the assisted telephone call from among a plurality of candidate states based on the user input received at the client device 110. In some versions of those embodiments, the automated assistant 115 can receive a selection of the target state before or when the assisted telephone call is initiated. In some additional or alternative versions of those embodiments, the automated assistant 115 can receive a selection sequence that results in the target state before or when the assisted telephone call is initiated. The automated assistant 115 can navigate the IVR tree until the target state is reached, and can cause a notification to be rendered at the client device 110 (e.g., using the rendering engine 113) that the automated assistant 115 has reached the target state. In various implementations, automated assistant 115 can additionally perform the task on behalf of the user upon reaching the target state, and can cause a notification including rendering (eg, using rendering engine 113 ) at client device 110 of the results of the performance of the task.

[0024] like Figure 1 As shown, the assisted telephone call system 160 can be implemented remotely (eg, via a server and / or other remote client device) and communicate with the client device 110. Figure 1190 and is shown as being remotely implemented and communicating with the client device 110 via the network 190, it should be understood that this is for the sake of example and is not meant to be limiting. For example, in various embodiments, the assisted telephone call system 160 can be implemented locally on the client device 110. As another example, in various embodiments, the assisted telephone call system 160 can be implemented remotely at the assisted telephone call system 160 without any connection to the client device 110 (e.g., a cloud-based assisted telephone call), and the client device 110 can optionally be in a closed state or not connected to any network (e.g., determined using the device state engine 112). In addition, although Figure 1 Automated assistant 115 is shown as being implemented locally on client device 110 and remotely at assisted telephone call system 160, but it should be understood that this is also for the sake of example and is not meant to be limiting. For example, in various embodiments, automated assistant 115 can be implemented locally on client device 110, while in other embodiments, automated assistant 115 can be implemented locally on client device 110 and interact with a separate cloud-based automated assistant (e.g., forming a logical instance of automated assistant 115).

[0025] In embodiments when the user input engine 111 detects a user's spoken input via a microphone of the client device 110 (e.g., during an auxiliary telephone call) and / or receives audio data that captures additional spoken input from an additional user transmitted to the client device 110 from an additional client device, the speech recognition engine 120A1 of the client device 110 can use the speech recognition model 120A to process the audio data that captures the spoken input and / or captures the additional spoken input. The speech recognition engine 120A1 can generate recognized text corresponding to the spoken input and / or the additional spoken input based on the processing of the audio data. In addition, the NLU engine 130A1 of the client device 110 can use the NLU model 130A to process the recognized text generated by the speech recognition engine 120A1 to determine the intent included in the spoken input and / or the additional spoken input. For example, if the client device 110 detects a verbal input from the user of “Call Hypothetical Airline to report lost luggage,” the client device 110 is able to use the speech recognition model 120A to process the audio data capturing the verbal input to generate recognized text corresponding to the verbal input of “Call Hypothetical Airline to report lost luggage,” and is able to use the NLU model 130A to process the recognized text to determine at least a first intent to initiate a call and a second intent to report lost luggage. As another example, if the client device 110 detects an additional verbal input of “Briefly tell us why you are calling,” the client device 110 is able to use the speech recognition model 120A to process the audio data capturing the additional verbal input to generate recognized text corresponding to the additional verbal input of “Briefly tell us why you are calling,” and is able to use the NLU model 130A to process the recognized text to determine the intent of the request natural language input including a request for information associated with why the auxiliary telephone call was initiated (e.g., to report lost luggage). In some versions of those embodiments, client device 110 may be capable of transmitting the audio data, recognized text, and / or intent to assisted telephone calling system 160 .

[0026] In other embodiments, when the user input engine 111 detects a user's spoken input via a microphone of the client device 110 and / or receives audio data that captures additional spoken input from an additional user transmitted to the client device 110 from an additional client device (e.g., during an assisted telephone call and / or during an ongoing call), the automated assistant 115 can cause the client device 110 to send the audio data that captures the spoken input and / or the additional spoken input to the assisted telephone call system 160. The speech recognition engine 120A2 and / or the NLU engine 130A2 of the assisted telephone call system 160 can process the audio data that captures the spoken input and / or the audio data that captures the additional spoken utterances in a manner similar to that described above with respect to the speech recognition engine 120A1 and / or the NLU engine 130A1 of the client device 110. In some additional and / or alternative embodiments, the speech recognition engine 120A1 and / or the NLU engine 130A1 of the client device 110 can be used in a distributed manner in conjunction with the speech recognition engine 120A2 and / or the NLU engine 130A2 of the assisted telephone call system 160. In addition, the speech recognition model 120A and / or the NLU model 130A can be stored locally on the client device 110 and / or remotely at a server that communicates with the client device 110 and / or the assisted telephone call system 160 via the network 190.

[0027] In various embodiments, speech recognition model 120A is an end-to-end speech recognition model, so that speech recognition engine 120A1 and / or 120A2 can use the model to directly generate the recognized text corresponding to the oral input. For example, speech recognition model 120A can be an end-to-end model for generating recognized text on a character-by-character basis (or on other token-by-token basis). A non-limiting example of such an end-to-end model for generating recognized text on a character-by-character basis is a recurrent neural network transducer (RNN-T) model. The RNN-T model is a sequence-to-sequence model in a form that does not adopt an attention mechanism. In other embodiments, speech recognition model 120A is not an end-to-end speech recognition model, so that speech recognition engine 120A1 and / or 120A2 can generate predicted phonemes (and / or other representations) alternatively. For example, the predicted phonemes (and / or other representations) can then be used by speech recognition engine 120A1 and / or 120A2 to determine the recognized text that meets the predicted phonemes. In doing so, speech recognition engines 120A1 and / or 120A2 can optionally employ decoding graphs, dictionaries, and / or other resources.

[0028] In embodiments where the user input engine 111 detects touch and / or typed input via a user interface input device of the client device 110, the automated assistant 115 can cause an indication of the touch input and / or an indication of the typed input to be transmitted from the client device 110 to the assisted telephone call system 160. In some versions of those embodiments, the indication of the touch input and / or the indication of the typed input can include underlying text or representation of the touch input and / or text of the typed input, and the speech recognition model 120A is an end-to-end speech recognition model, the underlying text can be processed using the NLU model 130A to determine the underlying text and / or the intent of the text.

[0029] As described herein, the assisted phone call engine 150 of the assisted phone call system 160 can further process the recognized text generated by the speech recognition engine 120A1 and / or 120A2, the underlying text or representation of the touch input detected at the client device 110, the underlying text of the typed input detected at the client device 110, and / or the intent determined by the NLU engine 130A1 and / or 130A2. In various embodiments, the assisted phone call engine 150 includes an entity recognition engine 151, an interactive voice response (IVR) engine 152, a parameter engine 153, a user interaction engine 154, a verification engine 155, a notification engine 156, and a task execution engine 157.

[0030] The entity recognition engine 151 can identify the entity that the user of the client device 110 wants to contact. The entity can be, for example, a personal entity, a business entity, a location entity, and / or other entities. For example, the personal entity can be a friend entity, a family member entity, a colleague entity, and / or other specific types of personal entities. In addition, the business entity can be a restaurant entity, an airline entity, a hotel entity, a salon entity, a doctor's office entity, and / or other specific types of business entities. In addition, the location entity can be a school entity, a museum entity, a library entity, a park entity, and / or other specific types of location entities. In some embodiments, the entity recognition engine 151 can also determine the specific entity of the identified entity. For example, the specific entity of the personal entity can be the name of the individual (e.g., Jane Doe, John Doe, etc.), the specific entity of the business entity can be the name of the business (e.g., Hypothetical Airlines, Example Airlines, etc.), and the specific entity of the location entity can be the name of the location (e.g., HypotheticalUniversity, Example National Park, etc.). Although the entities described herein can be defined at various levels of granularity, for simplicity they are collectively referred to herein as "entities."

[0031] In some embodiments, entity recognition engine 151 can identify an entity to be approached on behalf of a user of client device 110 based on user interactions with client device 110 prior to initiating an assisted telephone call using automated assistant 115. In some versions of those embodiments, an entity can be identified in response to receiving user input for initiating an assisted telephone call. For example, if a user of client device 110 directs (e.g., verbally or by touch) input to a call interface element of a software application (e.g., associated with a contact in a contact application, associated with a search result in a browser application, and / or associated with other callable entities included in other software applications), entity recognition engine 151 can identify an entity associated with the call interface element. For example, if the user input is directed to a call interface element associated with "HypotheticalAirlines" in a browser application, entity recognition engine 151 can identify "Hypothetical Airlines" (or more generally, a business entity or a restaurant entity) as an entity to be approached on behalf of a user of client device 110 during an assisted telephone call.

[0032] In addition, the entity recognition engine 151 can recognize an entity from among a plurality of entities stored in the entity database 151A. In some embodiments, the plurality of entities stored in the entity database 151A can be indexed by entity and / or by a particular type of entity. For example, if the entity recognition engine 151 recognizes the entity "Hypothetical Airlines", "Hypothetical Airlines" can be indexed as a business entity in the entity database 151A, and can optionally be further indexed as an airline entity. By storing and indexing the recognized entities in the entity database 151A, the entity recognition engine 151 can easily recognize and retrieve the entities, thereby reducing subsequent processing for identifying the entities when encountering the entities in future assisted telephone calls. In addition, in various embodiments, each entity can be associated with a task in the entity database 151A.

[0033] The IVR engine 152 can identify an IVR tree (and optionally, audio data associated with the IVR tree) associated with an entity identified via the entity identification engine 151. In some embodiments, the IVR tree associated with the identified entity can be stored in the IVR tree database 152A in association with the identified entity. For example, the IVR tree stored in the IVR tree database 152A can be mapped to a corresponding entity stored in the entity database 151A. In some additional or alternative embodiments, the IVR tree can be stored in the entity database 151A in association with the corresponding entity, and the IVR tree database 152A can be omitted. Each IVR tree can include multiple corresponding candidate states, and the automated assistant 115 can determine a target state for a given IVR tree from among the multiple corresponding candidate states based on user input detected via the user input engine 111. During an assisted telephone call, the automated assistant can automatically navigate the IVR tree to the target state.

[0034] In some embodiments, and prior to receiving user input for initiating an auxiliary telephone call, automated assistant 115 (and / or additional instances of the automated assistant) can initiate automated telephone calls with multiple entities stored in entity database 151A. During these automated telephone calls, automated assistant 115 (and / or additional instances of the automated assistant) can navigate IVR systems associated with the entities to map out corresponding IVR trees associated with each of the multiple entities, and the mapping of the corresponding IVR trees can be stored in IVR tree database 152A. In some versions of those embodiments, automated assistant 115 (and / or additional instances of the automated assistant) can initiate multiple automated telephone calls to a given entity in the multiple entities to map variations of a given IVR tree associated with the given entity. For example, automated assistant 115 can initiate a first automated telephone call with a restaurant entity when the restaurant entity is open, and can initiate a second automated telephone call with the restaurant entity when the restaurant is closed. In other words, IVR engine 152 can enable automated assistant 115 to map a given IVR tree associated with a restaurant entity in different contexts (e.g., a context associated with a given client device or a context associated with a user) to determine whether the given IVR tree is dynamic. Thus, when IVR engine 152 identifies an IVR tree associated with an entity, it can identify an IVR tree that matches the current context of client device 110 and / or the user of client device 110 (e.g., as referenced in FIG. 1 ). Figure 7 and FIG. 8A to FIG. 8D In some additional or alternative embodiments, and prior to receiving user input for initiating an auxiliary telephone call, the entity can provide a representation of the IVR tree and the IVR tree can be stored in the IVR tree database 152A.

[0035] In various embodiments, automated assistant 115 can track changes to the IVR tree over time, and optionally the extent of the changes, based on assisted telephone calls initiated in response to user input for initiating assisted telephone calls. These changes can be identified based on differences in state between a stored IVR tree and an IVR tree currently being navigated by automated assistant 115 and / or differences in state between stored audio data associated with the IVR tree and audio data associated with the IVR tree currently being navigated by automated assistant 115. For example, if automated assistant 115 determines that the IVR tree associated with a given entity and traversed during the assisted telephone call is different from the IVR tree stored in IVR tree database 152A (e.g., the IVR tree currently being navigated includes a new introduction message, a different internal node or lead node, etc.), automated assistant 115 can store the different state of the IVR tree currently being navigated along with the IVR tree already stored in IVR tree database 152A associated with the given entity. Additionally, the inclusion of the new introduction message can serve as a signal to the automated assistant 115 that the IVR tree may include other downstream changes, and monitor those changes rather than immediately attempt to navigate the IVR tree (if possible). As a result, the automated assistant 115 can. As another example, if the automated assistant 115 determines that the audio data received during navigation of the IVR is different from the audio data stored in association with the IVR tree (e.g., based on comparing acoustic features of the received audio data and the stored audio data and / or comparing recognized text corresponding to the received audio data and the stored audio data), the automated assistant 115 can store the different audio data of the IVR tree currently being navigated along with the IVR tree already stored in the IVR tree database 152A associated with the given entity. If the difference between the stored IVR tree and the navigated IVR tree persists for a threshold number of assisted telephone calls with the given entity, the different IVR tree can replace the IVR tree associated with the given entity. In some versions of those embodiments, a different IVR tree may only be considered for replacement of an IVR tree if a different IVR tree is encountered in the same context (e.g., as determined based on context information associated with client device 110 and / or a user of client device 110). In some versions of those embodiments, a confidence level associated with an IVR tree associated with a given entity may be based on a frequency of matches between IVR trees traversed during an assisted telephone call and IVR trees stored in IVR tree database 152A associated with the IVR trees. In some further versions of those embodiments, and as described with respect to Figures 6B to 6E As described, automated assistant 115 can render multiple candidate states in different ways based on the confidence level and whether the confidence level meets one or more thresholds.

[0036] In some embodiments, parameter engine 153 can identify parameters associated with navigating an IVR tree identified via IVR engine 152 based on an entity identified via entity identification engine 151. The parameters of the IVR tree can be stored in association with the IVR tree identified via IVR engine 152 and can vary based on a navigation path used to navigate the IVR tree to a target state. Automated assistant 115 can navigate the IVR tree using corresponding values ​​of the parameters. In some embodiments, corresponding values ​​of the parameters can be stored in user profile database 153A, and automated assistant 115 can retrieve corresponding values ​​of the parameters from user profile database 153A in response to identifying the parameters associated with navigating the IVR tree and without requesting any values ​​from the user of client device 110. In some additional or alternative versions of those embodiments, automated assistant 115 can request corresponding values ​​of the parameters from the user in response to identifying the parameters associated with navigating the IVR tree and in response to determining that the corresponding values ​​cannot be retrieved from user profile database 153A. For example, automated assistant 115 can initiate a dialog with a user of the client device and cause one or more prompts to be rendered at client device 110 via rendering engine 113 to request corresponding values ​​for any parameters that cannot be resolved using information from user profile database 153A.

[0037] The user profile database 153A can include information about the user from various sources, including, for example, electronic communications associated with the user of the client device 110 (e.g., emails, text messages, voice messages, etc.), user account information associated with various user accounts of the user of the client device 110 (e.g., account status information, account number, account username, account password, etc.), application information from applications installed on the client device 110 and / or accessible by the client device 110 (e.g., smart device applications, note applications, reminder applications, phone applications, calendar applications, and / or other software applications), payment information of the user of the client device 110 (e.g., bank account number, credit card number, payment application information, etc.), a photo album of the user of the client device 110, a social media profile of the user of the client device 110, user preferences of the user of the client device 110, personal information of the user of the client device 110 (e.g., name, phone number, physical address, email address, social security number, date of birth, etc.), and / or other information with restricted access. For example, to navigate the IVR tree associated with the airline entity to a target state associated with canceling a flight reservation, the parameter engine 153 can identify parameters for canceling a flight reservation and can determine values ​​for those parameters. For example, a flight confirmation number parameter can be identified and the flight confirmation number can be retrieved from a flight confirmation email, a frequent flyer number parameter can be identified and the frequent flyer number of the user of the client device 110 associated with the airline whose flight is being canceled can be retrieved from an application associated with the airline. In contrast, to navigate the IVR tree associated with the airline entity to a different target state associated with changing a flight reservation, the parameter engine 153 can identify the above parameters and corresponding values, and can also identify a date or time parameter for the new flight and a destination parameter for the new flight, and can request corresponding values ​​for those parameters from the user of the client device 110.

[0038] For example, briefly refer to Figure 2 , depicting an example IVR tree 200 . Figure 2 The IVR tree 200 depicted in FIG. 1 includes a plurality of nodes connected by a plurality of edges. The plurality of nodes can include, for example, at least one root node and at least one leaf node. For example, Figure 2As shown, the IVR tree includes a root node 211, a plurality of internal nodes 221, 222, and 223, and a plurality of leaf nodes 231, 232, 233, 234, 235, and 236. The leaf nodes 231, 232, 233, 234, 235, and 236 can correspond to a plurality of candidate states, and a given one of the leaf nodes 231, 232, 233, 234, 235, and 236 can correspond to a target state. It is noteworthy that a plurality of nodes are connected by a plurality of corresponding edges. These nodes and / or edges can be stored in the IVR tree database 152A as an IVR tree (and optionally together with audio data associated with each node and / or edge). Each of the corresponding edges can be traversed based on input generated by the automated assistant 115 during the assisted telephone call. For example, the corresponding edge can be traversed based on input corresponding to a numeric indicator that can be simulated as button presses by automated assistant 115 (e.g., pressing "1" to traverse from root node 211 to internal node 1 221, pressing "2" to traverse from root node 211 to internal node 2222, pressing "3" to traverse from root node 211 to leaf node 1 231, etc., as shown by Figure 2 ), or traverse the corresponding edges based on input corresponding to free-form natural language input (e.g., saying "book a flight" to traverse from root node 211 to internal node 1 221, etc.).

[0039] In some embodiments, the free-form natural language input can be synthesized speech audio data generated by the automated assistant 115 that includes synthesized speech to be rendered at an additional client device to traverse along the edges of the IVR tree. For example, the automated assistant can generate synthesized speech audio data that includes an indication of the above-mentioned digital indicator and / or a brief summary of why the auxiliary telephone call is being placed. In some additional or alternative embodiments, the free-form natural language input can be audio data that captures verbal input of a user of the client device 110 and is provided before and / or during the initiation of the auxiliary telephone call. For example, if the IVR tree requires the user to provide verbal utterances to verify the identity of the user of the client device 110 (e.g., using speech recognition), then the audio data can capture the user's verbal input.

[0040] In various embodiments, the automated assistant 115 can track the navigation of the IVR tree 200. By tracking the navigation of the IVR tree 200, the automated assistant 115 can determine when input (e.g., synthesized speech, simulated button presses, and / or audio data) is provided for navigating the IVR tree 200. For example, the automated assistant 115 can track the initiation of an auxiliary telephone call starting from the root node 211 and begin traversing along the edges of the IVR tree 200 to the internal nodes 221, 222, and 223 and / or the leaf nodes 231, 232, 233, 234, 235, and 236 to navigate the IVR tree 200 to a target state determined based on the user input detected at the client device 110. As the automated assistant 115 traverses along the edges to different nodes to reach the target state, the automated assistant 115 can track the traversed edges and / or nodes. In this example, if the IVR tree 200 includes an introductory message before presenting the main menu associated with the root node 211, the automated assistant 115 can determine that the introductory message is new and suppress any input for navigating the IVR tree until the main menu associated with the root node 211 is presented (or any other changes are presented). As another example, the automated assistant 115 can determine that any input to be provided to navigate the IVR tree to a target state may not be provided until the IVR system provides all options for navigating the IVR tree (e.g., the IVR tree does not support providing input at any time). In this example, the automated assistant 115 can suppress any input for navigating the IVR tree until all options for navigating the IVR tree are presented. In some implementations, the automated assistant 115 can return to a previous node traversed during navigation of the IVR tree 200.

[0041] As a non-limiting example, assume that the IVR tree 200 corresponds to an IVR tree associated with a hypothetical entity, Example Airlines. In this example, the root node 211 may correspond to the main menu of the IVR system associated with Example Airlines, the internal nodes may correspond to intermediate states 221, 222, and 223, and the leaf nodes 231, 232, 233, 234, 235, and 236 may correspond to the target states of the IVR tree. Further assume that leaf node 1 231 is associated with the target state of "talk to a representative", leaf node 2 232 is associated with the target state of "upcoming flight information", leaf node 3 233 is associated with the target state of "report lost luggage", leaf node 4 234 is associated with the target state of "book a flight", leaf node 5 235 is associated with the target state of "change flight reservation", and leaf node 6 236 is associated with the target state of "cancel flight reservation". In some implementations, the target state to which automated assistant 115 navigates the IVR tree during the assisted telephone call can be determined based on user input detected via user input engine 111 of client device 110 prior to initiating the assisted telephone call and / or during the assisted telephone call.

[0042] For example, assume that user input indicating that the user wants to report lost luggage is received at client device 110. In this example, automated assistant 115 can determine to navigate to leaf node 3 233 associated with the goal state of "report lost luggage." In this example, automated assistant 115 can know to traverse from root node 211 to interior node 1 221, and from interior node 1 221 to leaf node 3 233 to report lost luggage because IVR tree 200 was previously mapped by automated assistant 115 (and / or additional automated assistants), as described above with respect to IVR engine 152. Upon navigating from root node 211 to leaf node 3 233 and / or upon reaching leaf node 3 233, automated assistant 115 can cause corresponding values ​​of parameters requested when navigating IVR tree 200 to be rendered at additional client devices associated with the entity. In this example, automated assistant 115 can render, at an additional client device associated with the entity, flight information associated with a recent flight (e.g., flight confirmation number, departure destination, arrival destination), corresponding values ​​of personal information (e.g., name and phone number) of a user of client device 110, and / or other corresponding values ​​of the parameters in response to requesting corresponding values ​​of the parameters. Upon reaching leaf node 3 233, automated assistant 115 can optionally generate (e.g., via notification engine 156) a notification to be rendered at client device 110 (e.g., via rendering engine 113) that requests the user to join the auxiliary phone call and / or alerts the user of client device 110 of the results of reporting lost luggage.

[0043] As another example, assume that user input indicating that the user wants to cancel a lost reservation is received at client device 110. In this example, automated assistant 115 can determine to navigate to leaf node 6 236 associated with the goal state of "report lost luggage." In this example, automated assistant 115 can know to traverse from root node 211 to internal node 1 221, from internal node 1 221 to internal node 3 223, and from internal node 3 223 to leaf node 6 236 to cancel the flight reservation because IVR tree 200 was previously mapped by automated assistant 115 (and / or additional automated assistants), as described above with respect to IVR engine 152. Upon navigating from root node 211 to leaf node 6 236 and / or upon reaching leaf node 6 236, automated assistant 115 can cause corresponding values ​​of parameters requested when navigating IVR tree 200 to be rendered at additional client devices associated with the entity. In this example, automated assistant 115 can render, at an additional client device associated with the entity, flight information associated with the upcoming flight (e.g., flight confirmation number, departure destination, arrival destination), corresponding values ​​of account information associated with the Example Airlines account (e.g., frequent flyer number), and / or other corresponding values ​​of the parameters in response to requesting corresponding values ​​of the parameters. Upon reaching leaf node 6 236, automated assistant 115 can optionally generate (e.g., via notification engine 156) a notification to be rendered at client device 110 (e.g., via rendering engine 113) that requests the user to join the auxiliary phone call and / or alerts the user of client device 110 of the result of canceling the flight.

[0044] Although Figure 2 The IVR tree 200 of FIG. 1 is depicted as having certain nodes configured in a certain manner based on corresponding edges, but it should be understood that this is for the sake of example and is not meant to be limiting, and it should be understood that the IVR tree can be configured in almost infinite different ways. In addition, based on certain contexts, some nodes of the IVR tree 200 may or may not be available to the user of the client device 110 (e.g., as described with respect to FIG. 1 ). Figure 7 and Figures 8A to 8D For example, assume that a given IVR tree is associated with a restaurant entity. In this example, the target state associated with placing a takeout order may be available when the restaurant entity is open. Conversely, the target state associated with placing a takeout order may not be available when the restaurant entity is closed.

[0045] Return to reference Figure 1, the user interaction engine 154 can detect user interactions at the client device 110. The user interactions can include, for example, search interactions with a web browser or web browser application, browsing interactions with a web browser or web browser application, navigation interactions with a web browser or navigation application, assistant interactions with the automated assistant 115, and / or other user interactions. In some embodiments, the user interaction engine 154 can detect user interactions of the user of the client device 110 at the client device 110 or at additional client devices in communication with the client device 110 (e.g., via the network 190) after the automated assistant 115 initiates execution of the assisted telephone call. In some versions of those embodiments, the automated assistant 115 can determine whether the detected user interactions are associated with an entity that was engaged on behalf of the user of the client device 110 during the assisted telephone call. If the automated assistant 115 determines that the detected user interactions are associated with an entity, the automated assistant 115 can generate (e.g., via the notification engine 156) one or more prompts that ask the user of the client device 110 whether the navigation of the IVR tree should be modified. In some further versions of those embodiments, modifying the navigation of the IVR tree can include navigating to a different target state. For example, if the automated assistant 115 initiates a secondary telephone call to navigate an IVR tree associated with a home improvement entity to navigate the IVR tree to a target state associated with a painting department of the home improvement entity, and the user of the client device 110 subsequently interacts with a webpage associated with the home and garden department of the home improvement entity, the automated assistant 115 may generate one or more prompts asking the user whether to additionally or alternatively navigate to the target state associated with the home and garden department. In other further versions of those embodiments, modifying the navigation of the IVR tree may include navigating to the same IVR tree with different parameters. For example, if the automated assistant 115 initiates a secondary telephone call to navigate an IVR tree associated with a home improvement entity to navigate the IVR tree to a target state associated with a painting department of the home improvement entity, and the user of the client device 110 subsequently interacts with a webpage associated with a particular paint manufacturer, the automated assistant 115 may generate one or more prompts asking the user whether to utilize the particular paint manufacturer as a corresponding value for a parameter when navigating to the target state associated with the painting department. Below (e.g., with reference to Figure 3 and FIG. 4A to FIG. 4C ) Describes in detail the navigation for modifying the IVR tree.

[0046] In some additional or alternative embodiments, user interaction engine 154 can detect user interactions by a user of client device 110 at client device 110 or at an additional client device in communication with client device 110 (e.g., via network 190) prior to automated assistant 115 initiating execution of the assisted telephone call. In some versions of those embodiments, automated assistant 115 can identify an entity associated with the user interaction, and if an assisted telephone call is subsequently initiated with the identified entity, can cause candidate states of the IVR tree to be rendered based on the user interaction. For example, if a user is browsing a web page associated with a hypothetical airline entity, Example Airlines, researching how to report lost luggage prior to initiating an assisted telephone call with Example Airlines, automated assistant 115 can render (alone or more prominently than other selectable elements) a selectable graphical element associated with a target state associated with reporting lost luggage of the IVR tree associated with Example Airlines. In other words, based on the user researching how to report lost luggage prior to initiating the assisted telephone call, automated assistant 115 can cause the selectable graphical element to be rendered as an "IVR deep link" to the target state of the IVR tree (e.g., as described in relation to Figure 5 and Fig. 6A described above).

[0047] In some embodiments, the verification engine 155 can verify whether the state to which the automated assistant 115 navigates during the assisted telephone call is actually the target state. In some versions of these embodiments, when a human representative associated with the entity joins the assisted telephone call, the automated assistant 115 can process the received audio data that captures the human representative's verbal input to verify that the state to which the automated assistant 115 navigates is actually the target state. For example, assume that the automated assistant 115 is navigating the IVR tree associated with the home improvement entity to the target state associated with the painting department of the home improvement entity. Further assume that the human representative takes over the assisted telephone call from the IVR system associated with the home improvement entity and provides a verbal input of "Thanks for calling the painting department, what can I do to help you today?" In this example, the verification engine 155 can cause the automated assistant 115 to use the speech recognition engine 120A1 and / or 120A2 to process the audio data that captures the verbal input, and can determine that the verbal input includes "painting department" based on processing the audio data that captures the verbal input. In some additional or alternative versions of these embodiments, verbal input may be received in response to synthesized speech audio data including synthesized speech to verify a target state for navigating the IVR tree. Continuing with the above example, assume that a human representative takes over an auxiliary telephone call from an IVR system associated with a home improvement entity and provides a verbal input of "Hello." In this example, the verification engine 155 enables the automated assistant 115 to render the synthesized speech "Hello, is this the painting department?" at an additional client device associated with the human representative. The verification engine 155 enables the automated assistant 115 to process the audio data capturing the verbal input in response to the synthesized speech using the speech recognition engines 120A1 and / or 120A2, and is able to determine whether the verbal input verifies reaching the target state (e.g., "yes," "this is the painting department," or some other confirmation).

[0048] In various embodiments, if the state reached while navigating the IVR does not correspond to the target state, the verification engine 155 can cause the automated assistant 115 to render, at an additional client device associated with the human representative, a synthesized speech requesting that the assisted telephone call be transferred to the target state. Continuing with the above example, if it is determined that the verbal input indicates that the automated assistant 115 has reached a state associated with the home and garden department (rather than the painting department), the verification engine 155 can cause the automated assistant 115 to render, at an additional client device associated with the human representative, a synthesized speech "Can you transfer me to the painting department?" and can again verify that the state after the transfer corresponds to the desired target state of the user of the client device 110.

[0049] The notification engine 156 can generate various notifications based on navigating the IVR tree associated with the identified entity and / or based on tasks performed by the automated assistant 115 on behalf of the user of the client device 110 during the assisted telephone call, without pausing the assisted telephone call. For example, the automated assistant 115 can cause the notification engine 156 to generate a notification requesting the user of the client device 110 to join the assisted telephone call upon reaching a target state. In some implementations, the notification engine 156 can cause the automated assistant 115 to generate a notification requesting the user of the client device 110 to join the assisted telephone call only after the verification engine 155 verifies that the automated assistant has in fact reached the target state, as described above with respect to the verification engine 155. As another example, if the automated assistant 115 cannot resolve the corresponding values ​​of the parameters during the assisted telephone call, the automated assistant 115 can cause the notification engine 156 to generate one or more prompts to request the corresponding values ​​of any unresolved parameters required to continue navigating the IVR tree associated with the entity. As yet another example, if the automated assistant performs a task on behalf of a user of client device 110 during an assisted phone call, automated assistant 115 can cause notification engine 156 to generate a notification that includes the results of the performance of the task.

[0050] In some implementations, the type of notification generated by the notification engine 157 and rendered at the client device 110 via the rendering engine 113 and / or one or more properties of the rendered notification (e.g., volume, brightness, size) can be based on the state of the client device 110 (e.g., determined using the device state engine 112) and / or the state of the ongoing assisted phone call. The state of the ongoing assisted phone call can be based on, for example, a software application running in the foreground of the client device 110, a software application running in the background of the client device 110, whether the client device 110 is in a locked state, whether the client device 110 is in a dormant state, whether the client device 110 is in a closed state, sensor data generated by sensors of the client device 110, and / or other data associated with the client device 110. For example, if the state of the client device 110 indicates that a software application (e.g., an automated assistant application, a phone application, an assisted phone call application, and / or other software application) that displays a transcription of the assisted phone call is running in the foreground of the client device 110, the type of notification can be a banner notification, a pop-up notification, and / or other type of visual notification. As another example, if the state of client device 110 indicates that client device 110 is in a dormant or locked state, the type of notification can be an audible indication via a speaker and / or a vibration via a speaker or other hardware component of client device 110 that simulates a phone call. As yet another example, if sensor data from a presence sensor, accelerometer, and / or other sensor of the client device indicates that the user is not currently nearby and / or is currently holding the client device, a more intrusive notification (e.g., visual and audible at a first volume level) can be provided. On the other hand, if such sensor data indicates that the user is currently nearby and / or is currently holding the client device, a less intrusive notification (e.g., visual only, or visual and audible at a second volume level that is less than the first volume level) can be provided. As yet another example, when the state of a conversation indicates that the conversation is nearing completion, a more intrusive notification can be provided, while when the state of the conversation indicates that the conversation is not nearing completion, a less intrusive notification can be provided.

[0051] Furthermore, in embodiments in which the automated assistant 115 causes a notification to be rendered at the client device 110 requesting the user to join the auxiliary telephone call but the user does not join the auxiliary telephone call within a threshold duration (e.g., 15 seconds, 30 seconds, 60 seconds, and / or other duration), the automated assistant can cause the auxiliary telephone call to terminate. In some versions of these embodiments, the threshold duration can begin when the notification requesting the user to join the auxiliary telephone call is rendered on the client device 110. In some additional or alternative versions of those embodiments, and before terminating the auxiliary telephone call, the state of the auxiliary telephone call can be stored in one or more databases of the client device 110 (e.g., a random access memory of the client device). If the user of the client device 110 subsequently interacts with the notification to join the terminated auxiliary telephone call, the automated assistant 115 can re-initiate execution of the auxiliary telephone call by loading the stored state of the auxiliary telephone call, so that the auxiliary telephone call resumes in a given state, which corresponds to the stored state of the auxiliary telephone call. In this manner, the automated assistant 115 does not need to navigate portions of the IVR tree that have already been navigated by the automated assistant 115.

[0052] In various embodiments, and after the automated assistant 115 navigates to the target state of the IVR tree associated with the entity, the automated assistant can perform the task on behalf of the user of the client device 110. In some versions of these embodiments, the task engine 157 can determine the task to be performed on behalf of the user of the client device 110 based on the user input detected at the client device 110 for initiating the assisted telephone call or based on other user input detected at the client device 110 before the automated assistant 115 initiates the assisted telephone call. For example, if the user of the client device 110 provides a verbal input of "Call Example Café to make a reservation tonight," the task engine 157 can utilize the intent (e.g., determined using the NLU model 130A) to initiate a call and make a restaurant reservation to determine a task to make a restaurant reservation based on the verbal input. The parameters described herein determined by the parameter engine 153 can also identify parameters of the task identified by the task engine 157 and determine corresponding values ​​of the parameters of the task in a similar manner as described above with respect to the parameter engine 153.

[0053] The task engine 157 can enable the automated assistant 115 to participate in a conversation with a human representative associated with the identified entity during an assisted telephone call to perform a task. For example, the task engine 157 can provide text and / or phonemes based on corresponding values ​​of parameters to speech synthesis engines 140A1 and / or 140A2 to generate synthesized speech audio data. The synthesized speech audio data can be transmitted to an additional client device of the human representative for audible rendering at the additional client device. For example, the speech synthesis engine 140A1 and / or 140A2 can determine a phoneme sequence determined to correspond to the value of the parameter requested by the human representative, and can use the speech synthesis model 140a to process the phoneme sequence to generate synthesized speech audio data. The synthesized speech audio data can be, for example, in the form of an audio waveform. When determining the phoneme sequence corresponding to the value of the parameter, the speech synthesis engine 140A1 and / or 140A2 can access a token to phoneme mapping stored locally on the client device 110 or stored at a server (e.g., via the network 190). The task engine 157 can cause the speech recognition engines 120A1 and / or 120A2 to process any audio data capturing the verbal input provided by the human representative, and the automated assistant 115 can then use the speech synthesis engines 140A1 and / or 140A2 to generate further synthesized speech audio data.

[0054] As described herein, the rendering engine 113 can render various notifications or other outputs at the client device 110. The rendering engine 113 can audibly and / or visually render the various notifications described herein. In addition, the rendering engine 113 can cause a transcription of the conversation to be rendered on a user interface of the client device 110. In some implementations, the transcription can correspond to a conversation between a user of the client device 110 and the automated assistant 115. In some additional or alternative implementations, the transcription can correspond to a conversation between an IVR system associated with the identified entity and the automated assistant 115. In other implementations, the transcription can correspond to a conversation between a human representative associated with the identified entity and the automated assistant 115.

[0055] In some embodiments, the user input detected at the client device 110 via the user input engine 111 can be a search query received via a search interface of the client device 110. For example, the search interface can be implemented as part of a phone or contact application, a browser application, an automated assistant application, a web-based browser, and / or any other interface that enables search functionality. The search engine 114 can identify and retrieve search results rendered at the client device 110 via the rendering engine 113. In some versions of these embodiments, the search results can include content responsive to the search query and can optionally include one or more selectable elements. The one or more selectable elements can include, for example, deep links to one or more candidate states of an IVR tree associated with an entity, and when selected based on additional user input (e.g., touch, typing, and / or verbal), the automated assistant 115 navigates to a target state of the IVR tree corresponding to the selected candidate state of the IVR tree included in the search results. For example, if a user of client device 110 submits a search query for "home improvement store" via the search interface of the phone application, selectable elements associated with various departments corresponding to one or more candidate states of the IVR tree associated with home improvement stores can be included in the search results responsive to the search query. Additionally, if the user has recently interacted with other search results associated with "paint" (e.g., as described above with respect to user interaction engine 154), a particular selectable element (e.g., "painting department") can be rendered more prominently than other selectable elements.

[0056] By using the techniques described herein, various technical advantages can be achieved. As a non-limiting example, the automated assistant 115 can end the assisted telephone call more quickly because the automated assistant does not need to hear the options of the IVR system before navigating the IVR tree because it is stored in association with the entity. Network and computing resources can be saved because the length of the assisted telephone call can be reduced by using the techniques disclosed herein. In addition, the amount of user input can be reduced, so that client device resources can be preserved because a given user only needs to select "IVR deep link" in various embodiments to initiate the execution of the assisted telephone call. In addition, battery resources of the client device 110 can also be preserved because the user is not an active participant in the assisted telephone call, and the client device 110 can be placed in a locked or low-power state during the assisted telephone call.

[0057] Now go to Figure 3 , depicts a flow chart illustrating an example method 300 for modifying navigation of an IVR tree based on user interaction at a client device associated with a given user. For convenience, the operations of the method 300 are described with reference to a system that performs the operations. The system of the method 300 includes a computing device (e.g., Figure 1 , Figure 4A To Figure 4D, FIG. 6A to FIG. 6E and / or FIG. 8A to FIG. 8D The client device 110 and / or Fig. 9 One or more processors and / or other components of the computing device 910, one or more servers and / or other computing devices). In addition, although the operations of method 300 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, and / or added.

[0058] At block 352, the system receives user input for initiating an auxiliary telephone call from a given user via a client device associated with the given user. The user input for initiating the auxiliary telephone call can be one or more of a verbal input for initiating the auxiliary telephone call, a touch input for initiating the auxiliary telephone call, or a typed input for initiating the auxiliary telephone call. In addition, the user input for initiating the auxiliary telephone call can be received at various interfaces (e.g., a search interface, a voice interface, an automated assistant interface, and / or other interfaces from which an auxiliary telephone call can be initiated).

[0059] At box 354, the system identifies an entity to be contacted on behalf of a given user during the assisted telephone call based on the user input. The system can identify the entity to be contacted during the assisted telephone call based on the user input received at box 352 and / or the user interaction with the client device immediately before initiating the assisted telephone call conversation. For example, the system can identify the entity of "Example Airlines" based on processing the verbal input "Report my lost luggage to Example Airlines" received at the client device. As another example, the system can identify the entity of "Example Airlines" based on determining that a given user of the client device selected a call interface element associated with Example Airlines or a particular graphical element associated with a candidate state of an IVR tree associated with Example Airlines.

[0060] At block 356, the system identifies an IVR tree associated with the identified entity, the IVR tree comprising a plurality of candidate states. The IVR tree associated with the identified entity can be stored in association with the entity identified at block 354 (e.g., in Figure 1 In the entity database 151A and / or Figure 1The stored IVR tree associated with the entity may have been previously stored in one or more databases based on the system (and / or other instances of the system) previously crawling an IVR system associated with the identified entity, and / or the identified entity is capable of providing an IVR tree for the IVR system associated with the identified entity. Furthermore, the system can continuously update the IVR tree for the IVR system associated with the identified entity based on multiple assisted telephone calls navigating the IVR tree. Multiple candidate states can correspond to specific nodes of the IVR tree that the system is capable of navigating to during an assisted telephone call.

[0061] At box 358, the system determines a target state for the IVR tree from among a plurality of candidate states based on the user input or additional user input. In some embodiments, the target state can be determined based on the user input received at box 352. For example, assume that the IVR tree is associated with an entity of Example Airlines, and assume that a given candidate state of the IVR corresponds to the "report lost luggage" candidate state of the IVR tree associated with the entity of Example Airlines. Further assume that a verbal input of "report my lost luggage to Example Airlines" is received at the client device at box 352. In this example, the target state of "report lost luggage" can be identified based on the verbal input. In some additional or alternative embodiments, the target state can be determined based on additional user input in addition to the user input received at box 352. For example, assume that the IVR tree is associated with an entity of Example Airlines, and assume that a given candidate state of the IVR corresponds to the "lost luggage" candidate state of the IVR tree associated with the entity of Example Airlines. Assume further that a verbal input of "Call Example Airlines" is received at the client device at block 352, or a selection of a selectable graphical element associated with calling Example Airlines is received at the client device at block 352. In this example, one or more of a plurality of candidate states can be presented as output at the client device in response to receiving the user input, and the target state can be a target state based on selecting "Report Lost Baggage" from one or more of the plurality of candidate states presented. The output can be presented before or when the auxiliary telephone call is initiated.

[0062] At box 360, the system initiates execution of the auxiliary telephone call to navigate the IVR tree to the target state. The system can initiate execution of the auxiliary telephone call by establishing a communication session between a client device associated with the given user and an additional client device associated with the entity identified at box 354. The communication session can be established using a variety of voice communication protocols including, for example, VoIP, PSTN, and / or other telephone communication protocols. In some embodiments, the system can navigate the IVR tree to the target state by simulating button presses to traverse the nodes of the IVR tree, by rendering synthesized speech audio data including synthesized speech to traverse the nodes of the IVR tree, and / or by rendering audio data including verbal input of a given user associated with a client device (e.g., as described with respect to FIG. 1 ). Figure 2 In some versions of those embodiments, the system can also utilize corresponding values ​​of parameters associated with navigating the IVR tree. Figure 1 The user profile database 153A) can be retrieved and / or the corresponding value of the parameter can be requested from a given user associated with the client device.

[0063] At box 362, the system determines whether a user interaction is detected at the client device. The user interaction can include, for example, a search interaction with a web browser or web browser application, a browsing interaction with a web browser or web browser application, a navigation interaction with a web browser or navigation application, an assistant interaction with an automated assistant 115, and / or other user interactions with the client device. It is worth noting that when the system performs an auxiliary phone call, a given user associated with the client device can still interact with the client device so that the auxiliary phone call is performed in the background of the client device. If the system determines at an iteration of box 362 that a user interaction is not detected at the client device, the system can proceed to box 370. Box 370 is described in detail below. If the system determines at an iteration of box 362 that a user interaction is detected at the client device, the system can proceed to box 364.

[0064] At box 364, the system determines whether the user interaction detected at box 362 is associated with the entity identified at box 354. For example, the system can determine whether the user interaction was directed to a webpage associated with the entity identified at box 354, directed to obtain directions to a physical store associated with the entity identified at box 354, and / or other user interaction directed to the entity. If the system determines at an iteration of box 364 that the user interaction detected at box 362 is not associated with the entity identified at box 354, the system can proceed to box 370. Box 370 is described in detail below. If the system determines at an iteration of box 364 that the user interaction detected at box 362 is associated with the entity identified at box 354, the system can proceed to box 366.

[0065] At box 366, the system generates one or more prompts related to modifying the navigation of the IVR tree based on the user interaction detected at box 362. In some embodiments, modifying the navigation of the IVR tree can include modifying a target state for navigating the IVR tree to an alternate target state. For example, assume that a secondary telephone call is initiated to navigate to a target state associated with a flooring department of an IVR tree associated with a home improvement entity, and further assume that a user interaction directed to a portion of a web page associated with a lighting department of the home improvement entity is detected. In this example, the system can prompt a given user to change the target state from being associated with the flooring department to an alternate target state associated with the lighting department.

[0066] In some additional or alternative embodiments, modifying the navigation of the IVR tree can include adding additional target states for navigating the IVR tree. Continuing with the example above, instead of prompting a given user to change the target state from being associated with the floor department to an alternative target state associated with the lighting department, the system can prompt the given user to add the lighting department as an additional target state in addition to the target state associated with the floor department. In this example, the user can be notified to join a secondary telephone call upon reaching the target state associated with the floor department. After the given user interacts with the floor department, the given user can terminate the secondary telephone call. However, the system can keep the secondary telephone call connected and can request that a human representative associated with the floor department transfer the secondary telephone call to the lighting department, or the system can re-navigate the IVR tree to the additional target state associated with the lighting department.

[0067] In some additional or alternative embodiments, modifying the navigation of the IVR tree can include modifying corresponding values ​​of parameters used to navigate the IVR tree. For example, assume that an auxiliary telephone call is initiated to navigate to a target state associated with a lighting department of a home improvement entity and an IVR tree associated with the home improvement entity, and further assume that a user interaction directed to a portion of a web page associated with a specific brand of smart light bulbs sold at the home improvement entity is detected. In this example, the system can prompt a given user to include corresponding values ​​associated with the parameters associated with navigating to the target state associated with the lighting department and associated with the specific brand of smart light bulbs. In addition, the system can cause one or more prompts to be rendered visually and / or audibly at a client device associated with the given user.

[0068] At block 368, the system determines whether to modify the navigation of the IVR tree based on the additional user input received in response to the one or more prompts. For example, if the user affirmatively responds to the one or more prompts rendered at the client device, the navigation of the IVR tree can be modified based on the user interaction detected at block 362. However, if the user does not affirmatively respond (or negatively responds) to the one or more prompts, the navigation of the IVR tree may not be modified. If the system determines at an iteration of block 368 not to modify the navigation of the IVR tree based on the additional user input, the system may proceed to block 370.

[0069] At box 370, the system navigates the IVR tree to a target state. In embodiments in which the system proceeds to box 370 from boxes 362, 364, and / or 368, the system can continue to navigate the IVR tree based on the target state determined at box 358. The system can navigate the IVR tree based on simulating a button press to navigate the IVR tree, causing synthesized speech audio data including synthesized speech to be rendered at an additional client device associated with the entity identified at box 354, and / or causing audio data including spoken input to be rendered at an additional client device associated with the entity identified at box 354. If the system determines at an iteration of box 368 to modify navigation of the IVR tree based on additional user input, the system can proceed to box 372. At box 372, the system modifies corresponding values ​​of parameters used to navigate the IVR tree or modifies the target state to additionally include an additional target state or alternatively include an alternate target state, as described above with respect to box 366. In the embodiments described above with respect to box 366, the system modifies corresponding values ​​of parameters used to navigate the IVR tree or modifies the target state to additionally include an additional target state or alternatively include an alternate target state. Figure 2 Describes navigating the IVR tree.

[0070] From box 370 or box 372, the system can proceed to box 374. At box 374, the system can verify whether the auxiliary telephone call has reached the target state. The system can process verbal input received at the client device from a human representative associated with the entity or an additional automated assistant of the IVR system associated with the entity that has joined the call to determine whether the system has reached the target state. For example, assume that the target state of the IVR tree is associated with the lighting department of the home improvement store entity. Further assume that the human representative or the additional automated assistant provides verbal input of "Thank you for calling the lighting department, how can I help you?" In this example, the system can determine that the verbal input includes the term "lighting department" to verify that the system has actually reached the desired target state for the given user associated with the client device. In some embodiments, the verbal input received at the client device (e.g., "Hello, is this the lighting department" or other synthetic speech requesting verification of the target state) can be provided in response to the system requesting verbal input via a synthetic speech rendered at an additional client device associated with the human representative or the additional automated assistant. If the system determines at an iteration of box 374 that the auxiliary telephone call has reached another state other than the target state, the system can proceed to box 376. At box 376, the system requests that the auxiliary telephone call be transferred to a representative associated with the target state. The request to transfer the auxiliary telephone call can be included in a synthesized voice rendered at an additional client device associated with a human representative or an additional automated assistant (e.g., "Can you transfer me to the lighting department" or other synthesized voice requesting the transfer of the auxiliary telephone call). The system can then return to box 374 to verify whether the representative associated with the target state corresponds to the target state after the transfer. If the system determines at an iteration of box 374 that the auxiliary telephone call has reached the target state, the system can proceed to box 378.

[0071] At box 378, the system renders a notification via the client device indicating the result of the execution of the navigating IVR tree. For example, the notification can include a request for a given user associated with the client device to join an auxiliary telephone call. In embodiments described herein where the system also performs tasks on behalf of a given user associated with the client device and with respect to an entity, the notification can additionally or alternatively include the result of the execution of the task.

[0072] Reference now FIG. 4A to FIG. 4C , depicts various non-limiting examples of user interfaces associated with modifying the navigation of an IVR tree based on user interactions at a client device 110 associated with a given user. FIG. 4A to FIG. 4C Each depicts a client device 110 having a graphical user interface 180 and may include Figure 1 One or more components of a client device of the invention. An automated assistant associated with client device 110 (e.g., Figure 1 One or more aspects of the automated assistant 115 may be implemented locally on the client device 110 and / or in a distributed manner (e.g., via Figure 1 The network 190) is implemented on other client devices in network communication with the client device 110. For simplicity, FIG. 4A to FIG. 4C The operations are described herein as being performed by an automated assistant. FIG. 4A to FIG. 4C The client device 110 is depicted as a mobile phone, but it should be understood that this is not intended to be limiting. The client device 110 can be, for example, a stand-alone assistant device (e.g., with speakers and / or a display), a laptop computer, a desktop computer, a vehicle-mounted computing device, and / or any other client device capable of making a phone call.

[0073] FIG. 4A to FIG. 4C The graphical user interface 180 also includes: a text reply interface element 184 that the user can select to generate user input via a virtual keyboard or other touch and / or typing input; and a voice reply interface element 185 that the user can select to generate user input via a microphone of the client device 110. In some embodiments, the user can generate user input via the microphone without selecting the voice reply interface element 185. For example, active monitoring of audible user input via the microphone can be performed to avoid the need for the user to select the voice reply interface element 185. In some of these embodiments and / or other embodiments, the voice reply interface element 185 can be omitted. In addition, in some embodiments, the text reply interface element 184 can be omitted in addition and / or alternatively (for example, the user can only provide audible user input). FIG. 4A to FIG. 4C The graphical user interface 180 also includes system interface elements 181 , 182 , 183 that can be interacted with by a user to cause the computing device 110 to perform one or more actions.

[0074] In various embodiments described herein, user input can be received to initiate a telephone call (e.g., an assisted telephone call) with an entity using an automated assistant. The user input can be a verbal input, a touch input, and / or a typed input including an instruction to initiate an assisted telephone call. In addition, the automated assistant can navigate an IVR tree associated with an entity to a target state from among a plurality of candidate states, and / or perform a task with respect to the entity on behalf of a given user of the client device 110. Figure 4AAs shown, user interface 180 includes search results for a grocery store entity (e.g., as indicated by URL 411 of “www.exampleurl0.com / ”) from a browser application accessible at client device 110. In addition, the search results include a first search result 420 for “Hypothetical Grocer” located at a first position, and a second search result 430 for “Hypothetical Grocer” located at a second position.

[0075] In some embodiments, search results 420 and / or 430 can be associated with various selectable graphical elements that, when selected, cause client device 110 to perform corresponding actions. For example, when call graphical elements 421 and / or 431 associated with a given one of search results 420 and / or 430 are selected, user input can indicate that a telephone call action should be performed to the grocery store entity associated with search results 430 and 420. As another example, when directions graphical elements 422 and / or 432 associated with a given one of search results 420 and / or 430 are selected, user input can indicate that a navigation action should be performed to the grocery store entity associated with search results 430 and 420. As another example, when departments graphical elements 423 and / or 433 associated with a given one of search results 420 and / or 430 are selected, user input can indicate that a browser-based operation should be performed to display the departments of the grocer entity associated with search results 420 and / or 430. Although the assisted telephone call is not performed in Figure 4A In the example, the secondary phone call is initiated from a browser application, but it should be understood that this is for the sake of example and is not intended to be limiting. For example, if the secondary phone call is initiated from the home screen of the client device 110, from the lock screen of the client device 110, and / or other states of the client device 110 using verbal input, the secondary phone call can be initiated from various software applications accessible at the client device 110 (e.g., an automated assistant application, a phone application, a contacts application, an email application, a text or SMS messaging application, and / or other software applications).

[0076] As an example, assume that user input is detected at client device 110 to initiate a phone call using first search result 420 for “Hypothetical Grocer” at a first location. For example, the user input can be a verbal input of “Call Hypothetical Grocer” (and selected based on the proximity of the first location to client device 110) or a touch input directed to call graphical element 421, as indicated by 490A1. In some implementations, call details interface 470 can be rendered at client device 110 before receiving user input to initiate a phone call with “Hypothetical Grocer” or can be rendered at client device 110 in response to receiving user input to initiate a phone call with “Hypothetical Grocer” (as indicated by 490A1). In some versions of those implementations, call details interface 470 can be rendered at client device 110 as part of user interface 180. In some other versions of those embodiments, call details interface 470 can be a separate interface from user interface 180 that overlays user interface 180 and can include call details interface element 186 that allows a user to expand call details interface 470 to display additional call details (e.g., by swiping up on call details interface element 186) and / or close call details interface 470 (e.g., by swiping down on call details interface element 186). Although call details interface 470 is depicted as being located at the bottom of user interface 180, it should be understood that this is for the sake of example and is not intended to be limiting. For example, call details interface 470 can be rendered on top of user interface 180, on the side of user interface 180, or as an interface completely separate from user interface 180.

[0077] In various embodiments, call details interface 470 can include multiple graphical elements. In some versions of these embodiments, the graphical elements can be selectable, such that when a given one of the graphical elements is selected, client device 110 can perform a corresponding action. Figure 4AAs shown, the call detail interface 470 includes a first graphical element 471A for “Assisted Call, Hypothetical Grocer, First Location”, a second graphical element 472A for “Regular Call”, and a third graphical element 473A for “Save Contact'Hypothetical Grocer'”. It is worth noting that in some versions of those embodiments, the graphical element can include sub-elements to provide indications of candidate states for navigating the IVR tree associated with “Hypothetical Grocer”. For example, the first graphical element 471A may include: a first sub-element 471A1 of "Produce Department" associated with a target state of navigating the IVR tree associated with the Hypothetical Grocer to a section corresponding to the produce department; a second sub-element 471A2 of "Bakery Department" associated with a target state of navigating the IVR tree associated with the Hypothetical Grocer to a section corresponding to the bakery department; and a third sub-element 471A3 of "Deli Department" associated with a target state of navigating the IVR tree associated with the Hypothetical Grocer to a section corresponding to the deli department.

[0078] For example, assume that user input is detected at client device 110 to initiate execution of an assisted telephone call with HypotheticalGrocer to navigate an IVR tree associated with the Hypothetical Grocer to a target state associated with the produce department of the Hypothetical Grocer. The user input can be, for example, a verbal input of "Call the Produce Department of the Hypothetical Grocer" or a touch input directed to the first sub-element 471A1, as indicated by 490A2. In response to detecting the user input, the automated assistant can determine a target state for navigating the "Produce Department" of the IVR tree associated with the HypotheticalGrocer. In some embodiments, the automated assistant can identify corresponding values ​​for parameters associated with navigating the IVR tree to the target state and / or corresponding values ​​for tasks to be performed upon reaching the target state associated with the produce department, as described herein (e.g., with respect to Figure 1 Parameters of Engine 153 and about Figure 2 ) as described.

[0079] In some embodiments, and as Figure 4B As shown, while the automated assistant is navigating the IVR tree to a target state during an assisted phone call, a given user of client device 110 may still be able to interact with client device 110. In some versions of those embodiments, user interactions detected at client device 110 while the automated assistant is navigating the IVR tree may be used to modify the navigation of the IVR tree. For example, and as Figure 4B As shown, assume that a given user associated with client device 110 views a web page associated with first search result 420 for “Hypothetical Grocer” at a browser application accessible to client device 110 (e.g., as indicated by “www.exampleurl1.com / ”). In particular, the web page may include selectable graphical elements corresponding to information related to various departments of Hypothetical Grocer. For example, a given user associated with client device 110 may be able to select first graphical selectable element 471B1 to view produce availability at the produce department of Hypothetical Grocer, select second graphical selectable element 471B2 to view cake availability at the bakery department of Hypothetical Grocer, or select third graphical selectable element 471B3 to view meat availability at the deli department of Hypothetical Grocer, or view any other graphical selectable element that may be presented.

[0080] For purposes of example, while the automated assistant is navigating the IVR tree associated with the Hypothetical Grocer to a target state associated with the produce department, assume that the user selects third graphical selectable element 471B3 to view meat availability at the deli department of the Hypothetical Grocer, as indicated by 490B1. In response to determining that a given user associated with client device 110 interacted with a search result related to the deli department of the Hypothetical Grocer while the automated assistant is navigating the IVR tree associated with the Hypothetical Grocer to a target state not associated with the meat department, the automated assistant can generate one or more prompts and render the one or more prompts as notifications at user interface 180 of client device 110. For example, the automated assistant can cause notification 479 to be visually rendered in call detail interface 470. The notification 479 includes an instruction "Would you rather connect with the deli department during the ongoing assisted telephone call?" and also includes a first suggestion 479B1 of "yes" and a second suggestion 479B2 of "no". Figure 4B As shown, and in response to rendering notification 479, additional user input is received at client device 110 indicating that a given user of client device 110 would actually rather be contacted with the deli department rather than the produce department as originally intended (e.g., selecting first suggestion 479B1 of "yes" as indicated by 490B2). In this example, navigation of the IVR tree associated with Hypothetical Grocer can be modified based on user interaction with a web page at client device 110. For example, rather than navigating the IVR to a target state associated with the produce department as originally intended by a given user associated with client device 110, the automated assistant can instead navigate to an alternate target state associated with the deli department.

[0081] In various embodiments, the automated assistant may only prompt a given user of client device 110 to modify navigation of the IVR tree if the user interaction is associated with an entity engaged during the assisted telephone call. For example, a given user of client device 110 may only be prompted to modify navigation of the IVR tree based on a user interaction associated with Hypothetical Grocer. In some additional or alternative embodiments, the automated assistant may still prompt a given user of client device 110 to modify navigation of the IVR tree even if the user interaction is not associated with an entity engaged during the assisted telephone call. For example, a given user of client device 110 may be prompted to modify navigation of the IVR tree based on a user interaction associated with a web page for a particular brand of deli meats other than Hypothetical Grocer. In this example, one or more prompts may also include a coupon or advertisement for the particular brand of deli meats.

[0082] In some implementations, modifying the navigation of the IVR system can be based on navigating the state of the IVR tree. For example, if the automated assistant has not reached a target state when additional user input is received in response to one or more prompts for modifying the navigation of the IVR tree, the automated assistant can modify corresponding values ​​of parameters utilized when navigating the IVR tree to navigate to the target state, modify internal nodes traversed when navigating the IVR tree (e.g., such as those regarding the state of the IVR tree), and modify the state of the IVR tree. Figure 2 As another example, if the automated assistant has already reached a target state when additional user input is received in response to one or more prompts for modifying navigation of the IVR tree, the automated assistant can request that the secondary telephone call be transferred to an alternate target state determined based on the additional user input, and / or restart navigation of the IVR tree.

[0083] For example, and with specific reference to Figure 4C , assume that the automated assistant has navigated the IVR tree to the target state as indicated by the spoken utterance 452C1 “Now connecting you with the produce department” from the Hypothetical Grocer IVR and the subsequent spoken utterance 452C2 “Hello, produce department” from the Hypothetical Grocer produce representative. Further assume that the automated assistant, upon receiving the spoken utterance 452C1 from the Hypothetical Grocer IVR, Figure 4B490B1 indicates that the target state has been reached before the user input for modifying the navigation of the IVR to an alternative target state associated with the prepared food department. Typically, at this stage of navigating the IVR tree associated with the Hypothetical Grocer, the automated assistant can verify that it has reached the target state (e.g., based on subsequent spoken utterance 452C2), generate a notification requesting the user to join the auxiliary telephone call, and / or can perform tasks on behalf of the given user of the client device 110. However, since the given user provided Figure 4B , so that the automated assistant can cause synthesized speech 454C1 of “Hello, can you transfer me to the deli department”, and can receive spoken utterance 452C3 “One moment please” in response to synthesized speech 452C3 when the Hypothetical Grocer produce representative keeps the automated assistant online to transfer the call to the deli department, and the automated assistant can remain online until receiving spoken utterance 452C4 of “Hello, deli department”. Thus, at this stage of navigating the IVR tree associated with the Hypothetical Grocer, the automated assistant can verify that it has reached the target state (e.g., based on spoken utterance 452C4), and generate a notification requesting the user to join the assisted telephone call (e.g., as indicated by 454C2), and / or can perform tasks on behalf of a given user of the client device 110.

[0084] Furthermore, in various embodiments, the automated assistant can cause a transcript of various conversations to be visually rendered at the user interface 180 of the client device 110 (e.g., as Figure 4C). For example, the transcription can be displayed on the home screen of the client device 110, at various software applications (e.g., an automated assistant application, a call application, and / or other applications). In some embodiments, the transcription can include a conversation between the automated assistant, the IVR system, and / or a human representative associated with the entity. In addition, in various embodiments, the automated assistant can also cause various graphical elements to be rendered in the call details interface 470. For example, a first selectable graphical element 442C associated with ending the auxiliary telephone call can be provided, a second selectable graphical element 443C associated with ending the auxiliary telephone call for a given user can be provided, a third selectable graphical element 444C associated with the auxiliary telephone call audibly perceptible via a speaker of the client device 110 can be provided, and / or other selectable graphical elements associated with the auxiliary telephone call can be provided. In various embodiments, these selectable graphical elements can be rendered only at the call details interface 470 in response to generating a notification as indicated by 452C2.

[0085] Although described herein with respect to modifying the navigation of an IVR tree associated with a Hypothetical Grocer from an original destination state associated with a produce department to an alternate destination state associated with a prepared foods department FIG. 4A to FIG. 4C , but it should be understood that this is for the sake of example and is not meant to be limiting. For example, and as described above (e.g., with respect to Figure 1 User interaction engine 154, about Figure 2 IVR tree 200 and about Figure 3 ), modifying the navigation of the navigation IVR tree can include adding a prepared food department as an additional target state that can be navigated to after navigating to an original target state associated with the produce department. As another example, and also as described above (e.g., with respect to Figure 1 User interaction engine 154, about Figure 2 IVR tree 200 and about Figure 3 In method 300 of the embodiment of the present invention, modifying the navigation of the IVR tree can include modifying the corresponding value of the parameter used to navigate the IVR tree to the original target state associated with the produce department. For example, if when navigating to the target state associated with the produce department, a given user indicates that the automated assistant should ask about avocados, but begins viewing tomatoes on the produce section of a webpage associated with the Hypothetical Grocer, the user can be prompted as to whether the automated assistant should additionally or alternatively ask about tomatoes.

[0086] In addition, although Figure 4CThe transcription of the automated assistant depicted as including navigating an IVR tree associated with the Hypothetical Grocer, though it should be understood that this is for the sake of example and is not meant to be limiting. For example, it should be understood that the above-described assisted phone call can be performed while the client device 110 is in a dormant state, a locked state, while other software applications are running in the foreground, and / or in other states. Furthermore, in embodiments in which the automated assistant causes notifications to be rendered at the client device 110, the type of notification rendered at the client device is based on the state of the client device 110, as described herein. Furthermore, while the automated assistant is described herein with respect to navigating an IVR tree associated with the Hypothetical Grocer, FIG. 4A to FIG. 4C , however, it should be understood that this is also not intended to be limiting, and the techniques described herein can be used to navigate any previously stored IVR tree and utilized with respect to a plurality of different entities.

[0087] Now turn to Figure 5 , depicts a flow chart illustrating an example method 500 for navigating an IVR tree from a search interface at a client device associated with a given user. For convenience, the operations of method 500 are described with reference to a system that performs the operations. The system of method 500 includes a computing device (e.g., Figure 1 , Figure 4A To Figure 4D, FIG. 6A to FIG. 6E and / or FIG. 8A to FIG. 8D The client device 110 and / or Fig. 9 The method 500 may be implemented by one or more processors and / or other components of a computing device 910, one or more servers, and / or other computing devices. In addition, although the operations of the method 500 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0088] At box 552, the system receives user input from the given user via a search interface of a client device associated with the given user. The user input for initiating the auxiliary telephone call can be one or more of a verbal input for initiating the auxiliary telephone call, a touch input for initiating the auxiliary telephone call, or a typed input for initiating the auxiliary telephone call. In addition, the user input for initiating the auxiliary telephone call can be received at various interfaces (e.g., a search interface, a voice interface, an automated assistant interface, and / or other interfaces from which the auxiliary telephone call can be initiated). In addition, for example, the search interface can be implemented as part of a phone or contacts application, a browser application, an automated assistant application, a web-based browser, and / or any other interface that enables search functionality.

[0089] At block 554, the system issues a search query based on the user input at a search interface of the client device. The search query can be issued to one or more databases (e.g., restricted access databases and / or public databases), search engines, and / or any other search-based system accessible to the client device.

[0090] At box 556, the system identifies a given search result associated with an entity to be approached on behalf of a given user during an auxiliary telephone call based on the issued search query, and the given search result is associated with a target state of an IVR tree for being associated with the identified entity from among a plurality of candidate states. At box 558, the system at least renders the given search result at the client device. At box 560, the system receives a selection of the given search result from a given user via a search interface of the client device. The given search result can include content in response to the search query issued at box 554. In some embodiments, additional given search results corresponding to other candidate states can also be rendered with the given search result. In some versions of those embodiments, the given search result can be presented more prominently than additional given search results associated with other candidate states based on recent user interactions at the client device. For example, if the user is interacting with other search results related to various brands of paint or painting equipment, and the issued search query is for a home improvement entity, the given search result can be associated with the home improvement entity, or more specifically, the painting department of the home improvement entity. Thus, upon selecting a given search result, the system can determine the painting department as a target state for navigating the IVR tree associated with the home improvement entity. The selection of the given search result can be based on a touch input, a spoken input, and / or a typed input.

[0091] At box 562, the system initiates execution of the auxiliary telephone call in response to the selection to navigate the IVR tree to the target state. The system can initiate execution of the auxiliary telephone call by establishing a communication session between a client device associated with the given user and an additional client device associated with the entity identified at box 556. The communication session can be established using a variety of voice communication protocols including, for example, VoIP, PSTN, and / or other telephone communication protocols. In some embodiments, the system can navigate the IVR tree to a target state by simulating button presses to traverse nodes of the IVR tree, by rendering synthesized speech audio data including synthesized speech to traverse nodes of the IVR tree, and / or by rendering audio data including verbal input of a given user associated with a client device (e.g., as described with respect to FIG. 1 ). Figure 2 In some versions of those embodiments, the system can also utilize corresponding values ​​of parameters associated with navigating the IVR tree. Figure 1The user profile database 153A) can be retrieved and / or the corresponding value of the parameter can be requested from a given user associated with the client device.

[0092] At block 564, the system renders, via the client device, a notification indicating the result of the execution of the navigating IVR tree. For example, the notification can include a request for a given user associated with the client device to join an auxiliary telephone call. In embodiments described herein where the system also performs tasks on behalf of a given user associated with the client device and with respect to an entity, the notification can additionally or alternatively include the result of the execution of the task. In various embodiments, the notification regarding the navigating IVR tree can be modified based on user interactions at the client device. Figure 5 Navigation of the IVR tree as described above Figure 3 and FIG. 4A to FIG. 4C as described.

[0093] Reference now FIG. 6A to FIG. 6E , depicts various non-limiting examples of user interfaces associated with navigating an IVR tree from a search interface at a client device 110 associated with a given user. FIG. 6A to FIG. 6E Each depicts a client device 110 having a graphical user interface 180 and may include Figure 1 The client device 110 may be substantially similar to FIG. 4A to FIG. 4C Client devices, and including the above FIG. 4A to FIG. 4C Many of the same components described, including instances of automated assistants (e.g., Figure 1 Automated assistant 115).

[0094] In various embodiments described herein, user input can be received, and a search query can be issued based on the user input. The user input can be a spoken input, a touch input, and / or a typed input including the search query. For example, and as Fig. 6AAs shown, assume that a user provides a search query 684A (e.g., as included in the text reply interface element 184) for "Lost Baggage Hypothetical Airlines" at a browser application accessible at a client device 110 (e.g., as indicated by URL 611 of "www.exampleurl0.com / "). In addition, the search results include a first search result 620 for "Hypothetical Airline" that provides flights serving North America. In some embodiments, the search results 620 can be associated with various selectable graphical elements that, when selected, cause the client device 110 to perform corresponding actions. For example, when the call graphical element 621 associated with the search result 620 is selected, the user input can indicate that a phone call action should be performed to Hypothetical Airline. As another example, when the user account graphical element 622 associated with the search result 620 is selected, the user input can indicate that an account lookup action should be performed for a user account associated with Hypothetical Airline. As yet another example, when flights graphical element 623 associated with search result 620 is selected, the user input can indicate that a flight lookup action should be performed for flights associated with HypotheticalAirline.

[0095] In various embodiments, the search results 620 can also include one or more IVR deep links to candidate states of the IVR tree associated with the entity. Fig. 6AAs shown, a first IVR deep link 620A1 corresponding to a candidate state for reporting lost luggage when navigating an IVR tree associated with Hypothetical Airline may be provided, and a second IVR deep link 620A2 corresponding to a candidate state for talking to a representative of Hypothetical Airline when navigating an IVR tree associated with Hypothetical Airline may be provided. An additional graphical element 620A3 for "see more" may optionally be provided, and when selected, may cause candidate states associated with one or more additional IVR deep links to be rendered with the first deep link 620A1 and the second deep link 620A2. In some versions of those embodiments, one or more of the rendered IVR deep links 620A1 and / or 620A2 can be based on a recent user interaction at the client device 110. For example, a first IVR deep link 620A1 for “report lost baggage” may be rendered at a client device based on a search query 684A including “lost baggage” or based on a given user of the client device 110 previously interacting with other search results related to reporting lost baggage. As another example, a second IVR deep link 620A2 for “speak with arepresentative” may be rendered at the client device 110 based on it being the most popular target state from among the candidate states. In these examples, the first IVR deep link 620A1 may be rendered more prominently than the second IVR deep link 620A2 based on a determination that it is more relevant to a given user of the client device 110 (e.g., determined based on the search query 684A and / or based on the given user of the client device 110 previously interacting with other search results).

[0096] A secondary telephone call for navigating an IVR tree associated with HypotheticalAirlines to a target state associated with reporting lost luggage can be initiated based on user input directed to the first IVR deep link 620A1, and execution of the secondary telephone call for navigating the IVR tree to the target state can be initiated. In some embodiments, the automated assistant can utilize resources of the client device 110 to initiate execution of the secondary telephone call. In some additional or alternative embodiments, the automated assistant can utilize resources of a remote system to initiate execution of the secondary telephone call (e.g., a cloud-based secondary telephone call). For example, assume that the user selects the first IVR deep link 620A1, but the client device 110 is unable to perform a telephone call using VoIP, PSTN, or other telephone communication protocols (e.g., via a laptop or desktop computer). In this example, the automated assistant 115 can cause a remote system (which may or may not be in communication with the client device 110) to initiate and execute the secondary telephone call on behalf of the user.

[0097] In various embodiments, call details interface 670 can be rendered at client device 110 before receiving user input for initiating a phone call with "Hypothetical Airline," or can be rendered at client device 110 in response to receiving user input for initiating a phone call with Hypothetical Airline. In some versions of those embodiments, call details interface 670 can be rendered at client device 110 as part of user interface 180. In some other versions of those embodiments, call details interface 670 can be a separate interface from user interface 180 that overlays user interface 180, and can include call details interface element 186 that allows a user to expand call details interface 670 to display additional call details (e.g., by swiping up on call details interface element 186) and / or close call details interface 670 (e.g., by swiping down on call details interface element 186). Although call details interface 670 is depicted as being located at the bottom of user interface 180, it should be understood that this is for the sake of example and is not intended to be limiting. For example, call details interface 670 can be rendered on top of user interface 180 , on the side of user interface 180 , or as an interface completely separate from user interface 180 .

[0098] In various embodiments, call details interface 670 can include multiple graphical elements. In some versions of these embodiments, the graphical elements can be selectable, such that when a given one of the graphical elements is selected, client device 110 can perform a corresponding action. Fig. 6AAs shown, the call details interface 670 includes a first graphical element 671 of “Assisted Call, Hypothetical Airline”, a second graphical element 672A of “Regular Call”, and a third graphical element 673A of “Save Contact'Hypothetical Airlines'”. An assisted telephone call can be initiated from the call details interface 670 additionally or alternatively based on a user input directed to the first selectable graphical element 671, and execution of the assisted telephone call is based on selection of the first selectable graphical element 671.

[0099] In various embodiments, one or more IVR deep links associated with a candidate state may not be rendered. In some versions of those embodiments, the automated assistant may visually render various nodes for navigating an IVR tree associated with an entity (e.g., such as a node associated with a candidate state). Figure 2 The visual rendering of various nodes may be based on a confidence level of an IVR tree stored in association with the entity. The confidence level of the IVR tree stored in association with the entity may be based on a number of assisted telephone calls with the entity that included navigating an IVR tree different from the IVR tree stored in association with the entity, and optionally based on a degree of difference between the navigated IVR tree and the stored IVR tree. In particular, FIG. 6B to FIG. 6E Various non-limiting examples of how these various nodes may be presented to a user based on different confidence levels of an IVR tree associated with a Hypothetical Airline are illustrated.

[0100] For about FIG. 6B to FIG. 6E , assuming that the entity to be approached during the assisted telephone call is Fig. 6A , and the user is viewing the first search result 610 associated with Hypothetical Airline, but no user input including the search query 684A has been received. Figure 6B, further assuming that the automated assistant has confidence in the height of the IVR tree stored in association with the Hypothetical Airline. The automated assistant can have confidence in the height of the IVR tree stored in association with the Hypothetical Airline based on the confidence level determined based on the nodes of the IVR tree remaining the same for most, if not all, assisted telephone calls that include the same IVR tree. In this example, rather than providing one or more IVR deep links, the automated assistant can cause information including instructions on how to navigate the IVR tree associated with the Hypothetical Airline prior to receiving user input for initiating execution of the assisted telephone call. For example, the call detail interface 670 may include first information 671A of “Press '1', '3', '5' to report lost baggage” indicating a sequence of button presses that a given user of the client device 110 can provide to reach a target state of the IVR tree associated with reporting lost baggage; and second information 671B of “Press '5' to speak with a representative” indicating a sequence of button presses that a given user of the client device 110 can provide to reach a target state of the IVR tree associated with speaking with a representative. A “see more” graphical element 671C may optionally be provided, and when selected, may cause additional information associated with other candidate states to be rendered together with the first information 671A and the second information 671B. Similar to Fig. 6A , information associated with the candidate states rendered in call detail interface 670 may optionally be rendered based on recent user interactions with client device 110 and / or the overall popularity of the candidate states.

[0101] Specific reference Figure 6C and similar to Figure 6B, further assuming that the automated assistant has high confidence in the IVR tree stored in association with HypotheticalAirline. In this example, rather than providing one or more IVR deep links or information related to direct navigation to a desired target state via a button press sequence prior to initiating the assisted telephone call, the automated assistant is enabled to render the candidate states in a notification 679C in the call detail interface 670 prior to receiving a spoken utterance 654C1 from the Hypothetical Airline IVR including the candidate states, as indicated by 652C1. In particular, the candidate states can include those indicated by 679C1, 679C2, 679C3, 679C4, and 679C5. Each candidate state included in the notification 679C can be selectable, such that a given user associated with the client device 110 can simply select a given one of the candidate states as the target state, and the automated assistant automatically navigates the assisted telephone call to the target state.

[0102] Specific reference Fig.6D And with Figure 6C In contrast, further assume that the automated assistant has moderate confidence in the IVR tree stored associated with HypotheticalAirlines. Based on the confidence level determined based on the nodes of the IVR tree including some differences from some assisted telephone calls that included the same IVR tree, the automated assistant may have moderate confidence in the IVR tree stored associated with HypotheticalAirlines. In this example, rather than providing notification 679D in call detail interface 670 prior to receiving a spoken utterance including a candidate state from the HypotheticalAirlines IVR, the automated assistant may wait until a first portion 652D1 of a spoken response including a subset of the candidate states is provided by the HypotheticalAirlines IVR to render notification 679C including the candidate states. For example, the degree of difference between the stored IVR trees associated with HypotheticalAirlines may be moderately confident only for a subset of the candidate states included in the first portion 652D1 of the spoken utterance, but highly confident for the remaining candidate states. Thus, after the subset of candidate states is included in the first portion 652D1 of the spoken utterance, the automated assistant can generate a notification 679D before receiving the second portion 652D2 of the spoken utterance, as indicated by 654D1. Figure 6C , each candidate state included in notification 679D can be selectable, enabling a given user associated with client device 110 to simply select a given one of the candidate states as a target state, and the automated assistant will automatically navigate the assisted telephone call to the target state.

[0103] Specific reference Fig. 6E And with Figure 6C and Fig.6D In contrast, further assume that the automated assistant has no confidence in the IVR tree stored associated with HypotheticalAirline. Based on a confidence level determined based on a node of the IVR tree including a threshold number of differences in assisted telephone calls that include different IVR trees, the automated assistant may have no confidence in the IVR tree stored associated with HypotheticalAirline. In this example, rather than providing notification 679E in call detail interface 670 prior to receiving spoken utterances, the automated assistant can wait until a spoken response 652E1 including a candidate state is provided by the Hypothetical Airline IVR to render notification 679D including a candidate state, as indicated by 654E1. Similar to Figure 6C , each candidate state included in notification 679E can be selectable, such that a given user associated with client device 110 can simply select a given one of the candidate states as a target state, and the automated assistant will automatically navigate the assisted telephone call to the target state.

[0104] Although this article describes the entity Hypothetical Airline FIG. 6A to FIG. 6E , and certain graphical elements and information are depicted with the search results 620 and call detail interface, but it should be understood that this is for the sake of example and is not intended to be limiting. It should be understood that an IVR tree associated with any entity that employs an IVR system can be stored and subsequently used to perform assisted telephone calls. Furthermore, it should be understood that the graphical elements and information provided by the automated assistant can be based on an IVR tree, and that the configuration of an IVR tree associated with an entity is virtually unlimited.

[0105] Now turn to Figure 7 , depicts a flow chart illustrating an example method 700 of navigating a dynamic interactive voice response (IVR) tree. For convenience, the operations of the method 700 are described with reference to a system that performs the operations. The system of the method 700 includes a computing device (e.g., Figure 1 , Figure 4A To Figure 4D, FIG. 6A to FIG. 6E and / or FIG. 8A to FIG. 8D The client device 110 and / or Fig. 9 The method 700 may be implemented by one or more processors and / or other components of a computing device 910, one or more servers, and / or other computing devices. In addition, although the operations of the method 700 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0106] At block 752, the system receives user input for initiating an auxiliary telephone call from a given user via a client device associated with the given user. The user input for initiating the auxiliary telephone call can be one or more of a verbal input for initiating the auxiliary telephone call, a touch input for initiating the auxiliary telephone call, or a typed input for initiating the auxiliary telephone call. In addition, the user input for initiating the auxiliary telephone call can be received in various interfaces (e.g., a search interface, a voice interface, an automated assistant interface, and / or other interfaces from which an auxiliary telephone call can be initiated).

[0107] At block 754, the system identifies an entity to be approached on behalf of a given user during the assisted telephone call based on the user input. The system can identify the entity to be approached during the assisted telephone call based on the user input received at block 752 and / or user interaction with the client device immediately prior to initiating the assisted telephone call. For example, the system can identify the entity of "Example Airlines" based on processing verbal input received at the client device of "report my lost luggage to Example Airlines." As another example, the system can identify the entity of "Example Airlines" based on determining that a given user of the client device selected a call interface element associated with Example Airlines or a particular graphical element associated with a candidate state of an IVR tree associated with Example Airlines.

[0108] At block 756, the system identifies an IVR tree associated with the entity identified at block 754, the IVR tree including a plurality of candidate states. The IVR tree associated with the identified entity can be stored in association with the entity identified at block 754 (e.g., in Figure 1 Entity database 151A and / or Figure 1 The stored IVR tree associated with the entity may have been previously stored in one or more databases based on the system (and / or additional instances of the system) previously crawling an IVR system associated with the identified entity, and / or the identified entity is capable of providing an IVR tree for the IVR system associated with the identified entity. Furthermore, the system is capable of continuously updating the IVR tree for the IVR system associated with the identified entity based on multiple auxiliary telephone calls navigating the IVR tree. Multiple candidate states can correspond to specific nodes of the IVR tree that the system is capable of navigating to during an auxiliary telephone call.

[0109] At box 758, the system receives context information. In some embodiments, box 756 can include optional sub-box 758A and / or optional sub-box 758B. If included, at optional sub-box 758A, the system receives context information associated with the client device. The context information associated with the client device can include, for example, device state information (e.g., determined by device state engine 112), date and / or time information associated with the current location of the client device, the current location of the client device, and / or other context information generated by the client device. If included, at optional sub-box 758B, the system receives context information associated with a given user of the client device. Contextual information associated with a given user of a client device can include, for example, electronic communications created by or received by the user of the client device (e.g., emails, text messages, voice messages, etc.), user account information associated with various user accounts of the given user of the client device (e.g., account status information, account number, account username, account password, etc.), monetary information of the given user of the client device (e.g., bank account number, credit card number, payment application information, etc.), photo albums of the given user of the client device, social media profiles of the given user of the client device, user preferences of the user of the client device, personal information of the given user of the client device (e.g., name, telephone number, physical address, email address, social security number, date of birth, etc.), and / or other information associated with the given user of the client device.

[0110] At box 760, the system determines the availability of multiple candidate states as target states for the IVR tree based on contextual information associated with the client device and / or a given user of the client device. In various embodiments, based on the contextual information, some candidate states of the IVR tree may or may not be available to a given user of the client device. For example, assume that the entity identified at box 754 is a restaurant entity. In this example, some candidate states may only be available during the restaurant's business hours (e.g., candidate states associated with placing a takeout order). As another example, assume that the entity identified at box 754 is an airline entity. In this example, some candidate states may only be available when the user account state is above a threshold membership state (e.g., only for gold status and above members). In other words, the IVR tree associated with the identified entity can be dynamic because the candidate states may or may not be available in certain contexts.

[0111] At box 762, the system initiates execution of the auxiliary telephone call to navigate the IVR tree to the target state. The system can initiate execution of the auxiliary telephone call in response to additional user input selecting the target state. In addition, the system can initiate execution of the auxiliary telephone call by establishing a communication session between a client device associated with a given user and an additional client device associated with the entity identified at box 754. The communication session can be established using a variety of voice communication protocols including, for example, VoIP, PSTN, and / or other telephone communication protocols. In some embodiments, the system can navigate the IVR tree to a target state by simulating button presses to traverse nodes of the IVR tree, by rendering synthesized speech audio data including synthesized speech to traverse nodes of the IVR tree, and / or by rendering audio data including verbal input of a given user associated with a client device (e.g., as described with respect to FIG. 1 ). Figure 2 In some versions of those embodiments, the system can also utilize corresponding values ​​of parameters associated with navigating the IVR tree. Figure 1 The user profile database 153A) can be retrieved and / or the corresponding value of the parameter can be requested from a given user associated with the client device.

[0112] At block 764, the system renders, via the client device, a notification indicating the result of the execution of the navigating IVR tree. For example, the notification can include a request for a given user associated with the client device to join an auxiliary telephone call. In embodiments described herein where the system also performs tasks on behalf of a given user associated with the client device and with respect to an entity, the notification can additionally or alternatively include the result of the execution of the task. In various embodiments, the notification regarding the navigating IVR tree can be modified based on user interactions at the client device. Figure 5 Navigation of the IVR tree as described above Figure 3 and Figures 4A to 4C described.

[0113] Reference now FIG. 8A to FIG. 8D , depicts various non-limiting examples of user interfaces associated with navigating a dynamic IVR tree. FIG. 8A to FIG. 8D Each depicts a client device 110 having a graphical user interface 180 and may include Figure 1 The client device 110 may be substantially similar to FIG. 4A to FIG. 4C Client devices, and including the above FIG. 4A to FIG. 4C Many of the same components described, including automated assistants (e.g. Figure 1 An instance of an automated assistant 115 ).

[0114] In various embodiments described herein, the IVR tree associated with an entity can be a dynamic IVR tree in that one or more candidate states associated with the IVR tree may or may not be available. The availability of one or more candidate states may be determined based on context information. In some embodiments, the context information may be context information associated with the client device 110, and may include, for example, device state information of the client device 110 (e.g., determined via the device state engine 112), a date and / or time associated with the current location of the client device 110, the current location of the client device, and / or other context information generated by the client device 110. In some additional or alternative embodiments, the contextual information may be contextual information associated with a given user of client device 110, and may include, for example, electronic communications created by or received by the user of the client device (e.g., emails, text messages, voice messages, etc.), user account information associated with various user accounts of the given user of client device 110 (e.g., account status information, account number, account username, account password, etc.), monetary information of the given user of client device 110 (e.g., bank account number, credit card number, payment application information, etc.), photo albums of the given user of client device 110, social media profiles of the given user of client device 110, user preferences of the user of client device 110, personal information of the given user of the client device (e.g., name, phone number, physical address, email address, social security number, date of birth, etc.), and / or other information associated with the given user of the client device.

[0115] Specific reference Fig. 8A and Figure 8B, assume that a given user of client device 110 is viewing search results for a restaurant entity at a browser application accessible at client device 110 (e.g., as indicated by URL 811A of "www.exampleurl0.com / "). Further assume that the search results include at least search result 820A for "Hypothetical Restaurant" that is open from 9:00 a.m. to 11:00 p.m. In some embodiments, search result 820A can be associated with various selectable graphical elements that, when selected, cause client device 110 to perform corresponding actions. For example, when call graphical element 821A associated with search result 820A is selected, user input can indicate that a phone call action to the Hypothetical Restaurant should be performed. As another example, when directions graphical element 822A associated with search result 820A is selected, user input can indicate that a navigation action to the physical location of the Hypothetical Restaurant should be performed. As yet another example, when the menu graphical element 823A associated with the search result 820A is selected, the user input can indicate that a menu search action should be performed for a menu associated with Hypothetical Airline.

[0116] In various embodiments, call details interface 870 can be rendered at client device 110 before receiving user input for initiating a phone call with "HypotheticalRestaurant", or can be rendered at client device 110 in response to receiving user input for initiating a phone call with "HypotheticalRestaurant". In some versions of those embodiments, call details interface 870 can be rendered at client device 110 as part of user interface 180. In some other versions of those embodiments, call details interface 870 can be a separate interface from user interface 180 that overlays user interface 180, and can include call details interface element 186 that allows a user to expand call details interface 870 to display additional call details (e.g., by swiping up on call details interface element 186) and / or close call details interface 870 (e.g., by swiping down on call details interface element 186). Although call details interface 870 is depicted as being located at the bottom of user interface 180, it should be understood that this is for the sake of example and is not intended to be limiting. For example, call details interface 870 can be rendered on top of user interface 180 , on the side of user interface 180 , or as an interface completely separate from user interface 180 .

[0117] In various embodiments, call details interface 870 can include multiple graphical elements. In some versions of these embodiments, the graphical elements can be selectable, such that when a given one of the graphical elements is selected, client device 110 can perform a corresponding action. Fig. 8A and Figure 8B As shown, the call details interface 870 includes a first graphical element 871A for “Assisted Call, Hypothetical Restaurant”, a second graphical element 872A for “Regular Call”, and a third graphical element 873A for “Save Contact'Hypothetical Restaurant'”. The graphical elements may also include child elements corresponding to candidate states associated with the IVR tree associated with the HypotheticalRestaurant. For example, and as Fig. 8A As shown, the first graphical element 871A can include: a first sub-element 871A1 associated with "Make Reservations" associated with navigating the IVR tree associated with the Hypothetical Restaurant to a target state corresponding to making a restaurant reservation; a second sub-element 871A2 associated with "Order Carryout" associated with navigating the IVR tree associated with the Hypothetical Restaurant to a target state corresponding to ordering takeout from the restaurant; and a third sub-element 871A3 associated with "Order Delivery" associated with navigating the IVR tree associated with the Hypothetical Restaurant to a target state corresponding to ordering delivery from the Hypothetical Restaurant. In contrast, and as Figure 8B As shown, the first graphical element 871A may include: a first sub-element 871B1 of "MakeReservations" associated with a target state of navigating the IVR tree associated with the Hypothetical Restaurant to a portion corresponding to making a restaurant reservation; and a second sub-element 871B2 of "Hours of Operation" associated with a target state of navigating the IVR tree associated with the Hypothetical Restaurant to a portion corresponding to listening to the business hours of the Hypothetical Restaurant.

[0118] It is worth noting that, including Fig. 8A and8B The difference in the candidate states in may be based on contextual information associated with a given user's client device 110. It is worth noting that, and with reference to Fig. 8A , the time and date information 812A at the current location of the client device 110 indicates a time of 9:30 AM, and this corresponds to the HypotheticalRestaurant being open for business (e.g., based on the business hours being 9:00 AM to 11:00 PM). Therefore, the candidate states associated with the selectable child elements 871A2 and 871A3 may be available because the Hypothetical Restaurant is open for business. Conversely, and with reference to Figure 8B , time and date information 812B at the current location of client device 110 indicates a time of 8:30 AM, and this corresponds to the Hypothetical Restaurant being closed (e.g., based on the business hours being 9:00 AM to 11:00 PM). Thus, the candidate states associated with selectable sub-elements 871A2 and 871A3 may not be available because the Hypothetical Restaurant is closed, but the automated assistant may still be able to make a restaurant reservation and / or inquire about business hours by navigating the IVR tree associated with the HypotheticalRestaurant, even though it is closed. Thus, the IVR tree may be dynamic in that some candidate states may be available in a given context, while other candidate states may not be available in a given context. As another example, assume that in Fig. 8A The Hypothetical Restaurant is open for business, but the current location of the client device 110 is outside the delivery radius of the Hypothetical Restaurant. In this example, the candidate state associated with the third sub-element 871A3 of "Order Delivery" can be omitted. As another example, suppose that Fig. 8A The Hypothetical Restaurant is open, but indoor and outdoor dining services are not provided at the Hypothetical Restaurant. In this example, the candidate state associated with the first sub-element 871A1 of "Make Reservations" can be omitted.

[0119] Specific reference Figure 8C and Fig.8D, assuming that a given user of client device 110 is viewing search results for a hotel entity on a browser application accessible at client device 110 (e.g., as indicated by URL 811C of "www.exampleurl0.com / "). It is further assumed that the search results include at least a first search result 820C of "Hypothetical Hotel" and a second search result 830C of "Example Hotel". In some embodiments, search results 820C and / or 830C can be associated with various selectable graphical elements that, when selected, cause client device 110 to perform corresponding actions. For example, when call graphical elements 821C and / or 831C, respectively associated with search results 820A and / or 830C, are selected, the user input can indicate that a telephone call action should be made to the corresponding hotel entity. As another example, when the directions graphical elements 822C and / or 832C associated with the search results 820A and / or 830C, respectively, are selected, the user input can indicate that a navigation action to the physical location of the corresponding hotel entity should be performed. As yet another example, when the account elements 823C and / or 833C associated with the search results 820A and / or 830C, respectively, are selected, the user input can indicate that an account lookup action to a user account associated with the corresponding hotel entity should be performed.

[0120] In various embodiments, the call details interface 870 can be rendered at the client device 110 before receiving a user input for initiating a phone call with a corresponding one of the hotel entities, or can be rendered at the client device 110 in response to receiving a user input for initiating a phone call with a corresponding one of the hotel entities. In some versions of those embodiments, the call details interface 870 can be rendered at the client device 110 as part of the user interface 180. In some other versions of those embodiments, the call details interface 870 can be an interface diagram separate from the user interface 180 that overlays the user interface 180, and can include a call details interface element 186 that allows the user to expand the call details interface 870 to display additional call details (e.g., by sliding up on the call details interface element 186) and / or close the call details interface 870 (e.g., by sliding down on the call details interface element 186). Although the call details interface 870 is depicted as being located at the bottom of the user interface 180, it should be understood that this is for the sake of example and is not intended to be limiting. For example, call details interface 870 can be rendered on top of user interface 180 , on the side of user interface 180 , or as an interface completely separate from user interface 180 .

[0121] In various embodiments, call details interface 870 can include multiple graphical elements. In some versions of these embodiments, the graphical elements can be selectable graphical elements such that when a given one of the graphical elements is selected, client device 110 can perform a corresponding action. Figure 8C As shown, the call detail interface 870 includes a first graphic element 871C for “Assisted Call, Hypothetical Hotel”, a second graphic element 872C for “Regular Call”, and a third graphic element 873C for “Save Contact 'Hypothetical Hotel'”. Figure 8C The graphical element may also include child elements corresponding to candidate states associated with the IVR tree associated with the HypotheticalHotel. Figure 8C As shown, the first graphical element 871C may include: a first sub-element 871C1 of "Change Reservation" associated with a target state of navigating the IVR tree associated with the Hypothetical Hotel to a portion corresponding to changing a hotel reservation; a second sub-element 871C2 of "Request Free Upgrade" associated with a target state of navigating the IVR tree associated with the Hypothetical Hotel to a portion corresponding to requesting a free upgrade for an existing hotel reservation; and a third sub-element 871C3 of "Concierge Direct Line" associated with a target state of navigating the IVR tree associated with the Hypothetical Hotel to a portion corresponding to contacting the concierge of the Hypothetical Hotel.

[0122] In contrast, and as Fig.8D As shown, the call details interface 870 includes: "Assisted Call, Example Hotel (Assisted Call, Example Hotel)" (such as Figure 8C a first graphical element 871D for “Hypothetical Hotel” in the example above); a second graphical element 872D for “Regular Call”; and a third graphical element 873D for “Save Contact'ExampleHotel'”. Fig.8D The graphical element may also include child elements corresponding to candidate states associated with the IVR tree associated with Example Hotel. Fig.8D As shown, the first graphical element 871D may include: a first sub-element 871D1 associated with "Change Reservation" associated with navigating the IVR tree associated with Example Hotel to a target state corresponding to changing the portion of the hotel reservation; a second sub-element 871D2 associated with "Cancel Reservation" associated with navigating the IVR tree associated with Example Hotel to a target state corresponding to canceling the existing hotel reservation; and a third sub-element 871D3 associated with "Front Desk" associated with navigating the IVR tree associated with Hypothetical Hotel to a target state corresponding to contacting the front desk at Example Hotel.

[0123] It is worth noting that Figure 8C and Fig.8D The differences in the candidate states included in may be based on contextual information associated with a given user of the given user's client device 110. It is worth noting that, and with reference to Figure 8C , the automated assistant is able to access a user account of a given user of client device 110 associated with the Hypothetical Hotel. For example, the automated assistant is able to access the user account status (e.g., bronze member, silver member, gold member, etc.), the user account number, the user account history including stays at various locations of the Hypothetical Hotel, etc. For the sake of example, assume that the given user of client device 110 is a gold member of the Hypothetical Hotel. As a result, candidate states associated with selectable sub-elements 871C2 and 871C3 may be available based on the gold member status of the given user of client device 110. In contrast, and with reference to Fig.8D , the automated assistant may not have any record of the user account associated with Example Hotel. As a result, the candidate states associated with selectable child elements 871C2 and 871C3 may not be available, but the automated assistant may still be able to navigate to Fig.8D The various candidate states are depicted in . Thus, the IVR tree may be dynamic in that some candidate states may be available in a given context while other candidate states may not be available in a given context.

[0124] Although described herein with respect to the availability of candidate states based on contextual information associated with a given user's client device 110, Fig. 8A and Figure 8B , and are described herein with respect to the availability of candidate states based on contextual information associated with a given user of client device 110 Figure 8C and Fig.8D , but it should be understood that this is for the sake of example and is not intended to be limiting. For example, the availability of candidate states can be based on contextual information associated with a given user's client device 110 and contextual information associated with the given user of client device 110. For example, and referring to 8C, if the given user of client device 110 initiates an auxiliary telephone call when the concierge of the Hypothetical Hotel is not working, then the candidate state associated with the third sub-element 871C3 of "Concierge Direct Line" can be omitted, even if the user account state of the user account associated with the Hypothetical Hotel enables the given user of client device 110 to directly contact the concierge of the Hypothetical Hotel.

[0125] Furthermore, although certain entities are described herein FIG. 8A to FIG. 8D , and certain graphical elements are depicted with the search results and call detail interfaces, but it should be understood that this is for the sake of example and is not intended to be limiting. It should be understood that an IVR tree associated with any entity that employs an IVR system can be stored and subsequently used to perform assisted telephone calls. Furthermore, it should be understood that the graphical elements and information provided by the automated assistant can be based on an IVR tree, and that the configuration of an IVR tree associated with an entity is virtually unlimited.

[0126] Fig. 9 910 is a block diagram of an example computing device 910 that can optionally be used to perform one or more aspects of the techniques described herein. In some implementations, one or more of the client device, cloud-based automated assistant components, and / or other components can include one or more components of the example computing device 910.

[0127] The computing device 910 typically includes at least one processor 914 that communicates with a number of peripheral devices via a bus subsystem 912. These peripheral devices may include: a storage subsystem 924 (including, for example, a memory subsystem 925 and a file storage subsystem 926), a user interface output device 920, a user interface input device 922, and a network interface subsystem 916. The input and output devices allow a user to interact with the computing device 910. The network interface subsystem 916 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0128] The user interface input devices 922 may include a keyboard, a pointing device such as a mouse, trackball, touchpad or graphics tablet, a scanner, a touch screen incorporated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways to input information into the computing device 910 or over a communication network.

[0129] The user interface output device 920 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display, for example, via an audio output device. Typically, the use of the term "output device" is intended to include devices and methods of all possible types of information output from the computing device 910 to a user or another machine or computing device.

[0130] The storage subsystem 924 stores programs and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 924 may include programs for executing selected aspects of the methods disclosed herein and implementing Figure 1 The logic of the various components is depicted in .

[0131] These software modules are typically executed by the processor 914 alone or in combination with other processors. The memory 925 used in the storage subsystem 924 can include multiple memories, including a main random access memory (RAM) 930 for storing instructions and data during program execution and a read-only memory (ROM) 932 in which fixed instructions are stored. The file storage subsystem 926 can provide permanent storage for program and data files, and can include a hard drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media box. Modules that implement the functions of certain embodiments can be stored in the storage subsystem 924 by the file storage subsystem 926, or in other machines accessible to the processor 914.

[0132] The bus subsystem 912 provides a mechanism for the various components and subsystems of the computing device 910 to communicate with each other as intended. Although the bus subsystem 912 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0133] The computing device 910 can be of various types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, for the purpose of illustrating some embodiments, Fig. 9The description of computing device 910 depicted in FIG. 1 is intended only as a specific example. Many other configurations of computing device 910 may have more Fig. 9 The computing devices depicted may have greater or fewer components.

[0134] In cases where the systems described herein collect or otherwise monitor personal information about a user or may utilize personal and / or monitoring information, the user may be provided with an opportunity to control whether a program or feature collects user information (e.g., information about a user's social network, social behavior or activity, occupation, user preferences, or a user's current geographic location), or to control whether and / or how content that may be more relevant to the user is received from a content server. In addition, prior to storing or using specific data, it may be processed in one or more ways so that personally identifiable information is removed. For example, the user's identity may be processed so that personally identifiable information about the user cannot be determined, or the user's geographic location may be summarized (such as to a city, zip code, or state level) where geographic location information is obtained so that the user's specific geographic location cannot be determined. Thus, the user may control how information about the user is collected and / or used.

[0135] In some implementations, a method implemented by one or more processors is provided, and the method includes: receiving user input from a given user via a client device associated with the given user for initiating an auxiliary telephone call; identifying an entity to be engaged on behalf of the given user during the auxiliary telephone call based on the user input; identifying an interactive voice response (IVR) tree associated with the identified entity, the IVR tree including a plurality of candidate states; determining a target state of the IVR tree from among the plurality of candidate states based on the user input or additional user input; and initiating execution of the auxiliary telephone call to navigate the IVR tree to the target state. Execution of the auxiliary telephone call includes automatically navigating the auxiliary telephone call to the target state of the IVR tree. The method further includes: during execution of the auxiliary telephone call, and when the IVR tree is navigated to the target state, detecting at least one user interaction directed to the client device and associated with the identified entity, and generating one or more prompts related to modifying navigation of the IVR tree based on the at least one user interaction directed to the client device. The method further includes modifying navigation of the IVR tree in response to determining that another additional user input is received in response to rendering one or more of the additional prompts, modifying execution of the assisted telephone call based on the another additional user input.

[0136] These and other implementations of the technology disclosed herein can optionally include one or more of the following features.

[0137] In some embodiments, the method may further include: after navigating the IVR tree to reach a particular state during the assisted telephone call, and based on the another additional user input, generating a notification requesting the given user associated with the client device to join the assisted telephone call, and causing the notification to be rendered at the client device associated with the given user. In some versions of those embodiments, causing the notification to be rendered at the client device associated with the given user may include one or more of: causing the client device to vibrate, causing the client device to audibly render the notification requesting the given user to join the assisted telephone call, or causing the client device to visually render the notification requesting the given user to join the assisted telephone call. In some additional or alternative versions of those embodiments, the method may further include, after navigating the IVR tree to the particular state, engaging in a conversation with a human representative associated with the identified entity to verify whether the particular state corresponds to the target state.

[0138] In some further versions of those embodiments, engaging in the conversation with the human representative to verify whether the particular state corresponds to the target state may include: using a speech recognition model to process audio data that captures the spoken utterances of the human representative; and determining whether the human representative verifies that the particular state corresponds to the target state based on processing the audio data. Generating the notification requesting the given user associated with the client device to join the auxiliary telephone call may be in response to determining that the human representative verifies that the particular state corresponds to the target state. In other versions of those embodiments, it may further include causing synthetic speech audio data including synthetic speech for verifying whether the IVR tree is successfully navigated to be rendered at an additional client device associated with the human representative. The audio data capturing the spoken utterances of the human representative may be received in response to rendering the synthetic speech audio data at the additional client device associated with the human representative.

[0139] In some further additional or alternative versions of those embodiments, the method may further include: in response to determining that the human representative indicates that the particular state does not correspond to the target state, rendering another synthesized speech audio data including another synthesized speech at an additional client device associated with the human representative requesting that the auxiliary telephone call be transferred to another human representative associated with the identified entity.

[0140] In some embodiments, the method may further include: identifying a task to be performed on behalf of the given user during the assisted telephone call based on the user input or the additional user input; and after navigating the IVR tree to the target state and during the assisted telephone call, engaging in a conversation with a human representative associated with the identified entity to complete the task on behalf of the given user. Engaging in the conversation with the human representative may include: causing the rendering of synthesized speech audio data including synthesized speech for completing the task at an additional client device associated with the human representative, processing audio data capturing spoken utterances of the human representative using a speech recognition model, and the audio data being received in response to the rendering of the synthesized speech, and determining a result associated with the performance of the task based on processing the audio data. In some versions of those embodiments, the method may further include: generating a notification based on the result associated with the performance of the task; and causing the rendering of the notification at the client device associated with the given user.

[0141] In some embodiments, the method may further include: in response to determining that the other additional user input received in response to rendering one or more of the additional prompts does not modify the navigation of the IVR tree, continuing the execution of the auxiliary telephone call to navigate the IVR tree to the target state.

[0142] In some implementations, the method may further include, in response to identifying the IVR tree associated with the identified entity, obtaining one or more prompts related to navigating the IVR tree associated with the identified entity, and causing one or more of the prompts to be rendered at the client device associated with the given user. The additional user input may be received in response to rendering one or more of the prompts.

[0143] In some embodiments, the at least one user interaction at the client device associated with the given user may include one or more of: a search interaction at the client device associated with the identified entity, a browsing interaction at the client device associated with the identified entity, or a navigation interaction at the client device associated with the identified entity.

[0144] In some embodiments, automatically navigating the auxiliary telephone call to the target state of the IVR tree may include one or more of the following: determining one or more values ​​associated with a corresponding parameter for navigating the IVR tree based on the user input or the additional user input. In some versions of those embodiments, modifying the navigation of the IVR may include determining an alternative target state of the IVR tree based on the user interaction or the other additional user input. Execution of the modification of the auxiliary telephone call may include automatically navigating the auxiliary telephone call to the alternative target state of the IVR tree based on one or more of the values ​​associated with the corresponding parameter. In some further versions of those embodiments, modifying the navigation of the IVR may include determining one or more alternative target states of the IVR tree based on the user interaction or the other additional user input. Execution of the modification of the auxiliary telephone call may include automatically navigating the auxiliary telephone call to the target state of the IVR tree based on one or more alternative values ​​associated with the corresponding parameter.

[0145] In some embodiments, automatically navigating the auxiliary telephone call to the target state of the IVR tree may include one or more of the following: simulating rendering of one or more button presses at an additional client device associated with the identified entity to automatically navigate the IVR tree until the target state is reached; causing rendering of synthesized speech audio data including synthesized speech at an additional client device associated with the identified entity to automatically navigate the IVR tree until the target state is reached; or causing rendering of audio data capturing the spoken words of the given user associated with the client device at an additional client device associated with the identified entity.

[0146] In some implementations, a method implemented by one or more processors is provided, and the method includes: receiving user input from a given user via a search interface of a client device associated with the given user; issuing a search query based on the user input at the search interface of the client device; identifying a given search result associated with an entity to be engaged on behalf of the given user during an assisted telephone call based on the issued search query, the given search result being associated with a target state of an interactive voice response (IVR) tree for being associated with the identified entity from among a plurality of candidate states; causing at least the given search result to be rendered at the search interface of the client device; in response to rendering the given search result at the search interface of the client device, receiving a selection of the given search result from the given user via the search interface of the client device; and in response to receiving the selection of the given search result, initiating execution of the assisted telephone call. Execution of the assisted telephone call includes automatically navigating the assisted telephone call to the target state of the IVR tree.

[0147] These and other implementations of the technology disclosed herein can optionally include one or more of the following features.

[0148] In some embodiments, the method may further include: identifying an additional given search result associated with the entity based on the issued search query, the additional given search result being associated with an additional target state for the IVR tree from the plurality of candidate states. In some versions of those embodiments, the method may further include causing the additional given search result to be rendered with the given search result. In some further versions of those embodiments, causing the additional given search result to be rendered with the given search result may include rendering the given search result more prominently than the additional given search result.

[0149] In still further versions of those embodiments, rendering the given search result more prominently than the additional given search results may include rendering the given search result more prominently than the additional given search results based on a recent user interaction of the given user with the client device. In still further versions of those embodiments, the recent user interaction of the given user with the client device may include the given user accessing content associated with the given search result. In still further additional or alternative versions of those embodiments, rendering the given search result more prominently than the additional given search results may include rendering the given search result more prominently than the additional given search results based on popularity of the given search result. In still further versions of those embodiments, the popularity of the given search result may be based on click-through rates of multiple users other than the given user associated with the given search result.

[0150] In some implementations, a method implemented by one or more processors is provided, and the method includes: receiving user input from a given user via a client device associated with the given user for initiating an auxiliary telephone call; identifying an entity to be engaged on behalf of the given user during the auxiliary telephone call based on the user input; identifying an interactive voice response (IVR) tree associated with the identified entity, the IVR tree including a plurality of candidate states; determining availability of the plurality of candidate states as target states for the IVR tree based on contextual information associated with the client device or the given user associated with the client device; and determining the target state of the IVR tree from among the available plurality of candidate states based on the user input or additional user input; and initiating execution of the auxiliary telephone call to navigate the IVR tree to the target state. Execution of the auxiliary telephone call includes automatically navigating the auxiliary telephone call to the target state of the IVR tree.

[0151] These and other implementations of the technology disclosed herein can optionally include one or more of the following features.

[0152] In some implementations, the context information may be associated with the client device, and the context information may include one or more of: a current location of the client device, a current time at the current location of the client device, or a current date at the current location of the client device. In some versions of those implementations, the given IVR tree selected from among the plurality of different IVR trees may be based on one or more of: the current location of the client device, or the current time at the current location of the client device.

[0153] In some embodiments, the context information may be associated with the given user associated with the client device, and wherein the context information may include one or more of: electronic communications created at or received at the client device, user account information associated with a user account of the given user, or monetary information associated with the given user. In some versions of those embodiments, the given IVR tree selected from the plurality of different IVR trees may be based on one or more of: electronic communications created at or received at the client device, user account information associated with the user account of the given user, or monetary information associated with the given user.

[0154] In addition, some embodiments include one or more processors (e.g., central processing units (CPUs), graphics processing units (GPUs), and / or tensor processing units (TPUs)) of one or more computing devices, wherein the one or more processors are operable to execute instructions stored in associated memory, and wherein the instructions are configured to cause performance of any of the above methods. Some embodiments also include one or more non-transitory computer-readable storage media storing computer instructions executable by one or more processors to perform any of the above methods. Some embodiments also include a computer program product comprising instructions executable by one or more processors to perform any of the above methods.

[0155] It should be understood that all combinations of the foregoing concepts and additional concepts described in greater detail herein are contemplated as being part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the beginning of this disclosure are contemplated as being part of the subject matter disclosed herein.

Claims

1. A method implemented by one or more processors, the method comprising: receiving user input from a given user via a client device associated with the given user for initiating a secondary telephone call; identifying, based on the user input, an entity to engage on behalf of the given user during the assisted telephone call; identifying an interactive voice response (IVR) tree associated with the identified entity, the IVR tree comprising a plurality of candidate states; determining a target state of the IVR tree from among the plurality of candidate states based on the user input or additional user input; as well as initiating execution of the auxiliary telephone call to navigate the IVR tree to the target state, wherein the execution of the auxiliary telephone call includes automatically navigating the auxiliary telephone call to the target state of the IVR tree; During execution of the assisted telephone call, and when the IVR tree is navigated to the target state: detecting at least one user interaction directed toward the client device and associated with the identified entity, and generating one or more prompts related to modifying navigation of the IVR tree based on the at least one user interaction directed to the client device, and In response to determining that another additional user input is received in response to rendering the one or more prompts, modifying navigation of the IVR tree: modifying execution of the secondary telephone call based on the further additional user input, wherein modifying execution of the secondary telephone call based on the further additional user input comprises: identifying an alternative target state for the IVR tree based on the further additional user input, and automatically navigating the assisted telephone call to the alternate target state of the IVR tree; identifying a task to be performed on behalf of the given user and during the assisted telephone call based on the user input, the additional user input, or the further additional user input; and After navigating the IVR tree to the alternate target state of the IVR tree, and during execution of the assisted telephone call: engaging in a conversation with a human representative associated with the identified entity to complete the task on behalf of the given user, wherein engaging in a conversation with the human representative comprises: causing rendering at an additional client device associated with the human representative of synthesized speech audio data including synthesized speech for completing the task, using a speech recognition model to process audio data captured from the human representative's spoken utterance and received in response to the rendering of the synthesized speech, and An outcome associated with performance of the task is determined based on processing the audio data.

2. The method according to claim 1, further comprising: generating a notification based on a result associated with performance of the task; as well as The notification is caused to be rendered at the client device associated with the given user.

3. The method according to claim 1, further comprising: In response to determining that the further additional user input received in response to rendering one or more of the prompts does not modify navigation of the IVR tree: The secondary telephone call is continued to navigate the IVR tree to the target state.

4. The method according to claim 1, further comprising: In response to identifying the IVR tree associated with the identified entity: obtaining one or more initial prompts associated with navigating the IVR tree associated with the identified entity, and causing one or more of the initial prompts to be rendered at the client device associated with the given user, Wherein the additional user input is received in response to rendering one or more of the initial prompts.

5. The method according to claim 1, wherein: The at least one user interaction at the client device associated with the given user comprises one or more of: a search interaction at the client device associated with the identified entity, a browse interaction at the client device associated with the identified entity, or a navigation interaction at the client device associated with the identified entity.

6. The method according to any one of claims 1 to 5, wherein: Automatically navigating the auxiliary telephone call to the target state of the IVR tree includes: One or more values ​​associated with corresponding parameters for navigating the IVR tree are determined based on the user input or the additional user input.

7. The method according to claim 6, wherein: Modifying the navigation of the IVR includes: determining the alternative target state of the IVR tree based on the user interaction or the further additional user input, Wherein the performing of the modification of the assisted telephone call includes automatically navigating the assisted telephone call to the alternative target state of the IVR tree based on one or more of the values ​​associated with the corresponding parameters.

8. The method according to claim 6, wherein: Modifying the navigation of the IVR includes: determining the alternative target state of the IVR tree based on the user interaction or the further additional user input, Wherein, performing the modification of the assisted telephone call includes automatically navigating the assisted telephone call to the target state of the IVR tree based on one or more alternative values ​​associated with the corresponding parameters.

9. The method according to any one of claims 1 to 5, wherein: Automatically navigating the assisted telephone call to the target state of the IVR tree includes one or more of the following: simulating rendering of one or more button presses at an additional client device associated with the identified entity to automatically navigate the IVR tree until the target state is reached, causing rendering of synthesized speech audio data including synthesized speech at an additional client device associated with the identified entity to automatically navigate the IVR tree until the target state is reached, and Audio data capturing spoken utterances of the given user associated with the identified entity is caused to be rendered at an additional client device associated with the client device.

10. The method according to claim 9, wherein: Automatically navigating the assisted telephone call to the target state of the IVR tree includes causing the synthesized speech audio data including the synthesized speech to be rendered at an additional client device associated with the identified entity to automatically navigate the IVR tree until the target state is reached.

11. The method according to claim 10, wherein: Causing rendering of the synthesized speech audio data including the synthesized speech at an additional client device associated with the identified entity to automatically navigate the IVR tree until the target state is reached comprises: receiving, from a human representative or an IVR system associated with the identified entity and via the additional client device and during the auxiliary telephone call, audio data capturing spoken utterances of the human representative or the IVR system; processing the audio data capturing the spoken utterances of the human representative or the IVR system using a speech recognition model; determining, based on processing the audio data, that the spoken utterance of the human representative or the IVR system includes a request; generating the synthesized speech audio data including the synthesized speech based on the request using a speech synthesis model; and The synthesized speech audio data including the synthesized speech is caused to be rendered at the additional client device.

12. The method according to claim 2, wherein: Causing the notification to be rendered at the client device associated with the given user includes one or more of: causing the client device to vibrate, causing the client device to audibly render the notification including the result associated with performance of the task, and causing the client device to visually render the notification including the result associated with performance of the task.

13. A method implemented by one or more processors, the method comprising: receiving user input from a given user via a client device associated with the given user for initiating a secondary telephone call; identifying, based on the user input, an entity to engage on behalf of the given user during the assisted telephone call; identifying an interactive voice response (IVR) tree associated with the identified entity, the IVR tree comprising a plurality of candidate states; determining a target state of the IVR tree from among the plurality of candidate states based on the user input or additional user input; as well as initiating execution of the auxiliary telephone call to navigate the IVR tree to the target state, wherein execution of the auxiliary telephone call includes automatically navigating the auxiliary telephone call to the target state of the IVR tree, wherein automatically navigating the auxiliary telephone call to the target state of the IVR tree includes causing rendering of synthesized speech audio data including synthesized speech at an additional client device associated with the identified entity to automatically navigate the IVR tree until the target state is reached; and During execution of the assisted telephone call, and when the IVR tree is navigated to the target state: detecting at least one user interaction directed toward the client device and associated with the identified entity, and generating one or more prompts related to modifying navigation of the IVR tree based on the at least one user interaction directed to the client device, and In response to determining that another additional user input is received in response to rendering the one or more prompts, modifying navigation of the IVR tree: The execution of the secondary telephone call is modified based on the further additional user input.

14. The method according to claim 13, wherein: Causing rendering of the synthesized speech audio data including the synthesized speech at an additional client device associated with the identified entity to automatically navigate the IVR tree until the target state is reached comprises: receiving, from a human representative or an IVR system associated with the identified entity and via an additional client device and during the assisted telephone call, audio data capturing spoken utterances of the human representative or the IVR system; processing the audio data capturing the spoken utterances of the human representative or the IVR system using a speech recognition model; determining, based on processing the audio data, that the spoken utterance of the human representative or the IVR system includes a request; generating the synthesized speech audio data including the synthesized speech based on the request using a speech synthesis model; and The synthesized speech audio data including the synthesized speech is caused to be rendered at the additional client device.

15. A method implemented by one or more processors, the method comprising: receiving user input from a given user via a client device associated with the given user for initiating a secondary telephone call; identifying, based on the user input, an entity to engage on behalf of the given user during the assisted telephone call; identifying an interactive voice response (IVR) tree associated with the identified entity, the IVR tree comprising a plurality of candidate states; determining a target state of the IVR tree from among the plurality of candidate states based on the user input or additional user input; as well as initiating execution of the auxiliary telephone call to navigate the IVR tree to the target state, wherein the execution of the auxiliary telephone call includes automatically navigating the auxiliary telephone call to the target state of the IVR tree; During execution of the assisted telephone call, and when the IVR tree is navigated to the target state: detecting at least one user interaction directed toward the client device and associated with the identified entity, and generating one or more prompts related to modifying navigation of the IVR tree based on the at least one user interaction directed to the client device, and In response to determining that another additional user input is received in response to rendering the one or more prompts, modifying navigation of the IVR tree: modifying execution of the secondary telephone call based on the further additional user input, wherein modifying execution of the secondary telephone call based on the further additional user input comprises: identifying an alternative target state for the IVR tree based on the further additional user input, and automatically navigating the assisted telephone call to the alternate target state of the IVR tree; After navigating the IVR tree to a particular state of the IVR tree, and during execution of the assisted telephone call: engaging in a conversation with a human representative associated with the identified entity to verify whether the particular state corresponds to the alternate target state, wherein engaging in a conversation with the human representative to verify whether the particular state corresponds to the alternate target state comprises: using a speech recognition model to process audio data capturing spoken utterances of the human representative, determining whether the human representative verified that the particular state corresponds to the alternate target state based on processing the audio data, and In response to determining that the human representative verified that the particular state corresponds to the target state, a notification is generated requesting the given user associated with the client device to join the assisted telephone call.

16. The method according to claim 15, further comprising: After navigating the IVR tree to the specific state during the assisted telephone call: generating a notification requesting the given user associated with the client device to join the secondary telephone call, and The notification is caused to be rendered at the client device associated with the given user.

17. The method according to claim 16, wherein: Causing the notification to be rendered at the client device associated with the given user includes one or more of: causing the client device to vibrate, causing the client device to audibly render the notification requesting the given user to join the auxiliary telephone call, and causing the client device to visually render the notification requesting the given user to join the auxiliary telephone call.

18. The method according to claim 15, further comprising: causing rendering at an additional client device associated with the human representative of synthesized speech audio data including synthesized speech for verifying whether the IVR tree was successfully navigated, wherein the audio data capturing the spoken utterance of the human representative is received in response to rendering the synthesized speech audio data at the additional client device associated with the human representative.

19. The method according to claim 15, further comprising: In response to determining that the human representative indicated that the particular state does not correspond to the target state: Causing rendering at an additional client device associated with the human representative of another synthesized speech audio data comprising another synthesized speech requesting that the secondary telephone call be transferred to another human representative associated with the identified entity.

20. At least one computing device comprising: at least one processor; as well as A memory storing instructions which, when executed, cause the at least one processor to perform a method according to any one of claims 1 to 19.

21. A non-transitory computer-readable storage medium storing instructions which, when executed, cause at least one processor to perform the method according to any one of claims 1 to 19.

Images