Selective execution of automated telephone calls to reduce latency and / or duration of assistant interaction

By identifying relevant entities and dynamically managing call timing based on response data, the automated assistant optimizes resource use and reduces redundant interactions, ensuring timely and efficient information delivery.

JP2024164047AActive Publication Date: 2024-11-26GOOGLE LLC
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
JP2024129935
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2020-08-04
Filing Date
2024-08-06
Publication Date
2024-11-26
Estimated Expiration
2040-12-15

AI Technical Summary

Technical Problem

Automated assistants often waste computational and network resources when entities cannot satisfy user requests, leading to unnecessary manual and automated calls to multiple entities, and users manually check entity status due to outdated information.

Method used

The automated assistant identifies a group of relevant entities based on query activity or user input, initiates parallel or sequential automated calls, processes responses, and dynamically determines time delays for subsequent calls based on entity availability, updating search results with real-time information to reduce redundant interactions.

Benefits of technology

This approach reduces latency and resource usage by ensuring up-to-date information is provided without unnecessary calls, conserving computational and network resources while optimizing interaction time with the automated assistant.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024164047000001_ABST
    Figure 2024164047000001_ABST
Patent Text Reader

Abstract

To provide methods and computing devices for realizing selective execution of automated telephone calls to reduce the latency and / or the duration of assistant interaction.SOLUTION: A method directed to using an assistant to initiate automated telephone calls with entities comprises: identifying an item of interest; identifying a group of entities associated with the item; initiating the calls with the entities. During a given call with a given entity, the assistant requests a status update regarding the item, and determines a temporal delay before initiating another call with the given entity to request a further status update regarding the item based on information received in response to the request. The method may further comprise: receiving a request to perform an action on behalf of a user; identifying a group of entities that can perform the action; and initiating a given call with a given entity.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] Relates to selectively making automated calls to reduce latency and / or length of assistant interactions. [Background technology]

[0002] Automated assistants can interact with users through a variety of computing devices, such as smartphones, tablet computers, wearable devices, automotive systems, standalone personal assistant devices, etc. The automated assistant receives input from the user (e.g., spoken, touch, and / or typed input) and responds with responsive output (e.g., visual and / or auditory).

[0003] A user can interact with an automated assistant to have the automated assistant perform an action on the user's behalf. As an example, the automated assistant can make a phone call on the user's behalf to perform a given action and can engage in a dialogue with additional users to perform the action. For example, a user can provide an input requesting an automated assistant to inquire about an item or service over the phone on the user's behalf. The automated assistant can initiate an automated call with a particular entity related to the item or service and can provide information to additional users related to the particular entity to ask questions. The automated assistant can then inform the user of the results of the automated call and / or update search results related to the particular entity based on the results. Summary of the Invention [Problem to be solved by the invention]

[0004] However, for some actions performed by an automated assistant on behalf of a user, a particular entity may not be able to fully satisfy the user's request, resulting in wasted computational and / or network resources utilized in performing the action. As an example, assume that an automated assistant is making a call on behalf of a user as described above, and further assume that a user associated with a particular entity indicates that an item or service is not available. Some automated assistants may determine that the item or service is not available and simply provide a notification to the user reflecting that the item or service is not available. Thus, computational and / or network resources utilized in performing the automated call are wasted. Furthermore, the user may manually make other calls to other entities and / or have the automated assistant make other automated calls to other entities in an attempt to satisfy the request, resulting in yet further use of computational and / or network resources. Still further, other users may make manual and / or automated calls to a particular entity for the same or similar request that the particular entity cannot satisfy, resulting in yet further use of computational and / or network resources. [Means for solving the problem]

[0005] Implementations are directed to initiating and executing an automated call using an automated assistant. In some implementations, the automated call may be initiated and executed in response to a request to initiate and execute an automated call. The automated assistant may identify a group of entities related to the request and may initiate a corresponding automated call with one or more of the entities of the group to perform an action related to the request. Additionally, the automated assistant may execute an action related to the request through rendering instances of synthetic speech related to the request. At least some of the instances of synthetic speech may be generated based on the request and based on processing responses during the automated call from representatives of the entities participating in the call.

[0006] In some implementations, the request to initiate and execute an automated call may be based on analyzing the query activity of multiple users. The automated assistant may identify terms corresponding to the item or service of interest that are included in a threshold number of queries (and optionally for a threshold length of time) based on analyzing the query activity. In response to identifying terms corresponding to the item or service of interest that are included in a threshold number of queries (and optionally for a threshold length of time), the automated assistant may identify a group of entities related to the item or service of interest in one or more databases. For example, assume there is a surge in queries about toilet paper. In this example, the request may be a request for a status update regarding toilet paper at each of the entities in the group, each of which may be identified based on the entity being stored in association with the item toilet paper. In some versions of those implementations, the entities included in the group may be constrained to a particular geographic area where the analyzed queries originate from (e.g., within 10 miles, 50 miles, or other threshold distance) and / or constrained to one or more entities of a particular type (e.g., pharmacy, supermarket, convenience store, grocery store, etc.) that are stored in association with the item of interest. Continuing with the above example, further assume that the most recent increase in queries originates from Jefferson County, Kentucky. In this example, the entities included in the group may be constrained to entities that are physically located in (or within a threshold distance from) Jefferson County, Kentucky and that are stored in association with the item toilet paper. The automated assistant may also intelligently identify corresponding meaningful types of entities that are relevant to the item of interest.Continuing with the example above, the automated assistant may identify that entities such as supermarkets, convenience stores, and grocery stores primarily sell toilet paper, and those associated stores in Jefferson County, Kentucky are included in the group of entities.

[0007] In some versions of those implementations, an instance of the automated assistant can be used to make calls to each of the entities of the group in parallel. During each of the automated calls, an action related to the request can be performed through rendering an instance of a synthetic utterance (e.g., requesting a status update) and information can be received from a representative associated with a corresponding one of the entities that responded to the synthetic utterance. Additionally, the automated assistant can determine a time delay before initiating a subsequent additional corresponding automated call with the given entity to perform the action again. Moreover, in response to determining that the time delay has elapsed, the automated assistant can initiate a subsequent additional corresponding automated call with the given entity to perform the action again.

[0008] For example, assume that the request is for a status update regarding toilet paper at each of the entities of the group, and assume that a representative associated with the given entity indicates that there is enough toilet paper at the given entity to not sell out for three days. In this example, the automated assistant can assign a time period of 72 hours as a time delay based on the information received from the representative. Thus, the automated assistant does not initiate a subsequent corresponding automated call with the given entity until the 72-hour period has elapsed. In contrast, assume that a representative associated with the given entity indicates that there may not be enough toilet paper at the given entity to not sell out by the close of business for that day. In this example, the automated assistant can assign a time period of 4 hours as a time delay based on the information received from the representative. Thus, the automated assistant initiates a subsequent corresponding automated call with the given entity at a later time on the same day. In these and other schemes, the corresponding time delay can be dynamically determined for each entity based on the information received during the automated call with the entity. Dynamically determining the time delay in such a manner can balance the technical advantages achieved by maintaining up-to-date information from the entity with the resources utilized in initiating and executing an automated call. In other words, dynamically determining the time delay can ensure that the information from the entity is up-to-date while preventing premature execution of another automated call to the entity, which would result in unnecessary utilization of computational and / or network resources.

[0009] In other implementations, the request to initiate and perform an automated call may be based on a user input from a user of a computing device associated with the automated assistant. The automated assistant may identify an item or service of interest related to the request based on processing the user input. In response to identifying the item or service of interest in the user input, the automated assistant may identify a group of entities related to the item or service of interest in one or more databases. For example, assume that a user of a computing device provides a user input of "Find a roof repair contractor." In this example, the request may be a request for availability of a roof repair contractor coming from each of the entities of the group, each of which may be identified based on the entity being stored in association with the topic roof repair. In some versions of those implementations, the entities included in the group may be constrained to entities that are within a threshold distance from the location of the user's computing device. Continuing with the above example, the entities included in the group may be constrained to entities that are within 10 miles of the location of the user's computing device and are stored in association with roof repair services. If there are no entities included in the group, the threshold distance may be increased until one or more entities are included in the group.

[0010] In some versions of those implementations, an instance of the automated assistant can be used to alternately call each of the entities of the group. The automated assistant can initiate a first automated call with a first of the entities of the group. An action related to the request can be performed through rendering an instance of a synthetic utterance (e.g., requesting a status update, requesting service availability, and / or other action) and can receive information from a representative associated with the first entity of the group who responded to the synthetic utterance. Additionally, the automated assistant can determine a call start time associated with each of the other entities of the group. As described in more detail herein (e.g., in connection with FIG. 1), the call start time can be a set amount of time after initiating each of the automated calls, can be determined based on call history data associated with each of the entities of the group, can be determined based on the progress of the automated calls, and / or can be other data related to the automated calls. The automated assistant can continue to initiate automated calls with the entities of the group until the request is satisfied.

[0011] Moreover, the automated assistant can generate a notification based on one or more results of the automated call, and thus can render one or more notifications for each of the automated calls initiated and performed by the automated assistant. Furthermore, the automated assistant can cause the user's computing device to render the notification audibly and / or visually at the user's computing device. In some further versions of those implementations, the notification can include a prompt that asks the user of the computing device to approve the provision of user information to a person associated with a given entity. As described herein, in various implementations, the call start time utilized in making alternating automated calls can be determined with consideration to attempting to reduce the latency in which the notification is provided and / or to reduce the overall length of the user's interaction with the automated assistant. In many of those implementations, the call start time is further determined with consideration to attempting to reduce unnecessary calls and / or unnecessarily long call times with one or more entities that are called later (e.g., in situations where the previously called entity can satisfy the action). Determining the call start time in such a manner can balance the technical advantages achieved by reducing the latency and / or interaction time with the resources utilized in performing the automated calls.

[0012] In various implementations, results related to the entities of the group may be updated based on information obtained during the automated call. For example, assume that a request is a request for a corresponding status update on an item, toilet paper, originating from an information retrieval system, such as a search interface. Assume further that first information received from a first representative associated with the first entity indicates that the first entity has enough toilet paper to last for three days, and second information received from a second representative associated with the second entity indicates that the second entity may not have enough toilet paper to last until the end of business for that day. In this example, results related to the first entity and the second entity may be updated to include respective indications of the information received during the automated call. In some versions of those implementations, the updated results may further include a temporal indication of when the information is received. In these and other ways, users are provided with an indication of information about an entity, and optionally a temporal indication of the information, which can at least selectively prevent users from manually calling the entity to look for such information and / or from initiating automated calls to the entity to look for such information, and / or can reduce the risk of users traveling to and from locations associated with the entity. At least in populations of users, and across large numbers of users, this can reduce the overall volume of calls to the entity, thereby saving network and / or computational resources that would otherwise be utilized in such calls.

[0013] The above description is provided as a summary of only some implementations disclosed herein. Those implementations and other implementations are described in additional detail herein. [Brief description of the drawings]

[0014] [Figure 1]FIG. 1 is a block diagram of an example environment illustrating various aspects of the disclosure in which implementations disclosed herein may be implemented. [Diagram 2] 1 is a flowchart illustrating an example method for determining a time delay for a subsequent automated call with a given entity based on a status update regarding an item of interest received during the automated call with the given entity, according to various implementations. [Diagram 3] 1 is a flowchart illustrating an example method for determining whether to initiate additional automated calls with additional entities based on information received during an automated call with a given entity, according to various implementations. [Figure 4A] FIG. 13 illustrates a non-limiting example of determining a time delay for subsequent automated calls with various entities based on the results of the automated calls to update the results associated with each of the various entities, according to various implementations. [Figure 4B] FIG. 13 illustrates a non-limiting example of determining a time delay for subsequent automated calls with various entities based on the results of the automated calls to update the results associated with each of the various entities, according to various implementations. [Figure 4C] FIG. 13 illustrates a non-limiting example of determining a time delay for subsequent automated calls with various entities based on the results of the automated calls to update the results associated with each of the various entities, according to various implementations. [Figure 4D] FIG. 13 illustrates a non-limiting example of determining a time delay for subsequent automated calls with various entities based on the results of the automated calls to update the results associated with each of the various entities, according to various implementations. [Figure 5A] FIG. 1 illustrates a non-limiting example of initiating automated calls with various entities on behalf of a user, with notifications related to each of the various entities being rendered based on the results of the automated calls, according to various implementations. [Figure 5B]FIG. 1 illustrates a non-limiting example of initiating automated calls with various entities on behalf of a user, with notifications related to each of the various entities being rendered based on the results of the automated calls, according to various implementations. [Figure 5C] FIG. 1 illustrates a non-limiting example of initiating automated calls with various entities on behalf of a user, with notifications related to each of the various entities being rendered based on the results of the automated calls, according to various implementations. [Figure 5D] FIG. 1 illustrates a non-limiting example of initiating automated calls with various entities on behalf of a user, with notifications related to each of the various entities being rendered based on the results of the automated calls, according to various implementations. [Figure 6A] FIG. 13 also illustrates a non-limiting example of initiating automated calls with various entities on behalf of a user, with notifications related to each of the various entities being rendered based on the results of the automated calls, according to various implementations. [Figure 6B] FIG. 13 also illustrates a non-limiting example of initiating automated calls with various entities on behalf of a user, with notifications related to each of the various entities being rendered based on the results of the automated calls, according to various implementations. [Figure 6C] FIG. 13 also illustrates a non-limiting example of initiating automated calls with various entities on behalf of a user, with notifications related to each of the various entities being rendered based on the results of the automated calls, according to various implementations. [Figure 7] FIG. 1 illustrates an exemplary architecture of a computing device, according to various implementations. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] 1 illustrates a block diagram of an example environment for illustrating various aspects of the present disclosure. A client device 110 is illustrated in FIG. 1 and includes, in various implementations, a user input engine 111, a rendering engine 112, a notification engine 113, a speech recognition engine 120A1, a natural language understanding ("NLU") engine 130A1, and a speech synthesis engine 140A1.

[0016] The user input engine 111 can detect user input at the client device 110. The user input detected at the client device 110 can include spoken input detected via a microphone of the client device 110 and / or additional spoken input transmitted to the client device 110 from additional client devices of additional users (e.g., during an automated call), touch input detected via a user interface of the client device 110, and / or typed input detected via a user interface of the client device 110. The additional users can be, for example, a human representative associated with the entity, an automated assistant associated with the entity, an interactive voice response system ("IVR" system) associated with the entity, and / or a combination thereof. The automated calls described herein can be performed using a variety of voice communication protocols including, for example, Voice over Internet Protocol (VoIP), Public Switched Telephone Network (PSTN), and / or other telephony communication protocols. Moreover, the automated calls described herein may be initiated and performed using an automated assistant 115 running locally on the client device 110 (e.g., a local assistant), an automated assistant 115 running remotely on one or more servers (e.g., a cloud-based assistant), and / or a combination thereof (e.g., as indicated by the dashed lines in FIG. 1 ).

[0017] In various implementations, the automated assistant 115 can use the automated request system 180 to initiate and execute the automated call. In implementations where the automated assistant 115 executes locally on the client device 110, the automated request system 180 can execute locally on the client device 110, such that the automated call is initiated and executed using only the resources of the client device 110. In implementations where the automated assistant 115 executes remotely on one or more of the servers, the automated request system 180 can execute remotely on one or more of the servers, such that the automated call is initiated and executed using only the resources of the server or servers. In implementations where the automated assistant 115 executes distributed in a manner such as that shown in FIG. 1, the automated request system 180 can execute remotely on one or more of the servers in communication with the client device 110 via the network 190. The network 190 can include, for example, Wi-Fi, Bluetooth, near field communications, local area networks, wide area networks, other networks, and / or any combination thereof.

[0018] The automatic request system 180, in various implementations, includes a speech recognition engine 120A2, an NLU engine 130A2, a speech synthesis engine 140A2, and an automatic call engine 150. Further, the automatic call engine 150, in various implementations, includes a query analysis engine 151, a request processing engine 152, an entity identification engine 153, a request execution engine 154, an update engine 155, an approval engine 156, and a time delay engine 157.

[0019] In some implementations, the automated assistant 115 can use the automated request system 180 to initiate and execute an automated call in response to detecting user input from a given user of the client device 110 via the user input engine 111, including a request to initiate an automated call. The user input can include typed or touch input detected via a touch screen or keyboard of the client device 110, and / or spoken input detected via a microphone of the client device 110. In implementations in which the user input engine 111 detects typed or touch input, the client device 110 can send an indication of the detected user input to the request processing engine 152 of the automated call engine 150 via the network 190. For example, the user input engine 111 can detect typed or touch input via a search interface, an automated assistant interface, and / or other interface of the client device 110, determine that the detected user input is directed to initiating and executing an automated call, and send the detected user input to the request processing engine 152 of the automated call engine 150 for further processing of the detected user input.

[0020] In an implementation in which the user input engine 111 detects a spoken input of a given user via a microphone of the client device 110, the speech recognition engine 120A1 of the client device 110 can use the speech recognition model 120A to process audio data generated by the microphone capturing the spoken input to generate recognized text corresponding to the spoken input. Additionally, the NLU engine 130A1 of the client device 110 can use the NLU model 130A to process the recognized text generated by the speech recognition engine 120A1 to determine the intent included in the spoken input. For example, if the client device 110 detects a spoken input of "find toilet paper near me" from a given user, the client device 110 can use the speech recognition model 120A to process the audio data capturing the spoken input to generate recognized text corresponding to the spoken input, and can use the NLU model 130A to process the recognized text to determine at least the intent of finding toilet paper near the given user's current location as an action included in the request to initiate and execute an automated call. In some versions of those implementations, the client device 110 can send the recognized text corresponding to the spoken input and / or intent determined based on the spoken input to a request processing engine 152 of the automatic call engine 150 for further processing of the recognized text and / or intent.

[0021] In other implementations in which the user input engine 111 detects a spoken input of a given user via a microphone of the client device 110, the automated assistant 115 can cause the client device 110 to transmit audio data capturing the spoken input to the automated request system 180. The speech recognition engine 120A2 and / or the NLU engine 130A2 of the call assistance system 180 can process the audio data capturing the spoken input in a manner similar to that described above with respect to the speech recognition engine 120A1 and / or the NLU engine 130A1 of the client device 110. Furthermore, the spoken input and / or intent is determined based on the spoken input to the request processing engine 152 of the automated call engine 150 for further processing of the recognized text and / or intent. In some additional and / or alternative implementations, speech recognition engine 120A1 and / or NLU engine 130A1 of client device 110 may be used in a distributed manner in conjunction with speech recognition engine 120A2 and / or NLU engine 130A2 of automatic request system 180. Moreover, speech recognition model 120A and / or NLU model 130A may be stored locally on client device 110, remotely at automatic request system 180, and / or remotely at other servers in communication with client device 110 and / or automatic request system 180 via network 190.

[0022] In various implementations, the speech recognition model 120A is an end-to-end speech recognition model, so that the speech recognition engines 120A1 and / or 120A2 can directly use the model to generate recognized text corresponding to the spoken input. For example, the speech recognition model 120A can be an end-to-end model used to generate recognized text character-by-character (or other token-by-token). One non-limiting example of such an end-to-end model used to generate recognized text character-by-character is the Recurrent Neural Network Transducer (RNN-T) model. The RNN-T model is a form of sequence-to-sequence model that does not utilize an attention mechanism. Unlike most sequence-to-sequence models that typically require processing the entire input sequence (e.g., audio data waveform, or Mel-Frequency Cepstral Coefficients (MFCC), or other representation) to generate a predicted output, the RNN-T model can be used to continuously process input samples and stream output symbols (e.g., letters of the alphabet). Also, for example, when the speech recognition model is not an end-to-end speech recognition model, speech recognition engine 120A1 and / or 120A2 can instead generate predicted phonemes (and / or other representations). For example, in such models, the predicted phonemes (and / or other representations) are then utilized by speech recognition engine 120A1 and / or 120A2 to determine recognized text that matches the predicted phonemes. In doing so, speech recognition engine 120A1 and / or 120A2 can optionally utilize a decoding graph, a lexicon, and / or other resources.

[0023] In some additional and / or alternative implementations, the automated assistant 115 can initiate and execute an automated call using the automated request system 180 without detecting any user input via the user interface input engine 111. In some implementations, the query analysis engine 151 can analyze the query activity stored in the query activity database 151A and can send a request to initiate and execute an automated call in response to determining that one or more conditions are satisfied. The query activity analyzed by the query analysis engine 151 can include queries submitted to a search engine (e.g., via a search interface or an automated assistant interface) by multiple users using respective client devices. In some versions of those implementations, the one or more conditions can include determining whether a threshold number of queries for a given item or service have been submitted by multiple users. For example, if the query analysis engine 151 determines that a threshold of queries for a given item (e.g., toilet paper) has been exceeded based on an analysis of query activity of multiple users using 30,000 queries for "toilet paper" or its variants (e.g., "TP", "bathroom tissue", etc.) by multiple users, the automated assistant 115 may send a request to the request processing engine 152 to initiate and execute an automated call. In some versions of those implementations, the query activity may be constrained to queries issued by users in a particular geographic region. For example, the query activity may be constrained to queries issued by users in a particular zip code, county, municipality, town, city, state, country, and / or other geographic region.

[0024] In some further versions of those implementations, the one or more conditions may include determining whether a threshold number of queries for a given item or service have been issued by multiple users within a given time period. Continuing with the above example, if the query analysis engine 151 determines that 30,000 queries for “toilet paper” or variations thereof have been issued within a two-week period, the automated assistant may not send a request to the request processing engine 152. However, if the query analysis engine 151 determines that 30,000 queries for “toilet paper” have been issued within a two-day period, the automated assistant may send a request to the request processing engine 152. In this manner, the query analysis engine 151 can detect whether there is an increase in queries for a given item over a particular period of time. Moreover, in various implementations, the query analysis engine 151 may analyze only a subset of queries issued in the query activity database 151A, such as those that are attempting to find or obtain an item of interest rather than just referring to “toilet paper” or variations thereof. In this manner, the query analysis engine 151 can identify spikes in queries originating from particular geographic regions.

[0025] The request processing engine 152 can process the received request to determine an action associated with the request. The action associated with the request can include requesting a status update on an item of interest at a given entity, requesting availability of a given entity related to a particular service for a period of time, and / or other actions that can be performed by the automated assistant 115 during an automated call. In an implementation in which the request is based on a user input detected at the client device 110 via the user input engine 111, the request processing engine 152 can determine an action associated with the request based on an indication of typed and / or touch input and / or generated recognized text and / or determined intent from the spoken input. For example, if the user input detected via the user input engine 111 is a spoken input of "Find an electrician who can work on my house tomorrow at 12:00 PM," the request processing engine 152 can determine that the action includes calling an electrician to request availability of an electrician tomorrow at 12:00 PM based on the generated recognized text and / or determined intent. In implementations where the request is based on analyzing query activity without detecting any user input at the client device 110 via the user input engine 111, the request processing engine can determine an action associated with the request based on a given item or service included in the query. For example, if the query analysis engine 151 determines that 5000 queries including the term "hand sanitizer" were submitted by users in Jefferson County, Kentucky during a one-hour period, the request processing engine 152 can determine that the action includes calling entities stored in association with hand sanitizer in Jefferson County, Kentucky, to request a status update regarding the availability of hand sanitizer at each of the entities.

[0026] The entity identification engine 153 can identify a group of entities to be engaged during the automated call. The entities can be, for example, person entities, business entities, location entities, and / or other entities. In some implementations, the entity identification engine 153 can also determine a specific type of entity to be included in the group. For example, the person entity type can be a friend entity, a family entity, a coworker entity, and / or other specific type of person entity. Further, the business entity type can be a service provider entity, an airline entity, a hotel entity, a retail store entity, and / or other specific type of business entity. Moreover, the location entity type can be a school entity, a museum entity, a library entity, and / or other specific type of location entity. In some implementations, the entity identification engine 153 can also determine a unique entity for the identified entity. For example, a unique entity for a person entity may be the person's name (e.g., John Doe, Example Plumber 1, etc.), a unique entity for a business entity may be the business's name (e.g., Example Super Store, Hypothetical Toy Store, Example Plumbing Company, etc.), and a unique entity for a location entity may be the location's name (e.g., Hypothetical Library, Example State Park, etc.). The entities described herein may be defined at various levels of granularity, but for simplicity, they are collectively referred to herein as "entities."

[0027] Each of the entities identified by the entity identification engine 153 may be stored in the entity database 153A. Additionally, each of the entities may be stored in the entity database 153A in association with an item associated with the entity, a service associated with the entity, a location associated with the entity, a web page associated with the entity, and / or other data related to each of the entities. In some implementations, the entity database 153A may be populated with the entities and associated data using a web crawler. For example, if a given entity is a Hypothetical Toy Store, a web crawler may crawl web pages related to the Hypothetical Toy Store to identify each location of the Hypothetical Toy Store, available inventory of specific items at the Hypothetical Toy Store, and the business hours of the Hypothetical Toy Store, and store this data in association with the Hypothetical Toy Store in the entity database 153A. As another example, if a given entity is a Hypothetical Cafe, a web crawler can crawl web pages related to the Hypothetical Cafe to identify each location of the Hypothetical Cafe, the menu items available at the Hypothetical Cafe, and the hours of operation of the Hypothetical Cafe, and store this data in association with the Hypothetical Cafe in entity database 153A. In this manner, entity database 153A includes a correspondence between each of the entities stored therein and data related to the entities. In some additional and / or alternative implementations, the entity or a user visiting the entity can provide an indication of items related to the entity.

[0028] In an implementation in which the request is based on received user input detected at the client device 110 via the user input engine 111, the entity identification engine 153 may identify a group of entities each capable of fulfilling the request in response to receiving the user input. The entity identification engine 153 may initially identify each of the entities stored in the entity database 153A associated with a given item or service included in the request as an initial group of entities each capable of fulfilling the request. Furthermore, the entity identification engine 153 may identify the location of the client device 110 based on location data generated by one or more sensors (e.g., GPS) of the client device 110 and may determine which of the entities in the initial group are within a threshold distance from the location of the client device 110. Moreover, the entity identification engine 153 may include in the group of entities those entities that are within a threshold distance from the location of the client device 110 and that are capable of fulfilling the request. For example, if the user input detected at client device 110 is the spoken input of “looking for paper towels,” entity identification engine 153 can determine the current location of client device 110, identify entities stored in entity database 153A related to paper towels, determine which of the identified entities are within a 10-mile radius of the current location of client device 110, and include each of the entities related to paper towels that are within a 10-mile radius of the current location of client device 110 in the group of entities.

[0029] In some versions of those implementations, the entities identified initially may be constrained to those entities stored in association with a location within a threshold distance from the location of the client device 110. Continuing with the above example, only entities that are within a 10 mile radius of the client device 110 and stored in association with paper towels may be considered for inclusion in the group. In some versions of those implementations, the entity identification engine 153 may only include a threshold number of entities in the group. For example, if the number of entities in the group exceeds the threshold number of entities, only entities that meet one or more criteria may be included in the group. The one or more criteria may include, for example, a location criterion, a time criterion, a rating criterion, a user preference criterion, and / or other criteria. Continuing with the above example, if there are 10 entities within a 10 mile radius of the current location of the client device 110 and related to paper towels, the 5 closest entities, the 5 entities with the highest ratings, and / or the 5 entities that are currently open as indicated by business hours, or the 5 entities that the user of the client device 110 has visited in the past or frequently visits. In some additional and / or alternative versions of those implementations, the entity identification engine 153 may identify additional groups of entities if none of the entities in the group of entities can fulfill the request. In response to determining that none of the entities in the group can fulfill the request, the threshold distance may be modified. Continuing with the above example, if none of the five entities in the group have paper towels available, five additional entities within a ten mile radius may be included in the additional group, or if there are not five additional entities within the ten mile radius, the ten mile radius may be expanded to fifteen miles or more. In this manner, the entities identified in response to the user input are personalized to the user who provided the user input.

[0030] In implementations where the request is based on analyzing the query activity via the query analysis engine 151, the entity identification engine 153, in response to receiving the request, can identify a group of entities each capable of satisfying the request. The entity identification engine 153 can identify each of the entities stored in the entity database 153A associated with a given item or service included in the request as a group of entities each capable of satisfying the request. For example, if the query analysis engine 151 determines that there has been a recent surge in queries submitted that include the term "Hot Christmas Toy 2020," each entity stored in association with Hot Christmas Toy 2020 can be included in the group of entities. In some versions of those implementations, the entity identification engine 153 can identify a particular geographical area from which queries in the query activity were submitted. Moreover, the entity identification engine 153 can include entities that are within a particular geographical area and that are capable of satisfying the request in the group of entities. Continuing with the above example, if query analysis engine 151 determines that a recent spike in queries for Hot Christmas Toy 2020 originated from San Francisco County, California, entity identification engine 153 may constrain the entities in the group to only those entities stored in association with locations within San Francisco County, California. In this manner, the entities identified in response to a recent spike in queries issued are personalized to the particular geographic region from which the queries originated.

[0031] The request execution engine 154 can initiate and execute a corresponding automated call with an entity of the group identified by the entity identification engine 153. More specifically, each of the automated calls described herein can be initiated and executed using a variety of voice communication protocols, including, for example, Voice over Internet Protocol (VoIP), Public Switched Telephone Network (PSTN), and / or other telephony communication protocols. Moreover, each of the automated calls described herein can be a call between the automated assistant 115 and a representative associated with a given entity of the group of entities. The representative can be, for example, a human representative associated with the given entity, an additional automated assistant associated with the given entity, an IVR system associated with the given entity, and / or any combination thereof.

[0032] More specifically, the request execution engine 154 can cause the automated assistant 115 to converse with a person associated with a given entity using synthetic speech during an automated call to execute an action included in the request. The request execution engine 154 can provide text and / or phonemes related to the execution of the request to the speech synthesis engine 140A1 of the client device 110 and / or the speech synthesis engine 140A2 of the call assistance system 180 to generate synthetic speech audio data. The synthetic speech audio data can be transmitted to an additional client device of a person associated with the given entity for aural rendering. The speech synthesis engine 140A1 and / or 140A2 can use the speech synthesis model 140A to generate synthetic speech audio data including synthetic speech corresponding to the text and / or phonemes related to the execution of the request. For example, assume that the request is a request for a status update regarding "Hot Christmas Toy 2020". In this example, speech synthesis engine 140A1 and / or 140A2 can determine a sequence of phonemes corresponding to the status update "Are there any Hot Christmas Toys 2020 available for sale," and can use speech synthesis model 140A to process the sequence of phonemes and generate synthetic speech audio data capturing the request for a status update regarding a Hot Christmas Toy. The synthetic speech audio data can be in the form of an audio waveform, for example. In determining the sequence of phonemes corresponding to at least the phonemes for fulfilling the request, speech synthesis engine 140A1 and / or 140A2 can access a token-to-phoneme mapping stored locally at client device 110 or remotely at a server via network 190.

[0033] In various implementations, prior to execution of a request during a corresponding automated call with a given entity, the automated assistant 115 may obtain consent from a person associated with the given entity to engage in a conversation with the automated assistant 115. For example, the automated assistant 115 may obtain consent when initiating an automated call by identifying itself as the automated assistant 115 and asking a person associated with the given entity to consent to engaging in a conversation with the automated assistant. If the automated assistant 115 obtains consent from a person associated with the given entity, the automated assistant 115 may use the automated request system 180 to execute the action included in the request. Furthermore, once consent is obtained, the automated assistant may not need to seek consent from the person in any subsequent automated calls. However, if the automated assistant 115 does not obtain consent from the person, the automated assistant 115 may terminate the automated call.

[0034] The request execution engine 154 can engage in a conversation with a representative associated with the given entity by rendering a corresponding instance of a synthetic utterance for the request. Rendering the corresponding instance of the synthetic utterance includes causing the synthetic utterance to be rendered at an additional computing device of the representative associated with the given entity. The corresponding instance of the synthetic utterance may be generated using the speech synthesis model 140A as described above, or may be generated based on the text and / or phonemes associated with the request. Moreover, the request execution engine 154 can utilize the speech recognition engines 120A1 and / or 1202A2 and the NLU engines 130A1 and / or 130A2 as described above to process any audio data corresponding to information received from the representative associated with the given entity during the given entity. The request execution engine 154 can cause the automated assistant 115 to proceed with the conversation based on information received during the conversation with the representative. For example, the automated assistant 115 can first verify that the given entity is capable of fulfilling the request. If a given entity agrees that the request can be fulfilled, the automated assistant 115 may inquire further about fulfilling the request. However, if a given entity indicates that the request cannot be fulfilled, the automated assistant 115 may end the call. This allows the automated assistant 115 to respond intelligently based on information received during the automated call.

[0035] For example, assume that the request is a request for a status update regarding the availability of latex gloves. In this example, the synthetic speech audio data of "Are there any latex gloves available?" may be rendered at the agent's additional client device, and the agent may provide information, e.g., "yes" or "no," and the audio data capturing the received information may be processed. Based on processing the received audio data, the automated assistant 115 may proceed with the conversation. For example, if the agent indicates that latex gloves are available at the given entity, the request fulfillment engine 154 may further ask the amount of latex gloves available, whether the given entity expects that latex gloves will be available by the end of business for the day, etc. However, if the agent indicates that latex gloves are not available at the given entity, the request fulfillment engine 154 may cause the automated assistant 115 to end the conversation. Engaging in a conversation with an agent associated with a given entity is described in more detail herein (e.g., with respect to FIGS. 4A-4C, 5B-5D, and 6B-6C).

[0036] In implementations where the request is a request for a corresponding status update regarding a given item, the update engine 155 can cause the results related to the given entity to be updated based on information received during a corresponding automated call with a representative associated with the given entity. Among other things, the results related to the given entity can be updated regardless of whether the request is based on received user input detected via the user input engine 111 or based on analyzing query activity via the query analysis engine 151. The updated results related to the given entity can include, for example, an indication that the given item is available at the given entity, an indication of the quantity of the item available at the given entity, and / or other information received during a corresponding automated call with a representative associated with the given entity. Among other things, the results (also referred to herein as "search results") can include any results that may be revealed to a user through various interfaces. For example, the results can be presented via an information retrieval system interface, an automated assistant interface, a map or navigation application interface, a customized interface, and / or any other interface used to reveal information to a user.

[0037] For example, assume that the request is a request for a status update regarding toilet paper. In this example, the automated assistant 115 can engage in a conversation with a representative associated with the given entity to request a status update regarding toilet paper at the given entity. Assume further that during the conversation, the representative associated with the given entity provides information indicating that there is toilet paper available at the given entity, and also provides information about an indication of the amount of toilet paper available (e.g., "We have enough for the end of the day today," "We have 100 packages of toilet paper," "We will likely sell out before noon," etc.). Based on this received information, the update engine can cause the results associated with the given entity to be updated to include an indication that there is toilet paper available at the given entity and an indication of the amount of toilet paper available. In contrast to the above example, assume that the information received from the representative associated with the given entity indicates that the entity does not sell toilet paper. In this example, the results associated with the given entity can be updated to indicate that the given entity is not associated with toilet paper. Additionally, the update engine 155 can send an indication that the given entity is no longer associated with toilet paper, and the entity identification engine 153 can remove the association between toilet paper and the given entity in the entity database 153A.

[0038] In some versions of those implementations, the updated results related to the given entity may also include a time indicator of when the information was received from a representative associated with the given entity. The time indicator may include, for example, a specific time the information was received (e.g., 9:00 AM, 2:00 PM), a specific date the information was received (e.g., yesterday, today, June 11, 2020), a range of dates the information was received (e.g., last week, yesterday morning, this afternoon), and / or any combination thereof. Updating the results related to the multiple entities based on information received during the corresponding automated calls is described in more detail herein (e.g., with respect to FIGS. 4A-4D). By updating the results related to the given entity, multiple users are prevented from having to call the given entity to request status updates on items, thereby saving computational resources of the client devices of the multiple users and the client devices associated with the given entity. In other words, providing a single automated call, and the information from that single automated call, to multiple users can prevent at least two or more of the multiple users from calling the given entity for such information. Moreover, by updating results relevant to a given entity, multiple users do not need to travel to multiple entities looking for an item, thereby conserving natural resources consumed in transportation to multiple entities by multiple users looking for an item.

[0039] In some versions of those implementations, the update engine 155 can also analyze the updated results associated with the given entity and previously updated results associated with the given entity to predict the future availability of an item at the given entity or the future available amount of an item at the given entity at a future time. The update engine 155 can predict the future availability of an item at the given entity or the future available amount of an item at the given entity at a future time using, for example, a machine learning predictive model or rules generated by the machine learning predictive model. For example, the update engine 155 can determine, based on analyzing the updated results associated with the given entity, that there was no toilet paper available at the given entity on Thursdays and Fridays of the past three weeks, but there was toilet paper available at the given entity on each of the Saturdays of the past three weeks. Based on this determination, the update engine 155 can predict that there will be no toilet paper available at the given entity on Thursday or Friday this week, but there will be toilet paper available on the following Saturday. Thus, at the beginning of the week, the update engine 155 may cause the results to be updated to also include an indication that there is no toilet paper available for a given entity on Thursday or Friday, but that there is expected to be toilet paper available on Saturday.

[0040] In implementations where the request is based on received user input detected at the client device 110 via the user input engine 111, the request execution engine 154 can initiate an automatic call with an entity of the group of entities to perform the action included in the request. The request execution engine 154 can determine a call start time for initiating execution of additional automatic calls with additional entities of the group. Additionally, the request execution engine 154 can determine each call start time before or during the automatic calls. In some versions of those implementations, the call start time can be a fixed threshold amount of time (e.g., 15 seconds, 30 seconds, 45 seconds, and / or other amount of time). For example, the request execution engine 154 may cause the automated assistant 115 to, for each of the entities in the group, initiate a first automated call with a first entity in the group, initiate a second automated call with a second entity in the group 30 seconds after initiating the first automated call with the first entity, initiate a third automated call with a third entity in the group 30 seconds after initiating the second automated call with the second entity, and so on.

[0041] In other versions of those implementations, the request fulfillment engine 154 can determine each call start time based on call history data (e.g., stored in the call history database 154A) associated with a corresponding one of the entities of the group and / or request complexities. The call history data can include, for example, an indication that the corresponding entity is associated with an IVR system and a length of time to operate the IVR system, an indication of the length of time of a previous automated call with the corresponding entity, an indication of the length of initial wait time between previous automated calls with the corresponding entity, and / or other call data derived from previous automated calls with each corresponding entity of the group. As one non-limiting example, if the call history data indicates that a first entity of the group is associated with no IVR system or wait time, but a second entity of the group is associated with an IVR system that takes 20 seconds to operate, the request execution engine 154 can cause the automated assistant 115 to initiate a first automated call with the first entity of the group simultaneously (e.g., simultaneously or within a threshold length of time (e.g., 500 milliseconds, 1 second, and / or other length of time)) with a second automated call with the second entity of the group. As another non-limiting example, if the call history data indicates that a first entity of the group is associated with an IVR system that takes 10 seconds to operate and a second entity of the group is associated with an IVR system that takes 30 seconds to operate, the request execution engine 154 can cause the automated assistant 115 to initiate a first automated call with the first entity of the group and initiate a second automated call with the second entity of the group 10 seconds later.As yet another non-limiting example, if the request is predicted to take 45 seconds to execute based on the complexity of the request, and if the call history data indicates that the first entity of the group is not associated with an IVR system (or any wait time), but the second entity of the group is associated with an IVR system that takes 20 seconds to operate, the request execution engine 154 can cause the automated assistant 115 to initiate a second automated call with the second entity of the group 25 seconds after initiating the first automated call with the first entity. In this example, based on the 20 seconds to operate the IVR and the 25 second delay to the call initiation time being equal to 45 seconds to execute the request, the automated assistant may be given sufficient time (e.g., 45 seconds) for the first entity to execute the request and initiate the second automated call with the second entity, such that a representative associated with the second entity will answer the second call after execution of the first automated call. Furthermore, this also gives the automated assistant an opportunity to end the second automated call with the second entity if the first entity indicates that it can fulfill the request. In contrast to this example, assume that the first entity is associated with an IVR system that takes 20 seconds to operate and the second entity is not associated with an IVR system or wait time. In this example, the call start time for the second automated call with the second entity may be 65 seconds to give the automated assistant 20 seconds to operate the IVR system and perform an action before initiating the second automated call with the second entity. The call start time may be determined in this dynamic manner for each of the entities in the group.

[0042] In various implementations, the call start time utilized in alternating automatic calls may be determined with consideration to reduce latency in providing notifications (e.g., as described below with respect to the notification engine 113) and / or reduce the overall length of the user's interaction with the automated assistant. In many of these implementations, the call start time is further determined with consideration to reduce unnecessary calls and / or unnecessarily long call times with one or more entities that are subsequently called (e.g., in situations where the previously called entity is able to satisfy the action). Determining the call start time in such a manner can balance the technical advantages achieved by reducing latency and / or interaction time with resources utilized in performing the automatic calls. For example, alternating automatic calls can selectively call entities without initiating and performing an automatic call with each entity in the group until a request contained in the user input is satisfied. Moreover, the automated assistant is connected to a representative associated with each of the entities at the appropriate time so that the automated assistant does not have to wait for the first automated call to be completed in order to initiate a second automated call and remain on the line or operate the IVR, thereby reducing interaction time in executing the automated calls.

[0043] In implementations where the request is based on a received user input detected at the client device 110 via the user input engine 111, the request execution engine 154 can continue to initiate the automated call with the group entities in any of the manners described above until the request is fulfilled by one of the entities of the group. In some versions of those implementations, the request execution engine 154 can cause the automated assistant 115 to preemptively terminate the automated call with the other entities when a representative associated with a given entity indicates that the given entity can fulfill the request. For example, assume that a representative associated with a first entity of the group provides information indicating that the first entity can fulfill the request and a second automated call with a second entity of the group is being initiated. In this example, because the first entity can fulfill the request, the request execution engine 154 can cause the automated assistant to terminate the second automated call with the second entity.

[0044] In other versions of those implementations, the request execution engine 154 can continue executing the execution of the additional automated call, rather than preemptively terminating the execution, even if the representative of a given entity of the automated call indicates that the given entity can fulfill the request. For example, if the additional automated call progresses past the breakpoint stage, the request execution engine 154 can cause the automated assistant 115 to continue executing the conversation with the additional entity. For example, assume that the request is for a status update regarding hand sanitizer, and further assume that the automated assistant 115 is engaged in a conversation with a representative associated with the first entity during the first automated call and has already initiated a second automated call with a second entity of the group. In this example, if the representative associated with the first entity indicates that the first entity has hand sanitizer available, and the automated assistant 115 is waiting on the call with the second entity or is operating an IVR system associated with the second entity, the automated assistant 115 can terminate the second automated call. However, if the automated assistant is already engaged in a conversation with a representative associated with the second entity, the automated assistant 115 may continue to perform the second automated call. Additionally, the update engine 155 can cause search results related to the second entity to be updated based on information received from the representative associated with the second entity, as described above.

[0045] In particular, the automated calls initiated and executed in response to receiving user input detected at the client device 110 via the user input engine 111 may be initiated and executed by an automated assistant running locally on the client device 110 or a cloud-based automated assistant. By alternating between initiating and executing automated calls in response to requesting user input, the techniques described herein can conserve computational resources of the client device associated with the entities. For example, by alternating between automatic calls to identify a given entity that can fulfill the request, the techniques described herein avoid calling each of a group of entities simultaneously to identify a given one of the entities that can fulfill the request. Moreover, by continuing execution of an ongoing automated call after another entity has already indicated that it can fulfill the request, the techniques described herein avoid wasting computational resources consumed by an ongoing automated call compared to simply terminating the ongoing call.

[0046] In implementations in which the request is based on received user input detected at the client device 110 via the user input engine 111 and in response to determining that a representative associated with the given entity indicates that the given entity can fulfill the request included in the user input, the request execution engine 154 can cause a synthetic utterance including user data (e.g., stored in a user data database) to be rendered at the client device of the representative associated with the given entity. The user data can include name information of the user of the client device 110, address information of the user of the client device 110, contact information of the user of the client device 110 (e.g., phone number, email address, and / or other contact information), financial information of the user of the client device 110, and / or other information of the user of the client device 110. In some versions of those implementations, the approval engine 156 can cause a prompt to be rendered on the display of the client device 110 (e.g., using the rendering engine 112) for the automated assistant 115 to approve providing the user information to the representative. The prompt may also include a selectable element that, when selected, allows the user of the client device 110 to participate in the automated call. If the user of the client device 110 approves the provision of the user data to the representative of the given entity, the automated assistant may cause the synthetic speech audio data including the user data to be audibly rendered at the client device of the representative associated with the given entity. However, if the user of the client device 110 does not approve the provision of the user data, the automated assistant 115 refrains from causing the synthetic speech audio data including the user data to be audibly rendered at the client device of the representative associated with the given entity.In other versions of those implementations, the user of the client device 110 may have previously approved the provision of user data, and the automated assistant 115 may cause the synthetic speech audio data including the user data to be audibly rendered at the client device of a representative associated with a given entity without seeking approval from the user.

[0047] In some versions of those implementations where the request included in the user input is a request for a status update regarding an item and a representative associated with a given entity indicates that the item is available, the automated assistant 115 can ask the representative to reserve the item at the given entity for the user of the client device 110, or can provide financial information to the representative to obtain the item on behalf of the user of the client device. The request fulfillment engine 154 can determine whether to reserve or obtain the item based on the user input (e.g., "find and reserve disinfecting towels," "find and get disinfecting towels for me"), or can prompt the user (e.g., via the rendering engine 112) as to whether the user of the client device 110 wants to reserve or obtain the item.

[0048] For example, assume that the request is for a status update regarding disinfectant towels, and further assume that a representative of the given entity indicates that the disinfectant towels are available at the given entity. Furthermore, in response to determining that the item is available at the given entity, and in response to determining that the user of the client device 110 desires to obtain the item, the automated assistant 115 can cause synthesized speech audio data including the user's name, contact information, and financial information to be rendered at the client device of the representative associated with the given entity (if approved). As another example, assume that the request is for the availability of plumbing repair services at a given time and date, and further assume that a representative of the given entity indicates that the plumbing repair services are available at the given time and date. Further, in response to determining that plumbing repair services are available for a given time and date, and in response to determining that a user of the client device 110 wishes to make an appointment for a given time and date, the automated assistant 115 can cause synthesized speech audio data including the user's name, address, and contact information to be rendered at the client device of a personnel associated with the given entity (if approved).

[0049] In an implementation in which the request is based on a received user input detected at the client device 110 via the user input engine 111, the automated assistant 115 can cause the notification engine 113 to generate a notification including one or more results of the automated call. The notification generated by the notification engine 113 can be rendered (e.g., using the rendering engine 112) visually via a display of the client device 110 and / or audibly via a speaker of the client device 110. The results of the conversation can include, for example, an indication of whether a given entity is able to fulfill the request, an indication of whether an item or service has been reserved or reserved for the user of the client device 110, an indication of whether an item has been obtained for the user of the client device 110, an indication that approval of submitting user data of the user of the client device 110 is required to obtain the item, an indication of a request to the user of the client device 110 to join the automated call with the given entity, and / or other results of the conversation. The notifications generated by the notification engine 113 may be banner notifications, pop-up notifications, email notifications, text message notifications, voice message notifications, and / or other types of notifications. Generating notifications based on the results of a conversation is described in more detail herein (e.g., with respect to Figures 5B-5D, 6B, and 6C).

[0050] In various implementations, the rendering engine 112 can cause a transcript of a conversation between the automated assistant 115 and a representative of a given entity to be rendered on the display of the client device 110. Moreover, the display can also include one or more selectable elements that, when selected, allow the user of the client device 110 to join the conversation, end the automated call with the given entity, and / or perform other actions. In implementations in which the automated assistant 115 is involved in multiple conversations with multiple entities, a corresponding transcript of each of the conversations can be rendered on the display of the client device 110. In some versions of those implementations, since the display area of ​​the screen of the client device 110 may be limited, each transcript can be associated with a tab in the automated assistant interface that allows the user of the client device 110 to switch between the transcripts of the multiple conversations. In other implementations, the transcripts may not be displayed, and the user of the client device 110 may simply receive a notification as described above.

[0051] In implementations where the request is based on analyzing the query activity via the query analysis engine 151, the request execution engine 154 can cause the automated assistant 115 to initiate a corresponding automated call with a given entity of the group of entities to perform the action included in the request. In some implementations, the request execution engine 154 can cause multiple instances of the automated assistant 115 described herein to simultaneously initiate and execute multiple corresponding automated calls with corresponding ones of the entities of the group. For example, a first instance of the automated assistant 115 can initiate and execute a first automated call with a first entity of the group for each of the entities of the group, a second instance of the automated assistant 115 can simultaneously initiate and execute a second automated call with a second entity of the group, and so on. In other words, the instances of the automated assistant 115 can initiate and execute each of the automated calls in parallel. In other implementations, the corresponding automated calls can be initiated and executed in batches (e.g., with a subset of the group of entities) or sequentially. Notably, each instance of the automated assistant 115 may run remotely on a server (e.g., a cloud-based automated assistant) along with the automated request system 180, and the client device 110 may be omitted.

[0052] Each of the instances of the automated assistant can engage in a corresponding conversation with a corresponding one of the entities to perform the action included in the request. The time delay engine 157 can determine a time delay before starting to perform a subsequent corresponding automated call with the corresponding entity based on information received during each of the corresponding conversations. Furthermore, the time delay engine 157 can assign various time periods as time delays based on information received during the corresponding automated calls, business hours associated with the corresponding entities, laws that constrain whether and / or when the automated call can be performed, calculations and / or network bandwidth considerations (e.g., "slow" times for phone calls, such as early morning hours). Moreover, the time delay engine 157 can store the time periods assigned as time delays in association with the corresponding entities in the time delay database 157A. In an implementation where the request is for a status update on an item, the time period assigned to the time delay can be based on an indication of the available quantity of the item. For example, if the available amount of the item is within a first amount range, a first period of time can be assigned as a time delay, if the available amount of the item is within a second amount range, a second period of time can be assigned as a time delay, and so on for various amount ranges of the item. As another example, if the available amount of the item meets a threshold amount, a first period of time can be assigned as a time delay, if the available amount of the item does not meet the threshold amount, a second period of time can be assigned as a time delay.

[0053] For example, assume that the request is a request for a status update regarding toilet paper, and further assume that a representative associated with the given entity indicates that toilet paper is available at the given entity. Moreover, assume that a representative associated with the given entity indicates that there is a sufficient amount of toilet paper that will not sell out by the end of business for the day. In this example, the time delay engine 157 can assign a first period to the time delay associated with the given entity such that the automated assistant 115 does not initiate a subsequent corresponding automated call with the given entity until at least the next day. In contrast, assume that a representative associated with the given entity indicates that there is only a sufficient amount of toilet paper that will sell out by the end of business for the day. In this example, the time delay engine 157 can assign a second period to the time delay associated with the given entity such that the automated assistant 115 initiates a subsequent corresponding automated call with the given entity at a later time in the day. Notably, these time delays elapse at different times based on information received during the automated call. Moreover, search results related to a given entity may be updated based on information received during the automated call as described above with respect to update engine 155. In these and other manners, a corresponding time delay may be dynamically determined for each entity based on information received during the automated call with the entity. Dynamically determining the time delay in such a manner may balance the technical advantages achieved by maintaining up-to-date information from the entity with the resources utilized in performing the automated call. In other words, dynamically determining the time delay may ensure that information from the entity is up-to-date while preventing premature execution of another automated call to the entity, which would result in unnecessary utilization of computational and / or network resources.

[0054] By using the techniques described herein, various technical advantages can be achieved. As one non-limiting example, an automated call can be intelligently initiated using the automated request system 180, so that the automated assistant 115 can successfully execute an automated call to satisfy the request more quickly and efficiently. By intelligently initiating an automated call, the length and number of conversations can be reduced by using the techniques disclosed herein, so that both network and computational resources can be saved. As another non-limiting example, search results related to the entity can be updated based on information received during the conversation. By updating search results related to the entity, the entity receives fewer calls from multiple other users, since each of the multiple other users does not have to individually initiate calls with each of the entity to request status updates on various items and / or to inquire about the availability of various services. Moreover, while FIG. 1 is described herein in connection with requesting status updates on a single item or service, it should be understood that this is for illustrative purposes and is not intended to be limiting. For example, the automated assistant 115 can request status updates on multiple items during a corresponding automated call with a given one of the entities of the group.

[0055] FIG. 2 illustrates a flow chart showing an example method 200 of determining a time delay for a subsequent automated call with a given entity based on a status update regarding an item of interest received during the automated call with the given entity. For convenience, the operations of the method 300 are described with reference to a system that performs those operations. The system of the method 300 includes one or more processors and / or other components of a computing device, such as the client device 110 of FIG. 1, the computing devices 410A, 410B of FIG. 4A-4D, the computing device 510 of FIG. 5A-5D, the computing device 610 of FIG. 6A-6C, the computing device 710 of FIG. 7, one or more remote servers, and / or other computing devices. Moreover, although the operations of the method 300 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0056] At block 252, the system identifies items of interest. In some implementations, the items of interest may be identified based on analyzing the query activity of multiple users to identify a spike in queries submitted, as described in more detail above (e.g., with respect to the query analysis engine 151 of FIG. 1). In some versions of those implementations, the spike in queries submitted may originate from a particular geographic region. In other implementations, the items of interest may be identified based on user input from a user that causes the system to perform the method 200. At block 254, the system identifies a group of entities related to the items of interest. The entities of the group may be stored in association with the items of interest in one or more databases (e.g., the entity identification engine 153 and the entity database 153A of FIG. 1). The system may retrieve the entities from one or more of the databases. In implementations where the spike in queries submitted may be from a particular geographic region, the entities in the group may be constrained to entities located in the particular geographic region.

[0057] In block 256, the system begins executing a corresponding automated call with each of the entities of the group to request a corresponding status update regarding the item of interest. The system can begin the automated call by rendering one or more corresponding instances of a synthetic utterance that automatically places a call and requests a corresponding status update regarding the item of interest. In some implementations, the system can begin executing multiple corresponding automated calls with each of the entities of the group simultaneously using a corresponding instance of an automated assistant (e.g., the automated assistant 115). In other implementations, the system may only begin executing calls with a subset of the entities such that the automated calls with each of the entities are initiated in batches until the execution of the automated calls with each of the entities of the group is completed. In various implementations, the instance of the automated assistant can be an instance of a cloud-based assistant.

[0058] At block 258, during the execution of the corresponding automated call with a given entity of the group of entities, the system A) requests a corresponding status update regarding the item of interest from a representative associated with the given entity, and B) receives information regarding the item of interest from the representative. The system can execute the corresponding automated call by engaging in a conversation with the representative associated with the given entity. Furthermore, the system can request the corresponding status update by rendering one or more of the corresponding instances of the synthetic speech during each of the automated calls. For example, the system can cause the synthetic speech (e.g., "Hello, do you sell toilet paper?", "Is there any toilet paper available?", "How much toilet paper do you have available?") to be rendered at a computing device of the representative associated with the given entity. Furthermore, the system can process audio data corresponding to information received during the corresponding automated call using a speech recognition model (e.g., as described in more detail with respect to the speech recognition model of FIG. 1). The information received from the representative can be in response to the rendering of the synthetic speech. As described in more detail above with respect to FIG. 1, the system can intelligently respond to the representative during the conversation.

[0059] At block 260, the system updates the search results associated with the given entity to include information about the item of interest. The search results may be updated for each of the entities in the group in response to receiving the corresponding information from each of the entities. In some implementations, the updated search results may also include a time indicator of when the corresponding automated call was made and the information was received. For example, assume that the received information indicates that Example Store 1 has toilet paper available. The search results associated with Example Store 1 may be updated to indicate that toilet paper is available at Example Store 1, along with the time and date that the information was received.

[0060] In block 262, the system determines whether the given entity is still associated with the item of interest. The system may determine whether the given entity is still associated with the item of interest based on processing the received information about the item of interest from block 258. If, in any iteration of block 262, the system determines that the given entity is still associated with the item of interest, the system proceeds to block 264. For example, assume that the system causes a synthetic utterance of "Do you sell toilet paper?" to be rendered at a computing device associated with the representative to verify that the given entity is still associated with the item of interest. Further assume that the received information indicates that Example Store 1 sells toilet paper. In this example, the system has verified that the given entity is still associated with the item of interest.

[0061] In block 264, the system determines a time delay before starting to execute a corresponding subsequent automated call with the given entity based on the received information about the item. Further, the system may assign the determined time delay to the given entity in one or more databases. The determined time delay may be a period of time that is assigned to the given entity based on the available amount of the item of interest. For example, assume that personnel associated with Example Store 1 indicate that they have enough toilet paper for three business days. In this example, the system may assign a period of three days as the time delay based on the received information indicating that Example Store 1 has enough toilet paper for three business days. In contrast, assume that personnel associated with Example Store 1 expects that they only have enough toilet paper for a morning business day. In this example, the system may assign a period of five hours as the time delay based on the received information indicating that Example Store 1 has enough toilet paper for a morning business day.

[0062] In block 266, the system determines whether the time delay assigned to the given entity determined in block 264 has elapsed. If in an iteration of block 266 the system determines that the time delay has not elapsed, the system continues to monitor the elapse of the time delay assigned to the given entity in block 266. Notably, corresponding automated calls may initially be performed in parallel, while corresponding subsequent automated calls are initiated based on the elapse of the determined time delay. However, if in an iteration of block 266 the system determines that the time delay determined in block 264 has elapsed, the system proceeds to block 268.

[0063] At block 268, the system begins making a corresponding subsequent automated call with the given entity to request further status updates regarding the item of interest. The system may request further status updates regarding the item of interest in a manner similar to that described above with respect to block 258. Additionally, the system may return to block 264 and determine further time delays based on information received during the corresponding subsequent automated call. The system may continue to determine time delays for the corresponding further automated calls until one or more conditions are met. The one or more conditions may include, for example, a decrease in queries issued for the item of interest, a threshold amount of time (e.g., one week, two weeks, one month, etc.), and / or other conditions are met.

[0064] However, if in an iteration of block 262 the system determines that the entity is no longer associated with the item of interest, the system may refrain from determining a time delay and proceed to block 270 to remove the given entity from the group of entities associated with the item of interest. As a result, the system will not initiate a subsequent automated call with the given entity to request further status updates regarding the item of interest. However, the system may still subsequently place calls to the given entity to request status updates regarding other items of interest.

[0065] It should be understood that while the time delays illustrated in FIG. 2 are described with respect to a single entity, this is for illustrative purposes and is not intended to be limiting. As noted above, the automated assistant can simultaneously initiate execution of multiple automated calls with a group of entities. During each of the automated calls, a time delay can be assigned to a corresponding one of the entities, and the time delay for each of the entities can be different based on information received regarding the item of interest from a representative associated with the corresponding one of the entities.

[0066] FIG. 3 illustrates a flow chart showing an example method 300 of determining whether to initiate an additional automated call with an additional entity based on information received during an automated call with a given entity. For convenience, the operations of method 300 are described with reference to a system that performs those operations. This system of method 300 includes one or more processors and / or other components of a computing device, such as client device 110 of FIG. 1, computing devices 410A, 410B of FIG. 4A-4D, computing device 510 of FIG. 5A-5D, computing device 610 of FIG. 6A-6C, computing device 710 of FIG. 7, one or more remote servers, and / or other computing devices. Moreover, although the operations of method 300 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0067] At block 352, the system receives a request from the user's computing device to initiate execution of an automated call on behalf of the user. The request may be based on a user input received at the user's computing device. In some implementations, the user input may be a touch input or a typed input from a search interface or an automated assistant interface displayed on the user's computing device. For example, the user input may be a touch input directed at a selectable element of the search interface that, when selected, causes the system to initiate an automated call to execute the request. In other implementations, the user input may be a spoken input detected via a microphone of the user's computing device. The spoken input may be processed using a speech recognition model and / or an NLU model to determine that the user input includes a request (e.g., as described with respect to FIG. 1).

[0068] At block 354, the system identifies an action to be performed during the automated call based on the request. The action may be identified based on one or more terms included in the user input. The action may include, for example, requesting a status update on the item of interest from a given entity or entities, reserving the item of interest at a location associated with the given entity, obtaining the item of interest at a location associated with the given entity, requesting availability of a service from a given entity or entities, and / or other actions that may be performed using the automated call described herein. For example, the action may include an action to obtain toilet paper at a specific entity, or an action to obtain toilet paper generally without specifying a specific entity. As another example, the action may include an action to find the cheapest electrical contractor available for a given time period. Identifying the action to be performed is described in more detail above (e.g., with respect to the request processing engine 152 of FIG. 1).

[0069] In block 356, the system identifies groups of entities each capable of fulfilling the request based on the action to be performed during the automated call. The entities of the group may be stored in one or more databases (e.g., entity identification engine 153 and entity database 153A of FIG. 1) in association with the item or service included in the request. The system may retrieve the entities from one or more of the databases. Moreover, the entities in the group may be constrained to entities located within a threshold distance from the current location of the user's computing device. For example, assume that the action included in the request is a request for a status update on an item, and the system may include in the group of entities only entities that are within a five mile radius associated with the item.

[0070] At block 358, the system initiates execution of the automated call with a given entity of the group of entities. The system may initiate the automated call by automatically placing the call and rendering one or more corresponding instances of synthetic speech requesting a corresponding status update on the item of interest. In some implementations, the system may initiate execution of the automated call using an automated assistant that runs locally on the computing device of the user that provided the request. In other implementations, the system may initiate execution of the automated call using an automated assistant that runs remotely on a server (e.g., a cloud-based automated assistant). In still other implementations, the system may initiate execution of the automated call using an automated assistant that runs partly on the user's computing device and partly on a server (e.g., the automated assistant 115 shown in FIG. 1).

[0071] At block 360, during the execution of the automated call with the given entity, the system converses with a representative of the given entity to perform an action. During the conversation, the system can perform the action and receive information about the action from the representative (e.g., information about the availability of an item or service and / or other information described herein). The system can perform the action by rendering one or more of the corresponding instances of the synthetic speech during each of the automated calls. For example, the system can cause the synthetic speech (e.g., "Hello, do you sell toilet paper?", "Is there toilet paper available?", "How much toilet paper do you have available?") to be rendered at an additional computing device of the representative associated with the given entity. Additionally, the system can process audio data corresponding to information received during the corresponding automated call using a speech recognition model (e.g., as described in more detail with respect to the speech recognition model of FIG. 1). The information received from the representative can be in response to the rendering of the synthetic speech. As described in more detail above with respect to FIG. 1, the system can intelligently respond to the representative during the conversation.

[0072] In block 362, the system determines whether to start making an additional automatic call with an additional entity of the group of entities. The system may determine to start making an additional automatic call with the additional entity based on a call start time associated with the additional entity. The call start time may be specific to the additional entity. In some implementations, the call start time may be determined based on call history data stored in association with the entity and / or the additional entity (e.g., call history data database 154A of FIG. 1). For example, the call start time for the additional entity may be determined based on whether the entity and / or the additional entity is associated with an IVR system, an average length of previous automatic calls with the entity and / or the additional entity, an average wait time associated with the entity and / or the additional entity, and / or any other call data that may be collected from previous automatic calls with the entity and / or the additional entity. For example, if the determined call start time is 30 seconds after the start of the automatic call with the given entity, it may be based on determining that both the given entity and the additional entity are associated with a wait time of 30 seconds. In these examples, the automated calls are staggered so that the automated assistant can engage with representatives associated with each of the entities 30 seconds apart, rather than simultaneously engaging representatives of both entities. In some additional and / or alternative implementations, the call start time can be determined based on the outcome of the automated calls with the entities. Continuing with the above example, if the system latency is only 20 seconds instead of the expected 30 seconds, the call start time can be changed from 30 seconds to 20 seconds so that a similar staggered execution of the automated calls is achieved. If, in the iteration of block 362, the system determines not to initiate execution of additional automated calls, the system proceeds to block 366. Among other things, this indicates that the request can be processed by the given entity and that the given entity has indicated that it can process the request before the call start time.For example, this may indicate that a given entity has quickly indicated that it can fulfill the request. However, if in an iteration of block 362 the system determines to initiate execution of additional automated calls with additional entities, the system may proceed to block 364. As described above with respect to block 358, the system may initiate the additional automated calls using automated assistants local to the user's computing device, remote at a server, and / or a combination thereof.

[0073] In block 364, the system determines whether to preemptively terminate the automated call with the given entity and / or the additional automated call with the additional entity. The system can determine whether to preemptively terminate the automated call and / or the additional automated call based on information received during each of the automated calls. For example, if either the entity or the additional entity indicates that the item or service is not available, the corresponding automated call can be preemptively terminated. In implementations where the given entity indicates that it can fulfill the request and additional personnel associated with the additional entity are already engaged in a conversation with the automated assistant, the system may not preemptively terminate the additional automated call. In some versions of those implementations, the system can still perform the action, and search results associated with the additional entity can be updated based on the execution of the action. In this way, the computational resources of additional computing devices of personnel associated with the additional entity are not wasted by terminating the additional automated call without knowing any information about the original request. If, in an iteration of block 364, the system does not determine to preemptively terminate the automatic call with the given entity and / or the additional automatic calls with the additional entities, the system may monitor for one or more signals to preemptively terminate the automatic call with the given entity and / or the additional automatic calls with the additional entities in block 364. However, if, in an iteration of block 364, the system determines to preemptively terminate the automatic call with the given entity and / or the additional automatic calls with the additional entities, the system proceeds to block 366.

[0074] At block 366, the system generates a notification based on the outcome of the conversation during the automated call with the given entity and / or the additional automated call with the additional entity. A notification may be generated for each automated call initiated by the system. Additionally, the notification may include, for example, an indication of availability for an item or service, whether the system has reserved an item or service on behalf of the user, whether the system has obtained an item on behalf of the user, and / or other results of the conversation. In some implementations, the notification may include a prompt for the user. The prompt may include, for example, a prompt asking the user to join the automated call, a prompt asking the user to acknowledge that the system may provide name information, address information, contact information, financial information, and / or other user data of the user of the computing device. At block 368, the system causes the notification to be rendered visually and / or audibly via the user's computing device. The notification may be rendered visually via a display of the computing device associated with the user or another computing device, and / or audibly via a speaker of the computing device associated with the user or another computing device. Generating and rendering notifications is described in more detail herein (eg, with respect to FIGS. 1, 5B-5D, 6B, and 6C).

[0075] While method 300 is described herein with respect to a given entity and additional entities, it should be understood that this is for purposes of illustration and not intended to be limiting. It should be understood that a call start time for each of the other entities in the group can be determined, and each of the other entities in the group can be called until the request is satisfied. If none of the entities in the group can satisfy the request, additional groups of entities can be identified as described above (e.g., with respect to entity identification engine 153 of FIG. 1).

[0076] 4A-4C, non-limiting examples of determining a time delay for a subsequent automated call with various entities are illustrated. In particular, a transcript of the automated call of FIG. 4A-4C is illustrated in the graphical user interface 480A of the computing device 410A. It should be understood that this is for purposes of example to illustrate the techniques described herein and is not intended to be limiting. For example, the automated call described herein may be initiated and executed by an automated assistant (e.g., using the automated request system 180 of FIG. 1) for simplicity. The automated assistant may be a cloud-based automated assistant that runs remotely on a server and / or other computing device.

[0077] In some implementations, as described in more detail above (e.g., in connection with the query analysis engine 151 and the query processing engine 152 of FIG. 1 ), the automated assistant can identify a request for a status update regarding an item or service of interest based on analyzing the query activity of multiple users (e.g., stored in the query activity database 151A of FIG. 1 ). The automated call can be initiated and executed by the automated assistant in response to identifying the request for a status update regarding the item or service of interest. The automated assistant can identify the request based, for example, on determining whether a threshold number of queries for a given item or service have been issued by multiple users. In some versions of those implementations, the one or more conditions can include determining whether a threshold number of queries for a given item or service have been issued by multiple users within a given time frame, as described above with respect to FIG. 1 . In some versions of those implementations, the query activity can be constrained to queries issued by users in a particular geographic region.

[0078] In some implementations, the automated assistant can further identify a group of entities related to the identified request, as described in more detail herein (e.g., in connection with entity identification engine 153 of FIG. 1). The group of entities can be stored in association with the item or service included in the request (e.g., in entity database 153A) along with other data (e.g., physical location, web page, and / or other data) regarding each of the entities. A corresponding one of the automated calls with a given one of the entities of the group can be initiated and executed by the automated assistant such that the automated assistant can initiate and execute a corresponding automated call with each individual one of the entities of the group. In implementations where the issued query originates from a particular geographical area, the entities included in the group can also be constrained to entities located in that particular geographical area. The automated assistant can begin initiating and executing the automated call in response to identifying the item or service based on analyzing the query activity and in response to identifying a group of entities related to the item or service included in the request. Additionally, the automated assistant can execute the corresponding one of the automated calls by rendering an instance of synthetic speech at a corresponding representative's computing device associated with the corresponding one of the entities. Moreover, the automated assistant can process audio data corresponding to information received from the corresponding agent during the corresponding automated call. In this manner, the automated assistant can engage in a conversation with the agent, and subsequent synthetic speech can be based on the information received from the corresponding agent.

[0079] For example, with reference to Figures 4A-4C, assume that an automated assistant identifies 15,000 submitted queries over a six-hour period that include the term "toilet paper" and originate from a geographic area of ​​Jefferson County, Kentucky. In this example, the automated assistant may determine that the 15,000 submitted queries over the six-hour period meet one or more conditions for initiating multiple corresponding automated calls with a group of entities located in Jefferson County, Kentucky to request corresponding status updates regarding the availability of toilet paper. Thus, the automated assistant may access one or more databases (e.g., entity database 153A of Figure 1) to identify multiple entities located in Jefferson County, Kentucky that are stored in association with the item toilet paper, and may include the identified entities in the group of entities. As shown in Figures 4A-4C, the group of entities includes at least Example Store 1, Example Store 2, and Example Store 3.

[0080] More specifically, the automated assistant may initiate a first automated call with Example Store 1 to request a status update regarding toilet paper in Example Store 1 as shown in FIG. 4A, a first automated call with Example Store 2 to request a status update regarding toilet paper in Example Store 2 as shown in FIG. 4B, and a first automated call with Example Store 3 to request a status update regarding toilet paper in Example Store 3 as shown in FIG. 4C. Notably, each of the first automated calls with each of the entities in FIG. 4A through FIG. 4C may be initiated and executed in parallel (e.g., as shown as being initiated at 9:00 AM on June 11, 2020). Additionally, each of the first automated calls may be initiated and executed using a corresponding instance of the automated assistant.

[0081] As described above, the automated assistant can execute each of the corresponding automated calls by causing an instance of the synthetic utterance to be rendered at a computing device of a representative associated with a given one of the entities. Specifically, the synthetic utterance may include a request for a status update regarding toilet paper, and optionally other information. For example, as shown in FIG. 4A, the automated assistant can cause a synthetic utterance 452A1 to be rendered at a computing device of a representative of Example Store 1, the synthetic utterance 452A1 including a request for a status update regarding toilet paper at Example Store 1 (e.g., "I'd like to see if there is any toilet paper available") and also including a request for the available amount of toilet paper at Example Store 1 (e.g., "Are we going to sell out by the end of today?"). Furthermore, the automated assistant can process audio data corresponding to information 454A1 received in response to the synthetic utterance (e.g., "We have some available, but we're going to sell out by this afternoon") to determine an indication of whether toilet paper is available at Example Store 1 and the amount available. As another example, as shown in FIG. 4B , the automated assistant may cause a synthetic utterance 452B1 including a request for a status update regarding toilet paper in Example Store 2 (e.g., “I'd like to see if there's any toilet paper available”) to be rendered at the computing device of the Example Store 2 representative, but may refrain from requesting the available amount of toilet paper in Example Store 2 until the Example Store 2 representative confirms that there is toilet paper available in Example Store 2 based on processing audio data corresponding to information 454B1 received from the Examples Store 2 representative (e.g., “Yes there is”).In response to determining that Example Store 2 has toilet paper available, the automated assistant may then request a status update on the available amount of toilet paper at Example Store 2 in a further synthesized utterance 452B2 (e.g., "Are you sure you're going to sell out by the end of today") and can determine an indication of the available amount of toilet paper at Example Store 2 based on processing further received information 454B2 (e.g., "No," indicating that Example Store 2 will not be selling out of toilet paper by the end of today). Additionally, the automated assistant can process audio data corresponding to information 454A1 received in response to the synthesized utterance (e.g., "We have some available, but we're going to sell out by this afternoon"). As yet another example, as shown in FIG. 4C , the automated assistant can cause a synthetic utterance 452C1 to be rendered at a computing device of an associate of Example Store 3 that includes a request for a status update regarding toilet paper in Example Store 3 (e.g., “I want to see if there is any toilet paper available”) and also includes a request for the available amount of toilet paper in Example Store 3 (e.g., “Are we going to sell out today?”). Additionally, the automated assistant can process audio data corresponding to information 454C1 received in response to the synthetic utterance (e.g., “We don't sell toilet paper”) to determine that Example Store 3 should not be stored in association with toilet paper. In this example, Example Store 3 can be removed from the group of entities based on the received information 454C1.

[0082] For each of the corresponding automated calls of Figures 4A-4C, the automated assistant can assign a period of time as a time delay to each of the entities of the group. Once the time delay has elapsed, for a corresponding one of the entities of the group, the automated assistant can initiate and execute a subsequent corresponding automated call with the corresponding entity to request a corresponding further status update regarding the item. The period of time assigned as a time delay can be determined based on information received from a representative associated with a given entity (e.g., as described in more detail above with respect to the time delay engine 157). In some implementations, the automated assistant can provide an indication of the period of time assigned to the time delay for the corresponding entity during each of the first automated calls.

[0083] For example, as shown in FIG. 4A, the automated assistant can cause a further synthetic utterance 452A2 (e.g., "I'll call you back in the afternoon for the next update") to be rendered at the computing device of the Example Store 1 representative indicating a period of time to be assigned as a time delay for Example Store 1. In this example, this period of time may be only a few hours (e.g., 5 hours as shown in FIG. 4A) because information 454A1 indicated that Example Store 1 "is likely to run out of [toilet paper] this afternoon." Thus, once the time delay has elapsed (e.g., five hours after the first automated call with Examples Store 1 (e.g., at 2:00 p.m. on the same day)), the automated assistant can initiate and perform a second automated call to request a corresponding further status update regarding the toilet paper by causing a further synthetic utterance 452A3 to be rendered at the computing device of the Example Store 1 representative, processing audio data corresponding to further information 454A2 received from the Example Store 1 representative's response to the further synthetic utterance 452A3, and causing a further synthetic utterance 452A4 including an indication of a further time delay determined based on the further information 452A2 received during the second automated call with the Example Store 1 representative. For example, if the Example Store 1 representative indicates in further information 452A2 that Example Store 1 will not have any toilet paper until a package arrives in three days, the period assigned as a further time delay may be 72 hours to avoid calling Example Store 1 for the next three days until the next package arrives, since Example Store 1 has already indicated that it will not have any toilet paper.

[0084] 4B, the automated assistant can cause yet a further synthetic utterance 452B3 (e.g., "I'll call you back tomorrow") to be rendered at the computing device of the Example Store 2 representative indicating a period of time to be assigned as a time delay for Example Store 2. In this example, the period of time may be a 24-hour period because information 454B2 indicated that Example Store 2 would not be out of toilet paper that day. Thus, once the time delay has elapsed (e.g., 24 hours after the first automated call with Examples Store 2 (e.g., at 9:00 a.m. the next day)), the automated assistant can initiate and execute a second automated call to request a corresponding further status update regarding the toilet paper by causing yet another synthetic utterance 452B4 to be rendered at the Example Store 2 representative's computing device, processing audio data corresponding to yet another information 454B3 received from the Example Store 2 representative's response to the yet another synthetic utterance 452B4, and causing yet another synthetic utterance 452A5 including an indication of a further time delay determined based on yet another information 452B3 received during the second automated call with the Example Store 2 representative. Thus, as illustrated by the time delays in FIGS. 4A and 4B, the automated assistant can determine a dynamic period to assign to the time delay based on the information received during the automated call.

[0085] 4C , the automated assistant may cause a further synthetic utterance 452C2 to be rendered at the computing device of the representative of Example Store 3 indicating that the automated assistant will refrain from asking Example Store 3 about toilet paper in the future. In this example, the automated assistant may also refrain from determining a period of time to assign as a time delay to Example Store 3 because, as described above, Example Store 3 was removed from the group of entities related to the item based on received information 454C1. Thus, a second automated call with Example Store 3 is not initiated or performed.

[0086] In various implementations, search results related to the entities of the group may be updated to include information received during the corresponding automated call. In some versions of those implementations, the search results may also be updated to include a time indicator of when a status update was requested. For example, FIG. 4D shows updated search results related to each of the various entities based on the results of the automated call from FIGS. 4A-4C on a graphical user interface 480B of a user's computing device 410B. The graphical user interface 480B of FIG. 4D further includes a text response interface element 484 that a user can select to generate user input via a virtual keyboard or other touch input and / or typed input, and a voice response interface element 485 that a user can select to generate user input via a microphone of the client device 410. In some implementations, a user may generate user input via a microphone without selecting the voice response interface element 485. For example, active monitoring of audible user input via a microphone may be performed to eliminate the need for a user to select the voice response interface element 485. In some of those and / or other implementations, the voice response interface element 485 may be omitted. Moreover, in some implementations, the text reply interface element 484 may additionally and / or alternatively be omitted (e.g., the user may only provide auditory user input).The graphical user interface 480 of Figures 4A-4D also includes system interface elements 481, 482, 483 with which a user may interact to cause the computing device 410 to perform one or more actions.

[0087] With reference to Figure 4D, assume that the user of computing device 410B is also located in Jefferson County, Kentucky, and further assume that the user of computing device 410 submitted a search query (e.g., spoken or typed via an automated assistant interface or a search interface) including the term "toilet paper" (e.g., as indicated by URL 411) after the first automated call of Figures 4A-4C, but before the second automated call. Further assume that a first search result 420 related to Example Store 1, a second search result 430 related to Example Store 2, and a third search result 440 related to Example Store 3 are each in response to the search query. In this example, the first search result 420 may be updated to include information 421 from the first automated call with Example Store 1 indicating that toilet paper was "available" at Example Store 1 at "9:00 a.m." "today," but that toilet paper may be sold out at Example Store 1 "this afternoon." Further, the second search result 430 may be updated to include information 431 from the first automated call with Example Store 2 indicating that toilet paper was "available" at Example Store 1 "this morning" but should not be "sold out today" at Example Store 2. Moreover, the third search result 440 may be updated to include information 441 from the first automated call with Example Store 3 indicating that Example Store 3 "does not sell toilet paper." The information 421, 431 in the search results 420, 430 may be updated in a similar manner after the second automated call and any other subsequent automated calls thereafter. In this manner, the search results associated with each of the entities in the group may be updated to reflect the availability of various items associated with each of the corresponding entities as the automated calls are performed by the automated assistant.

[0088] 4A to 4D are described herein for requesting a status update for a single item of interest, it should be understood that this is for illustrative purposes and is not intended to be limiting. For example, if an automated assistant identifies multiple items of interest for which a status update is to be requested, and a given one of the entities in the group is associated with multiple of the items, the automated assistant can request a status update for each of the multiple items in a first automated call to avoid having to initiate and execute multiple automated calls with the given entity to request a status update for each of the multiple items. Moreover, it should be noted that when an automated assistant requests a status update for multiple items, different durations can be assigned to the time delay for each of the multiple items. For example, if an automated assistant initially requests a status update regarding both toilet paper and hand sanitizer, and a representative associated with a given entity provides information indicating that there is sufficient toilet paper by the end of the day's business, but no hand sanitizer for the next four days, the automated assistant may allocate a 24-hour period for initiating and executing a subsequent automated call to request further status updates regarding the toilet paper, and a 96-hour period for initiating and executing a subsequent automated call to request further status updates regarding the hand sanitizer.

[0089] 4A-4D are described herein with respect to requesting status updates regarding items of interest, it should be understood that this is for purposes of illustration and is not intended to be limiting. As one non-limiting example, the automated assistant may also request status updates regarding the availability of services. For example, if the automated assistant determines that there is a spike in queries placed for auto body repairs and / or roof repairs (e.g., after a hailstorm), automated calls may be initiated and performed in a similar manner, and search results related to entities that provide these services may be updated in a similar manner.

[0090] 5A-5D, various non-limiting examples are illustrated for initiating automated calls with various entities on behalf of a user, with notifications associated with each of the various entities being rendered based on the outcome of the automated calls. Each of FIGS. 5A-5D illustrates a computing device 510 having a graphical user interface 580 related to a user request to initiate and / or execute an automated call on behalf of the user of the computing device 510. One or more aspects of an automated assistant (e.g., the automated assistant 115 of FIG. 1) associated with the user of the computing device 510 may be implemented locally on the computing device 510 and / or on other computing devices in network communication with the computing device 510 in a distributed manner (e.g., via the network 190 of FIG. 1). For simplicity, the operations of FIGS. 5A-5D are described herein as being performed by an automated assistant (e.g., using the automated request system 180 of FIG. 1). Although the computing device 510 of FIGS. 5A-5D is illustrated as a mobile phone, it should be understood that this is not intended to be limiting. The computing device 510 may be, for example, a server, a standalone speaker, a speaker connected to a graphical user interface, a laptop, a desktop computer, and / or any other client device capable of making phone calls and running instances of an automated assistant.

[0091] The graphical user interface 580 of FIGS. 5A-5D further includes a text response interface element 584 that a user can select to generate user input via a virtual keyboard or other touch input and / or typed input, and a voice response interface element 585 that a user can select to generate user input via a microphone of the computing device 510. In some implementations, a user may generate user input via a microphone without selecting the voice response interface element 585. For example, active monitoring of audible user input via a microphone may be performed to eliminate the need for a user to select the voice response interface element 585. In some of these and / or other implementations, the voice response interface element 585 may be omitted. Moreover, in some implementations, the text response interface element 584 may additionally and / or alternatively be omitted (e.g., a user may only provide audible user input). The graphical user interface 580 of FIGS. 5A-5D also includes system interface elements 581, 582, 583 that a user can interact with to cause the computing device 510 to perform one or more actions.

[0092] In various implementations, the automated call may be initiated and executed in response to receiving a user input requesting the automated assistant to initiate and execute the automated call. The user input may be a spoken, touch, and / or typed input including an instruction to initiate and execute the automated call. Furthermore, the automated assistant may execute the actions included in the request on behalf of a given user of the computing device 510 during the automated call. As shown in FIG. 5A, the graphical user interface 580 includes a first search result 520 related to the Hypothetical Toy Store 1 entity and a second search result 530 related to the Hypothetical Toy Store 2 entity displayed in response to the query "Hot Christmas Toy 2020" (e.g., as shown in the text response interface element 584). Furthermore, the search results 520, 530 are displayed in a browser application accessible on the computing device (e.g., as shown by the URL 511 of "www.exampleurl0.com / "). Additionally, the search results 520, 530 may also include, for example, corresponding information 521, 531 regarding one or more items of interest (e.g., Hot Christmas Toys). For example, first information 521 associated with the first search result 520 may indicate that Hypothetical Toy Store 1 ran out of Hot Christmas Toys six days ago, and second information 531 associated with the second search result 530 may indicate that the status of Hot Christmas Toys is unknown for Hypothetical Toy Store 2 and the status has never been updated.

[0093] In some implementations, the search results 520, 530 may be associated with various selectable graphical elements that, when selected, cause the computing device 510 to perform a corresponding action. For example, when the call graphical element 522, 532 associated with a given one of the search results 520, 530 is selected, the user input may indicate that a call should be initiated and performed. As another example, when the directions graphical element 523, 533 associated with a given one of the search results 520, 530 is selected, the user input may indicate that a navigation action should be initiated and performed. In other implementations, although not shown, when a general auto-call graphical element is selected, the user input may explicitly indicate that an auto-call should be initiated and performed.

[0094] In some implementations, as shown in FIG. 5A , in response to receiving user input to initiate and perform a call action with any entity (e.g., in response to receiving a selection of one of the call interface elements 522, 532 or the general auto-call graphical element), a call details interface 570 may be rendered at the computing device 510. In some versions of those implementations, the call details interface 570 may be rendered at the computing device 510 as part of a graphical user interface 580. In some versions of those implementations, the call details interface 570 may be a separate interface from the graphical user interface 580 that overlays the user interface and may include an interface element 586 that allows a user to expand the call details interface 570 to display additional call details (e.g., by swiping up on the interface element 586) and / or close the call details interface 570 (e.g., by swiping down on the interface element 586). In other implementations, the interface element 586 may be used to access and close the call details interface 570 at any time. It should be understood that while call details interface 570 is shown as being below graphical user interface 580, this is for purposes of illustration and is not intended to be limiting. For example, call details interface 570 may be rendered above graphical user interface 580, to the side of graphical user interface 580, or in an interface entirely separate from graphical user interface 580.

[0095] The call details interface 570 may include multiple segments of selectable text (or graphical elements) in various implementations. In some implementations, the segments may be selectable such that a given one of the segments may be selected to cause the computing device 510 to perform a corresponding action. As shown in FIG. 5A, the call details interface 570 includes a first segment of selectable text 571 called "Automatic Call" and a second segment of selectable text 572 called "Regular Call". Furthermore, the first segment of selectable text 571 may be selected to provide an indication of a desire to use the automated assistant to initiate an automated call, and the second segment of selectable text 572 may be selected to cause the automated assistant to initiate a regular call. Notably, in some versions of those implementations, the segments of selectable text may include other related segments of selectable text. For example, the first segment of selectable text "Auto-Call" 571 may include a first other segment of selectable text "Status Update on Toy" 571A associated with the action of requesting a status update during the automated call, a second other segment of selectable text "Reserve Toy" 571B associated with the action of setting aside the toy for the user of computing device 510 during the automated call, and a third other segment of selectable text "Buy Toy" 571C associated with the action of obtaining the toy for the user of computing device 510 during the automated call. It should be understood that while the automated calls described herein with respect to Figures 5A-5D are initiated from a browser-based application, this is for purposes of illustration and is not intended to be limiting.For example, the automated call may be initiated from various software applications accessible on the computing device 510 (e.g., an automated assistant application, a contacts application, an email application, a text or SMS messaging application, and / or other software applications), and the automated call is initiated using spoken input from a home screen of the computing device 510, from a lock screen of the computing device 510, and / or from other states of the computing device 510.

[0096] For example, assume that a user input is detected at the computing device 510 to initiate and execute an automated call to the first search result 520 of "Hypothetical Toy Store 1." The user input may be, for example, a spoken input of "Find Hot Christmas Toy 2020" or a touch input directed to the call interface element 522 (and / or one of the selected segments of text 571, 571A-571C). In various implementations, the automated assistant may identify a group of entities that are each stored in association with "Hot Christmas Toy" in one or more databases (e.g., entity database 153A of FIG. 1) based on the user input indicating "Hot Christmas Toy." In some implementations, the automated assistant may constrain the entities included in the group to entities that are stored in association with Hot Christmas Toy and that are within a threshold distance of the location of the computing device 510 (e.g., determined based on GPS data generated by a GPS sensor of the computing device 510). For example, the automated assistant may initially include in the group of entities all entities stored in association with Hot Christmas Toy that are within four miles of the location of computing device 510. If not, the automated assistant may increase the threshold distance to include all entities within eight miles of the location of computing device 510, and so on. In some additional and / or alternative implementations, the automated assistant may continue to identify entities to be included in the group until a threshold number of entities are identified for inclusion in the group. For example, the automated assistant may continue to identify entities to be included in the group by increasing the threshold distance until five entities are included in the group.If more than a threshold number of entities are included in the group, then entities that meet one or more criteria may be included in the group (e.g., as described in more detail above in connection with entity identification engine 153). Notably, a group of entities related to Hot Christmas Toy may be identified even though the user input may be directed to initiating an automated call with a particular entity and not specify an entity.

[0097] The automated assistant can initiate and run automated calls with one or more of the entities of the group until the request is satisfied. In some implementations, the automated assistant can initiate a first automated call with a first entity of the group in response to receiving user input and determine a call start time for starting execution of additional automated calls with corresponding additional entities of the group. In some versions of those implementations, the call start time of each of the additional automated calls can be a fixed threshold length of time (e.g., 15 seconds, 30 seconds, 45 seconds, and / or other length of time). For example, the automated assistant can, for each of the entities of the group, initiate a first automated call with the first entity of the group, initiate a second automated call with the second entity of the group 30 seconds after initiating the first automated call with the first entity, initiate a third automated call with the third entity of the group 30 seconds after initiating the second automated call with the second entity, and so on. In other versions of these implementations, the automated assistant can determine each call start time based on call history data (e.g., stored in call history database 154A) associated with a corresponding one of the entities of the group. The call history data can include, for example, an indication that the corresponding entity is associated with an IVR system and a length of time to operate the IVR system, an indication of a length of time of a previous automated call with the corresponding entity, an indication of a length of initial wait time between previous automated calls with the corresponding entity, and / or other call data derived from previous automated calls with each corresponding entity of the group. Determining the call resolution time associated with each of the entities of the group is described in more detail herein (e.g., in connection with request execution engine 154 of FIG. 1).Moreover, the automated assistant may cause a corresponding notification to be rendered at the computing device 510 (e.g., audibly via a speaker of the computing device 510 and / or visually via a graphical user interface 580 of the computing device 510) including the result of the automated call, a prompt for the user of the computing device 510, and / or other information regarding the automated call.

[0098] For example, with particular reference to FIG. 5B, assume that a user input (e.g., a touch input directed at the first other selectable segment of text 571A of FIG. 5A or a spoken input of "Find Hot Christmas Toy") is detected at computing device 510 to initiate an automated call to request a status update regarding Hot Christmas Toy. Further assume that the automated assistant identifies at least Hypothetical Toy Store 1 and Hypothetical Toy Store 2 as entities in a group related to Hot Christmas Toy. The automated assistant can initiate an automated call with Hypothetical Toy Store 1 and request a status update regarding Hot Christmas Toy as an action to be performed based on the user input. For example, the automated assistant can cause synthesized utterance 552B1 (e.g., "I'd like to check if Hot Christmas Toys are available at your store") to be rendered at a computing device associated with a representative of Hypothetical Toy Store 1, and can process audio data corresponding to information 554B1 (e.g., "Please wait while I check") received in response to the synthesized utterance 552B1.

[0099] Additionally, the automated assistant can process audio data corresponding to further information 554A2 when the representative of Hypothetical Toy Store 1 responds to the automated call (e.g., "We're out of stock") and cause further synthetic speech 552B2 (e.g., "Thanks for checking") to be rendered at a computing device associated with the representative of Hypothetical Toy Store 1. Based on the outcome of the conversation, the automated assistant can cause a notification 579B to be visually rendered in a graphical user interface (e.g., via a notification interface 590, which can replace call details interface 570 and can be expanded or closed using interface elements 586 similar to call details interface 570). Moreover, the automated assistant can cause the first search result 520 shown in FIG. 5A to be updated to indicate that Hot Christmas Toy is out of stock at Hypothetical Toy Store 1, and optionally to include a time indicator of when the further information 554B2 indicated that Hot Christmas Toy was out of stock (e.g., "9:00:00 AM on December 1, 2020," this morning, etc.). Updating the search results based on information received during the automated call is described in more detail above (e.g., in connection with FIGS. 1 and 4D).

[0100] During the automatic call with Hypothetical Toy Store 1, the automated assistant (or a separate instance thereof) may also initiate an additional automatic call with Hypothetical Toy Store 2 based on a determined call start time associated with Hypothetical Toy Store 2. In some implementations, the call start time associated with Hypothetical Toy Store 2 may be 0, such that the automated assistant can run automatic calls with Hypothetical Toy Store 1 and Hypothetical Toy Store 2 in parallel (e.g., in a manner similar to that described above in connection with initiating and running the first automatic call in FIG. 4A-4C). In other implementations, the additional automatic calls may be staggered, such that the automated assistant (and / or a separate instance thereof) generally only engages with one entity at a given time. In some versions of those implementations, the call start time associated with Hypothetical Toy Store 2 may be a set length of time associated with each of the entities in the group, such that each additional automatic call is initiated at 30 second intervals. In other versions of these implementations, the call start time associated with Hypothetical Toy Store 2 may be based on call history data associated with Hypothetical Toy Store 1 and Hypothetical Toy Store 2.

[0101] In some additional and / or alternative versions of those implementations, the call start time associated with Hypothetical Toy Store 2 may be dynamic and based on the progress of the conversation between the automated assistant and the representative of Hypothetical Toy Store 1. For example, assume that the call start time associated with Hypothetical Toy Store 2 determined according to one or more of the above techniques is 30 seconds. However, if the waiting time described above in connection with FIG. 5A exceeds the original 30-second call start time, the call start time associated with Hypothetical Toy Store 2 may be adjusted to a dynamic time when the conversation with the representative of Hypothetical Toy Store 1 resumes (e.g., when further information 554B2 is received). In contrast, assume that the representative of Hypothetical Toy Store 1 indicated in further information 554B2 (or immediately in information 554B1) that a Hot Christmas Toy was available. In this example, an additional automated call with Hypothetical Toy Store 2 may be preemptively terminated. However, if an additional automated call with Hypothetical Toy Store 2 has already been initiated and the automated assistant is engaged in an additional conversation with a representative from Hypothetical Toy Store 2, the automated assistant may continue the conversation but may change behavior to request a status update regarding the Hot Christmas Toy and update the search results related to Hypothetical Toy Store 2 accordingly. Determining call resolution times associated with entities of a group is described in more detail herein (e.g., in connection with request fulfillment engine 154 of FIG. 1 ).

[0102] For example, and with particular reference to Figure 5C, assume that a user input is detected at computing device 510 to initiate an automated call to obtain a Hot Christmas Toy on behalf of the user of computing device 510. The user input may be, for example, a touch input directed at the second other selectable segment 571B of text in Figure 5A, or a spoken input of "Find a Hot Christmas Toy and buy it for me." Assume further that the outcome of the automated call with a representative from Hypothetical Toy Store 1 indicates that Hypothetical Toy Store 1 does not have the Hot Christmas Toy in stock, based on the conversation shown in Figure 5B. Assume also that an additional automated call with Hypothetical Toy Store 2 is initiated 30 seconds after the automated call with Hypothetical Toy Store 1. Similar to the conversation described above in connection with FIG. 5B, the automated assistant can cause synthesized speech 552C1 (e.g., "I'd like to see if Hot Christmas Toys are available at your store") to be rendered at a computing device associated with a representative of Hypothetical Toy Store 2, and can process audio data corresponding to information 554C1 received in response to the synthesized speech (e.g., "Yes, there are two available").

[0103] Additionally, the automated assistant can process audio data corresponding to further information 554C2 when the Hypothetical Toy Store 2 representative responds to the further synthetic utterance 552C2 by causing further synthetic utterance 552C2 (e.g., "Can I buy for Jane Doe?", where Jane Doe is the user of computing device 510) to be rendered at a computing device associated with the Hypothetical Toy Store 2 representative. Based on the outcome of the conversation, the automated assistant can cause a notification 579C to be visually rendered via a graphical user interface (e.g., via a notification interface 590, which can replace the call details interface 570 and can be expanded or closed using an interface element 586 similar to the call details interface 570).

[0104] Among other things, the notification 579C can indicate that there is a Hot Christmas Toy at Hypothetical Toy Store 2 and may also include a prompt requesting the user of the computing device 510 (e.g., Jane Doe) to approve providing financial information to a representative of Hypothetical Toy Store 2 to obtain the Hot Christmas Toy at Hypothetical Toy Store 2 on the user's behalf. Moreover, the notification interface 590 may also include selectable graphical elements responsive to the prompt, such as a first selectable graphical element 579C1 for approving the provision of financial information, a second selectable graphical element 579C2, and a third selectable element 579C3 for the user to join an additional automated call. In various implementations, the user of the computing device 510 may have previously approved the provision of financial information and the prompt may be omitted. If the user indicates that the automated assistant should provide financial information (e.g., via selection of first selectable graphical element 579C1), the user may receive a further notification indicating that the Hot Christmas Toy has been obtained at Hypothetical Toy Store 2, provide directions to Hypothetical Toy Store 2, create a reminder to pick up the Hot Christmas Toy at Hypothetical Toy Store 2, and / or perform other actions on behalf of the user of computing device 510 based on the outcome of the conversation during the additional automated call with Hypothetical Toy Store 2. Moreover, once the automated assistant has obtained the Hot Christmas Toy, it may preemptively terminate other ongoing automated calls with other entities in the group.

[0105] For example, and with particular reference to Figure 5D, assume that a user input is detected at computing device 510 to initiate an automated call to reserve a Hot Christmas Toy on behalf of the user of computing device 510 (as opposed to, e.g., the request to get a Hot Christmas Toy in Figure 5C). The user input could be, for example, a touch input directed at the third other selectable segment 571C of the text in Figure 5A, or a spoken input of "Find a Hot Christmas Toy and reserve it for me.") Assume further that the outcome of the automated call with a representative from Hypothetical Toy Store 1 indicates that Hypothetical Toy Store 1 does not have the Hot Christmas Toy in stock, based on the conversation shown in Figure 5B. Assume also that an additional automated call with Hypothetical Toy Store 2 is initiated 30 seconds after the automated call with Hypothetical Toy Store 1. Similar to the conversation described above in connection with FIGS. 5B and 5C, the automated assistant can cause synthesized speech 552D1 (e.g., "I'd like to see if Hot Christmas Toys are available at your store") to be rendered at a computing device associated with a representative of Hypothetical Toy Store 2, and can process audio data corresponding to information 554D1 received in response to the synthesized speech (e.g., "Yes, there are two available").

[0106] Additionally, the automated assistant can process the audio data corresponding to the further information 554D2 when the Hypothetical Toy Store 2 representative responds to the further synthetic utterance 552D2 by causing the further synthetic utterance 554D2 (e.g., "Can I make a reservation for Jane Doe?", where Jane Doe is the user of the computing device 510) to be rendered at a computing device associated with the Hypothetical Toy Store 2 representative. Based on the outcome of the conversation, the automated assistant can cause a notification 579D to be visually rendered via a graphical user interface (e.g., via a notification interface 590, which can replace the call details interface 570 and can be expanded or closed using an interface element 586 similar to the call details interface 570). Among other things, the notification 579D can indicate that the Hypothetical Toy Store 2 has a Hot Christmas Toy and that the automated assistant has reserved it on behalf of the user of the computing device 510 at the Hypothetical Toy Store 2. Although not shown in FIG. 5D , the automated assistant may further cause the computing device 510 to render navigation information to Hypothetical Toy Store 2, pick-up instructions for when the user of the computing device 510 arrives at Hypothetical Toy Store 2 to pick up the Hot Christmas Toy, a link to a web page related to Hypothetical Toy Store 2, an indication that an appointment or reminder has been created to pick up the Hot Christmas Toy at Hypothetical Toy Store 2, and the like.

[0107] In various implementations, the automated assistant can cause the first search result 520 shown in FIG. 5A to be updated to indicate that the Hot Christmas Toy is out of stock at Hypothetical Toy Store 1, and optionally to include a time indicator of when the further information 554B2 indicated that the Hot Christmas Toy was out of stock (e.g., "December 1, 2020 at 9:00:00 AM," this morning, etc.). Additionally, the automated assistant can cause the second search result 530 shown in FIG. 5A to be updated to indicate that the Hot Christmas Toy is low in stock or that only one unit is left at Hypothetical Toy Store 2, and optionally to include a time indicator of the information 554C1, 554D1 that indicated that the Hot Christmas Toy was available (e.g., "December 1, 2020 at 9:00:30 AM," this morning, etc.). Updating search results based on information received during an automated call is described in more detail above (eg, in connection with Figures 1 and 4D).

[0108] Moreover, other automated assistants associated with other users located in the same geographic area can leverage the information obtained by the automated assistant during the automated call to perform further actions. For example, if a neighbor of the user of computing device 510 provides a user input to their respective automated assistant to get a Hot Christmas Toy on their behalf at 10:00 a.m., each of the neighbors' automated assistants can analyze the updated search results and remove Hypothetical Toy Store 1 from any identified groups because no Hot Christmas Toy was available at Hypothetical Toy Store 1 at 9:00 a.m., and may initiate an automated call with Hypothetical Toy Store 2 to see if one is still available. By leveraging the information obtained by the automated assistant of computing device 510, other automated assistants can conserve computational resources on their respective computing devices and computing devices associated with the entity.

[0109] It should be understood that while Figures 5A-5D are described herein as performing actions related to an item of interest on behalf of a user of the computing device 510, this is for illustrative purposes and is not intended to be limiting. For example, referring now to Figures 6A-6C, additional non-limiting examples are shown of initiating automated calls with various entities on behalf of a user, with notifications related to each of the various entities being rendered based on the results of the automated calls. However, in contrast to Figures 5A-5D, Figures 6A-6C are described herein in the context of requesting the availability of an entity for a given period of time. Similar to Figures 5A-5D, Figures 6A-6C each illustrate a computing device 610 having a graphical user interface 680 related to a user request to initiate and / or perform an automated call on behalf of a user of the computing device 610. One or more aspects of an automated assistant (e.g., automated assistant 115 of FIG. 1) associated with a user of computing device 610 may be implemented locally on computing device 610 and / or on other computing devices in network communication with computing device 610 in a distributed manner (e.g., via network 190 of FIG. 1). For simplicity, the operations of FIGS. 6A-6C are described herein as being performed by an automated assistant (e.g., using automated request system 180 of FIG. 1). Although computing device 610 of FIGS. 6A-6C is illustrated as a mobile phone, it should be understood that this is not intended to be limiting. Computing device 610 may be, for example, a server, a standalone speaker, a speaker connected to a graphical user interface, a laptop, a desktop computer, and / or any other client device capable of making calls and running instances of an automated assistant.

[0110] The graphical user interface 680 of FIG. 6A-6C further includes a text response interface element 684 that a user can select to generate user input via a virtual keyboard or other touch input and / or typed input, and a voice response interface element 685 that a user can select to generate user input via a microphone of the computing device 610. In some implementations, a user may generate user input via a microphone without selecting the voice response interface element 685. For example, active monitoring of audible user input via a microphone may be performed to eliminate the need for a user to select the voice response interface element 685. In some of these and / or other implementations, the voice response interface element 685 may be omitted. Moreover, in some implementations, the text response interface element 684 may additionally and / or alternatively be omitted (e.g., a user may only provide audible user input). The graphical user interface 680 of FIG. 6A-6C also includes system interface elements 681, 682, 683 that a user can interact with to cause the computing device 610 to perform one or more actions.

[0111] In various implementations, the automated call can be initiated and executed in response to receiving a user input directed to the automated assistant requesting the automated assistant to initiate and execute the automated call. The user input can be a spoken, touch, and / or typed input including an instruction to initiate and execute the automated call. Additionally, the automated assistant can execute the actions included in the request on behalf of a given user of the computing device 610 during the automated call. As shown in FIG. 6A, the graphical user interface 680 includes a conversation between the automated assistant and a user (e.g., Jane Doe) of the computing device 610. The conversation includes an initial user input 656A1 (e.g., typed or spoken) including a request (e.g., "Can you find me a plumber... between 2:00 and 4:00 this afternoon"). The automated assistant can provide a response 652A1 (e.g., a text response or a synthesized speech) indicating that the automated assistant will "call some local repairmen to see if they are available."

[0112] In some implementations, as shown in FIG. 6A , the call details interface 670 may be rendered at the computing device 610 in response to receiving a user input 656A1. In some versions of those implementations, the call details interface 670 may be rendered at the computing device 610 as part of the graphical user interface 680. In some versions of those implementations, the call details interface 670 may be a separate interface from the graphical user interface 680 that overlays the user interface and may include an interface element 686 that allows a user to expand the call details interface 670 to display additional call details (e.g., by swiping up on the interface element 686) and / or close the call details interface 670 (e.g., by swiping down on the interface element 686). In other implementations, the interface element 686 may be used to access and close the call details interface 670 at any time. It should be understood that while the call details interface 670 is shown as being below the graphical user interface 680, this is for illustrative purposes and is not intended to be limiting. For example, the call details interface 670 may be rendered above the graphical user interface 680 , to the side of the graphical user interface 680 , or in an interface entirely separate from the graphical user interface 680 .

[0113] The call details interface 670 may include multiple segments of selectable text (or graphical elements) in various implementations. In some implementations, the segments may be selectable such that a given one of the segments may be selected to cause the computing device 610 to perform a corresponding action. As shown in FIG. 6A, the call details interface 670 includes a first segment of selectable text 671 called "Automatic Call" and a second segment of selectable text 672 called "Regular Call". Furthermore, the first segment of selectable text 671 may be selected to provide an indication of a desire to use the automated assistant to initiate an automatic call, and the second segment of selectable text 672 may be selected to cause the automated assistant to initiate a regular call. Notably, in some versions of those implementations, the segments of selectable text may include relevant information for an automatic call. For example, the first segment 671 of the "automated call" selectable text may include a first portion 671A of associated information "Service: Plumber" related to the plumbing repair service included in the user input 656A1, a second portion 671B of associated information "Time: 2:00 PM - 4:00 PM" related to the time for which the plumbing repair service included in the user input 656A1 is requested in the automated call, and a third portion 671C of associated information "Date: June 11, 2020" related to the date for which the plumbing repair service included in the user input 656A1 is requested in the automated call. By displaying portions of the information 671A, 671B, 671C related to the automated call, the user of the computing device 510 may be provided with an opportunity to modify the service being requested, the requested time of the service being requested, the requested date of the service being requested, and / or other information related to the service being requested by the user of the computing device 610. It should be understood that while the automated calls described herein with respect to Figures 6A-6C are initiated based on user input directed to the automated assistant, this is for purposes of illustration and is not intended to be limiting.For example, the automated call may be initiated from various software applications accessible on the computing device 610 (e.g., a browser-based application, a contact application, an email application, a text or SMS messaging application, and / or other software applications), and the automated call is initiated using spoken input from the home screen of the computing device 610, from the lock screen of the computing device 610, and / or from other states of the computing device 610.

[0114] For example, assume that a user input is detected at the computing device 610 to initiate and execute an automated call to request availability of plumbing repair services at a given time and date (e.g., as indicated by portions of information 671A, 671B, 671C). In various implementations, the automated assistant can identify a group of entities that are each stored in association with “plumbing repair services” in one or more databases (e.g., entity database 153A of FIG. 1 ) based on the user input indicating “plumbing repair services.” In some implementations, the automated assistant can constrain the entities included in the group to those entities that are stored in association with plumbing repair services and that are within a threshold distance of the location of the computing device 610 (e.g., as determined based on GPS data generated by the GPS sensor of the computing device 610). For example, the automated assistant can initially include in the group of entities all entities that are stored in association with plumbing repair services and that are within 10 miles of the location of the computing device 610. If not, the automated assistant can increase the threshold distance to include all entities that are within 20 miles of the location of the computing device 610, and so on. In some additional and / or alternative implementations, the automated assistant can continue to identify entities to be included in the group until a threshold number of entities are identified for inclusion in the group. For example, the automated assistant can continue to identify entities to be included in the group by increasing the threshold distance until five entities are included in the group. If more than the threshold number of entities are included in the group, entities that meet one or more criteria may be included in the group (e.g., as described in more detail above in connection with the entity identification engine 153). Notably, a group of entities related to plumbing repair services may be identified, even though user input may be directed to initiating an automated call with a particular entity or may not specify an entity.

[0115] The automated assistant can initiate and execute automated calls with one or more of the entities of the group until the request is satisfied. In some implementations, the automated assistant can initiate a first automated call with a first entity of the group in response to receiving user input 656A1 in FIG. 6A and determine a call start time for starting execution of additional automated calls with corresponding additional entities of the group. As described in more detail above (e.g., in connection with the request execution engine 154 in FIG. 1 and the call start time in FIG. 5A), the call start time of each of the additional automated calls is a time of a fixed threshold length, determined based on call history data (e.g., stored in the call history database 154A) associated with the corresponding one of the entities of the group, and can be dynamically modified based on the progress of the automated call. Moreover, the automated assistant can cause a corresponding notification including the result of the automated call, a prompt for the user of the computing device 510, and / or other information regarding the automated call to be rendered at the computing device 510 (e.g., audibly via the speaker of the computing device 510 and / or visually via the graphical user interface 580 of the computing device 510).

[0116] For example, with particular reference to FIG. 6B , assume that a user input (e.g., a touch input directed at the first selectable graphical element 671 of FIG. 6A or a spoken input corresponding to user input 656A1) to initiate an automated call to request availability of plumbing repair services is detected at computing device 510. Assume further that the automated assistant identifies at least Example Plumber 1 and Example Plumber 2 as entities in a group related to plumbing repair services. The automated assistant can initiate an automated call with Example Plumber 1 and request a status update regarding availability of plumbing repair services as an action to be performed based on the user input. For example, the automated assistant can cause a synthesized utterance 652B1 (e.g., “Are you available to repair a burst pipe in your basement between 2:00 PM and 4:00 PM today”) to be rendered at a computing device associated with a representative of Example Plumber 1, and can process audio data corresponding to information 654B1 received in response to the synthesized utterance 652B1 (e.g., “No, but I am available at 5:00 PM today”).

[0117] Further, the automated assistant can cause further synthetic utterance 652B2 (e.g., “I’ll check with Jane Doe if 5:00 PM is OK”) to be rendered at a computing device associated with the personnel of Example Plumber 1 in response to receiving information 654B1. Based on the outcome of the conversation, the automated assistant can cause a notification 679B to be visually rendered in a graphical user interface (e.g., via a notification interface 690, which can replace call details interface 670 and can be expanded or closed using an interface element 686 similar to call details interface 670). Notably, notification 679B can include an indication that Example Plumber 1 is not available during a given time period (e.g., between 2:00 PM and 4:00 PM), but is available during a period of time after the given time period (e.g., available at 5:00 PM), and can also include a prompt (e.g., “Are you OK?”). Moreover, the prompt can cause a selectable graphical element responsive to the prompt to be rendered in notification interface 590. The selectable graphical elements may include, for example, a first selectable graphical element 679B1 for accepting the reservation for the 5:00 PM time, a second selectable graphical element 679B2 for declining the reservation for the 5:00 PM time, and a third selectable element 679B3 for the user to participate in an additional automated call. In various implementations, the automated assistant may further provide name, contact, and address information to the Example Plumber 1 representative if the user of the computing device 610 selects the first selectable graphical element 679B1 (or otherwise affirms the 5:00 PM time). However, if the user does not respond to the prompt within a threshold period (e.g., 30 seconds, 45 seconds, and / or other period), the automated assistant may preemptively terminate the automated call with the Example Plumber 1 representative to avoid further wasting computational resources.More specifically, the automated assistant can also cause further synthetic utterances to be rendered at the computing device of the Example Plumber 1 representative to indicate the user's response (or lack of response) to the prompt, and the automated assistant can optionally cause further notifications to be rendered via notification interface 590.

[0118] During the automatic call with Example Plumber 1, the automated assistant (or a separate instance thereof) may also initiate an additional automatic call with Example Plumber 2 based on a determined call start time associated with Example Plumber 2. In some implementations, the call start time associated with Example Plumber 2 may be 0, such that the automated assistant can perform automatic calls with Example Plumber 1 and Example Plumber 2 in parallel (e.g., in a manner similar to that described above in connection with initiating and performing the first automatic call in FIG. 4A-4C). In other implementations, the additional automatic calls may be staggered, such that the automated assistant (and / or a separate instance thereof) generally only engages with one entity at a given time. In some versions of those implementations, the call start time associated with Example Plumber 2 may be a set length of time associated with each of the entities in the group, such that each additional automatic call is initiated at 30 second intervals. In other versions of those implementations, the call start time associated with Example Plumber 2 may be based on call history data associated with Example Plumber 1 and Example Plumber 2. In some additional and / or alternative versions of those implementations, as described in more detail above (e.g., in connection with FIG. 5B ), the call start time associated with Example Plumber 2 may be dynamic and based on the progress of the conversation between the automated assistant and the representative of Example Plumber 1.

[0119] For example, with particular reference to FIG. 6C, assume that a user input is detected at computing device 610 to initiate an automated call to book plumbing repair service on behalf of a user of computing device 610. Assume further that the outcome of the automated call with the representative of Example Plumber 1 indicates that Example Plumber 2 is not available for a given period of time based on the conversation shown in FIG. 6B. Assume also that an additional automated call with Example Plumber 2 is initiated 15 seconds after the automated call with Example Plumber 1. Similar to the conversation described above in connection with FIG. 6B, the automated assistant can cause a synthesized utterance 652C1 (e.g., "Are you available...today between 2:00 PM and 4:00 PM") to be rendered at a computing device associated with the representative of Example Plumber 2, and can process audio data corresponding to information 554C1 received in response to the synthesized utterance (e.g., "Yes, I'm available at 3:00 PM. Can you give me your address?").

[0120] Further, the automated assistant can cause further synthetic utterance 652C2 (e.g., “Jane Doe’s address is 401 S. 4th St,” where Jane Doe is the user of computing device 610) to be rendered at a computing device associated with the representative of Example Plumber 2. Based on the outcome of the conversation, the automated assistant can cause a notification 679C to be visually rendered via a graphical user interface (e.g., via a notification interface 690, which can replace call details interface 670 and can be expanded or closed using an interface element 686 similar to call details interface 670). Among other things, notification 679C can indicate that Example Plumber 2 is available for a given period of time (e.g., 3:00 PM, which is within a desired time range of 2:00 PM to 4:00 PM). Moreover, the automated assistant can reserve a time for the user of computing device 610 and render notification 679C of FIG. 6C before receiving further user input in response to the prompt of FIG. 6B. In this example, the automated assistant may automatically close the notification and prompt of Figure 6B. In this manner, the automated assistant may also request available periods from various entities and reserve a given one of the available periods for the user of the computing device 610.

[0121] In various implementations, although not shown in FIG. 4A to FIG. 4C, FIG. 5B to FIG. 5C, FIG. 6B, or FIG. 6C, the automated assistant may ask the representative associated with the group of entities to agree to engage in a conversation with the automated assistant when initiating each of the corresponding automated calls. For example, in relation to FIG. 4A, the automated assistant may cause a synthetic utterance of "Do you agree to engage in a conversation with the automated assistant" to be rendered on the computing device of the representative of Example Store 1 before triggering the rendering of synthetic utterance 452A1. If the automated assistant receives information in response to the prompt indicating that the representative of Example Store 1 agrees to engage in the conversation, the automated assistant may continue the conversation in the manner described above in relation to FIG. 4A. Moreover, the automated assistant may not need to ask the representative of Example Store 1 to agree to engage in a conversation with the automated assistant in the second automated call because the representative of Example Store 1 has already agreed to engage in the conversation. However, if the automated assistant does not receive information in response to the prompt indicating that the Example Store 1 representative agrees to engage in the conversation, the automated assistant may end the first automated call with the Example Store 1 representative.

[0122] 7 is a block diagram of an example computing device 710 that may be optionally utilized to perform one or more aspects of the techniques described herein. In some implementations, one or more of the client device, the cloud-based automated assistant component, and / or other components may comprise one or more components of the example computing device 710.

[0123] Computing device 710 typically includes at least one processor 714 that communicates with a number of peripheral devices via a bus subsystem 712. These peripheral devices may include, for example, a storage subsystem 724 including a memory subsystem 725 and a file storage subsystem 726, user interface output devices 720, user interface input devices 722, and a network interface subsystem 716. The input and output devices enable user interaction with computing device 710. The network interface subsystem 716 provides an interface to external networks and is coupled to corresponding interface devices in other computing devices.

[0124] The user interface input devices 722 may include a keyboard and pointing device such as a mouse, trackball, touchpad, or graphics tablet, a scanner, a touch screen integrated into a display, a voice recognition system, an audio input device such as a microphone, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and methods for inputting information into the computing device 710 or into a communications network.

[0125] The user interface output devices 720 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a flat panel device such as a cathode ray tube (CRT), a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display, such as via an audio output device. In general, use of the term "output device" is intended to include all possible types of devices and methods for outputting information from the computing device 710 to a user or to another machine or computing device.

[0126] Storage subsystem 724 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, storage subsystem 724 may include logic for performing selected aspects of the methods disclosed herein, as well as for implementing the various components shown in FIG.

[0127] These software modules are generally executed by the processor 714 alone or in combination with other processors. The memory 725 used in the storage subsystem 724 may include several memories, including a main random access memory (RAM) 730 for storing instructions and data during program execution, and a read-only memory (ROM) 732 in which fixed instructions are stored. The file storage subsystem 726 may provide persistent storage for program and data files and may include a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules implementing the functionality of some implementations may be stored by the file storage subsystem 726, in the storage subsystem 724, or in other machines accessible by the processor 714.

[0128] The bus subsystem 712 provides a mechanism for allowing the various components and subsystems of the computing device 710 to communicate with each other as intended. Although the bus subsystem 712 is shown generally as a single bus, alternative implementations of the bus subsystem 712 may use multiple buses.

[0129] The computing device 710 can be of different types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of the computing device 710 shown in Figure 7 is intended only as a specific example to illustrate some implementations. Numerous other configurations of the computing device 710 are possible, having more or fewer components than the computing device shown in Figure 7.

[0130] In situations where the systems described herein may collect or otherwise monitor personal information about a user or utilize personal and / or monitored information, the user may be given the opportunity to control whether a program or feature collects user information (e.g., the user's social network, social behavior or activities, occupation, user preferences, or the user's current geographic location) or to control whether and / or how to receive content from a content server that may be relevant to the user. Also, some data may be handled in one or more ways before being stored or used, such that personally identifiable information is removed. For example, the user's identification information may be handled such that personally identifiable information cannot be determined about the user, or the user's geographic location may be generalized, in which case geographic location information is obtained (such as to the city, zip code, or state level) such that the user's specific geographic location cannot be determined. Thus, the user may control how information is collected about the user and / or used.

[0131] In some implementations, a method is provided that is implemented by one or more processors, the method including identifying an item of interest, identifying a group of entities related to the item of interest, and initiating execution of a corresponding automated call with each of the entities of the group by the automated assistant to render one or more corresponding instances of a synthetic utterance requesting a corresponding status update regarding the item of interest. The method further includes, during execution of the corresponding automated call with a given entity of the group of entities, requesting a corresponding status update regarding the item of interest from a representative associated with the given entity through rendering one or more corresponding instances of the synthetic utterance, and receiving information regarding the item of interest from the representative associated with the given entity in response to requesting the corresponding status update. The method further includes, based on the received information regarding the item of interest, determining a time delay before initiating execution of a subsequent corresponding automated call with the given entity to request a further corresponding status update regarding the item of interest, and initiating execution of the subsequent corresponding automated call with the given entity in response to the passage of the time delay by the automated assistant.

[0132] These and other implementations of the techniques disclosed herein may optionally include one or more of the following features.

[0133] In some implementations, the received information regarding the item of interest indicates availability of the item of interest at a given entity and the available quantity of the item of interest at a given entity.

[0134] In some versions of those implementations, determining the time delay based on the received information regarding the item of interest includes determining whether the available quantity of the item of interest at the given entity meets a quantity threshold, and assigning a first period of time as the time delay in response to determining that the available quantity of the item of interest at the given entity meets the quantity threshold. In some further versions of those implementations, the method further includes assigning a second period of time as the time delay in response to determining that the available quantity of the item of interest at the given entity does not meet the quantity threshold. The second period of time is shorter in length than the first period of time and, as a result, elapses before the first period of time. In still further versions of those implementations, the second period of time elapses during the same day that the automated call was initiated, and the first period of time elapses during the next day following the same day that the automated call was initiated.

[0135] In some versions of those implementations, the method further includes updating search results related to the given entity to indicate one or more of an availability of the item of interest at the given entity or an available quantity of the item of interest at the given entity. In some further versions of those implementations, the method further includes analyzing the updated search results related to the given entity and previously updated search results related to the given entity, and predicting a future availability of the item of interest at the given entity at a future time or predicting a future available quantity of the item of interest at the given entity at a future time based on analyzing the updated search results and the previously updated search results.

[0136] In some implementations, the method further includes verifying whether the given entity is associated with the item of interest during execution of a corresponding automated call with the given entity, and determining the time delay is in response to determining that the given entity is associated with the item of interest based on the verification. In some versions of those implementations, the method further includes refraining from determining the time delay in response to determining that the given entity is not associated with the item of interest based on the verification, and removing the given entity from the group of entities associated with the item of interest. Removing the given entity from the group of entities associated with the item of interest prevents the initiation of any subsequent automated call with the given entity regarding the item of interest.

[0137] In some implementations, the person associated with the given entity is a human person or one of an additional automated assistants associated with the given entity.

[0138] In some implementations, initiating execution of the corresponding automated call with each of the entities of the group to request a corresponding status update regarding the item of interest includes initiating execution of the corresponding automated call with the given entity and simultaneously initiating execution of at least additional corresponding automated calls with additional entities of the group of entities. In some versions of those implementations, the method further includes requesting a corresponding status update regarding the item of interest from an additional person associated with the additional entity through rendering one or more of the corresponding instances of the synthetic speech during execution of the additional corresponding automated call with the additional entity, and receiving additional information regarding the item of interest from the additional person associated with the additional entity in response to requesting the corresponding status update. The method further includes determining an additional time delay before initiating execution of a subsequent additional corresponding automated call with the additional entity to request a further corresponding status update regarding the item of interest based on the received additional information regarding the item of interest. A first period is assigned to the time delay, and a separate second period is assigned to the additional time delay. The method further includes initiating execution of a subsequent additional corresponding automated call with the additional entity in response to the passage of the additional time delay, by the automated assistant.

[0139] In some implementations, identifying the group of entities associated with the item of interest includes accessing one or more databases to identify entities stored in association with the item of interest as the group of entities, in some versions of those implementations, each of the identified group of entities is located in a particular geographic region.

[0140] In some implementations, the method for requesting a status update regarding an item of interest from a personnel associated with the given entity further includes generating synthetic speech audio data capturing one or more of the corresponding instances of the synthetic speech requesting a corresponding status update regarding the item of interest, and rendering the one or more of the corresponding instances of the synthetic speech includes causing the synthetic speech to be rendered at an additional computing device of the personnel associated with the given entity.

[0141] In some implementations, a method implemented by one or more processors is provided, the method including receiving a request from a user's computing device to initiate execution of an automated call on behalf of the user, identifying an action to be performed during the automated call based on the request, identifying a group of entities each capable of satisfying the request based on the action to be performed during the automated call, and initiating execution of the automated call with a given entity of the group of entities by the automated assistant. The method further includes, during execution of the automated call with the given entity, engaging in a conversation with a representative associated with the given entity to perform the action through rendering one or more corresponding instances of a synthetic utterance related to the request, and initiating execution of additional automated calls with additional entities of the group of entities by the automated assistant. The method further includes generating a notification including a result of the conversation with the representative, and causing the notification to be rendered via the user's computing device.

[0142] These and other implementations of the techniques disclosed herein may optionally include one or more of the following features.

[0143] In some implementations, the method further includes determining that the given entity will fulfill the request based on an outcome of the conversation during execution of the automated call, and terminating execution of an additional automated call with the additional entity in response to determining that the given entity will fulfill the request.

[0144] In some implementations, the method further includes, upon initiating execution of the automated call with the given entity, determining that a representative associated with the given entity has placed the automated call on hold, determining that a representative associated with the given entity has resumed the automated call, and in response to determining that a representative associated with the given entity has resumed the automated call, initiating execution of an additional automated call with an additional entity.

[0145] In some implementations, the method further includes, during execution of the corresponding automatic call with the given entity, determining a call start time for initiating execution of the additional automatic call with the additional entity based on call history data associated with the entity, and in response to determining that the current time matches the call start time, initiating execution of the additional automatic call with the additional entity. In some versions of those implementations, the call history data associated with the additional entity includes an indication that the entity is associated with the interactive voice response system and one or more of a length of time for operating the interactive voice response system, an indication of a length of time of a previous automatic call with the entity, or an indication of a length of initial wait time between previous automatic calls with the entity.

[0146] In some implementations, the action to be performed during the automated call includes requesting a status update regarding an item of interest associated with an entity of the group. In some versions of those implementations, engaging in a conversation with a representative associated with the given entity to perform the action includes requesting a status update regarding the item of interest from the representative associated with the given entity, and receiving information regarding the item of interest from the representative associated with the given entity in response to requesting the status update.

[0147] In some further versions of those implementations, the received information regarding the item of interest indicates availability of the item of interest at a given entity, and the notification including the results of the conversation indicates that the item of interest is available at the given entity.

[0148] In still further versions of those implementations, the method further includes, in response to determining that the received information regarding the item of interest indicates availability of the item of interest, requesting a representative associated with the given entity to reserve the item of interest at the given entity for the user, and the notification including the conversation result further indicates that the item of interest has been reserved at the given entity for the user. In still further versions of those implementations, the method further includes, in response to determining that the received information regarding the item of interest indicates availability of the item of interest, submitting additional information to obtain the item of interest on behalf of the user to a representative associated with the given entity. In still further versions of those implementations, the notification including the conversation result further requests approval from the user for submitting the additional information to be submitted to obtain the item of interest. In still further versions of those implementations, the notification including the conversation result further indicates that the additional information has been submitted to obtain the item of interest.

[0149] In still further versions of those implementations, the notification further requests the user to join the automated call. In still further versions of those implementations, the item of interest is specified in the request to initiate execution of the automated call on behalf of the user. In still further versions of those implementations, the method further includes, in response to determining that the item of interest is available at the given entity, completing execution of an additional automated call with an additional entity.

[0150] In some further versions of those implementations, the received information regarding the item of interest indicates that the item of interest is not available at the given entity, and the notification including the conversation result indicates that the item of interest is not available at the given entity. In still further versions of those implementations, the method further includes terminating execution of the automated call with the given entity in response to determining that the item of interest is not available at the given entity.

[0151] In some implementations, the action to be performed during the automated call includes requesting availability of an entity of the group for a given time period. In some versions of those implementations, engaging in a conversation with a representative associated with the given entity to perform the action includes requesting availability of the given entity for a given time period from a representative associated with the given entity and receiving information regarding availability of the given entity for a given time period from a representative associated with the given entity.

[0152] In some further versions of those implementations, the received information indicates that the given entity is available for a given period of time, and the notification including the conversation result indicates that the given entity is available for a given period of time. In still further versions of those implementations, the method further includes, in response to determining that the given entity is available for a given period of time, requesting a person associated with the given entity to keep the given entity available for the user for the given period of time, and the notification including the conversation result further indicates that the given entity will be kept available for the user for the given period of time. In still further versions of those implementations, the notification further requests the user to join the automated call. In still further versions of those implementations, the method further includes, in response to determining that the given entity is available for a given period of time, terminating execution of an additional automated call with the additional entity.

[0153] In some further versions of those implementations, the received information indicates that the given entity is not available in the given time period and the notification including the conversation result indicates that the given entity is not available in the given time period. In still further versions of those implementations, the method further includes terminating execution of the automated call with the given entity in response to determining that the given entity is not available in the given time period.

[0154] In some implementations, identifying a group of entities each capable of satisfying the request includes identifying a location of the user's computing device, determining whether the entity is within a threshold distance from the location of the user's computing device, determining whether the entity is capable of satisfying the request, and including the entity in the group of entities in response to determining that the entity is within a threshold distance from the location of the computing device and in response to determining that the entity is capable of satisfying the request. In some versions of those implementations, the method further includes receiving, after receiving the request to initiate execution of the automated call on behalf of the user, an additional request to initiate execution of another automated call on behalf of the additional user from an additional computing device of an additional user. The additional request from the additional user is the same as the request from the user. The method further includes refraining from initiating another automated call based on an outcome of the conversation with a representative associated with the given entity or an additional outcome of the additional conversation with an additional representative associated with the additional entity, generating an additional notification including the outcome of the conversation with the representative or the additional outcome of the additional conversation with the additional representative, and causing the additional notification to be rendered via an additional computing device of the user.

[0155] In some implementations, receiving a request to initiate performance of an automated call on behalf of the user includes one of receiving touch or typed input at the user's computing device via a search interface or an automated assistant interface, or receiving spoken input via one or more microphones of the user's computing device.

[0156] In some implementations, the person associated with the given entity is a human person or one of an additional automated assistants associated with the given entity.

[0157] In addition, some implementations include one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU)) of one or more computing devices, where the one or more processors are operable to execute instructions stored in associated memory, where the instructions are configured to cause performance of any of the aforementioned methods. Some implementations also include one or more non-transitory computer-readable storage media that store computer instructions executable by the one or more processors to perform any of the aforementioned methods. Some implementations also include a computer program product including instructions executable by the one or more processors to perform any of the aforementioned methods.

[0158] It should be appreciated that all combinations of the above concepts and additional concepts described in more detail herein are contemplated as being part of the subject matter disclosed herein, for example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein. [Explanation of symbols]

[0159] 110 Client Devices 111 User Input Engine 112 Rendering Engine 113 Notification Engine 115 Automated Assistants 120 Speech Recognition Engine 130A NLU engine 140A Speech Synthesis Engine 150 Auto Call Engine 151 Query Analysis Engine 152 Request Processing Engine 153 Entity Identification Engine 154 Request Execution Engine 155 Update Engine 156 Approval Engine 157 Time Delay Engine 158 User Data 190 Network 410 Computing Devices 411 URL 420 1st search result 421 Information 430 Second search result 431 Information 440 3rd search result 441 Information 480 Graphical User Interface 481 System Interface Elements 482 System Interface Elements 483 System Interface Elements 484 Text Response Interface Elements 485 Voice Response Interface Elements 510 Computing Devices 511 URL 520 1st search result 521 Information 522 Call Graphical Elements 523 Directions Graphical Elements 530 Second search result 531 Information 532 Call Graphical Elements 533 Directions Graphical Elements 570 Call Details Interface 571 First segment of selectable text 572 Second segment of selectable text 579 Notifications 580 Graphical User Interface 581 System Interface Elements 582 System Interface Elements 583 System Interface Elements 584 Text Response Interface Elements 585 Voice Response Interface Elements 586 Interface Elements 590 Notification Interface 610 Computing Devices 670 Call Details Interface 671 First segment of selectable text 672 Second segment of selectable text 679 Notifications 680 Graphical User Interface 681 System Interface Elements 682 System Interface Elements 683 System Interface Elements 684 Text Reply Interface Elements 685 Voice Response Interface Elements 686 Interface Elements 690 Notification Interface 710 Computing Devices 712 Bus Subsystem 714 Processor 716 Network Interface 720 User Interface Output Devices 722 User Interface Input Devices 724 Storage Subsystem 725 Memory Subsystem 726 File Storage Subsystem 730 RAM 732 ROM

Claims

1. 1. A method implemented by one or more processors, comprising: receiving a request from a user's computing device to initiate execution of an automated call on behalf of said user; identifying an action to be performed during the automated call based on the request; identifying a group of entities each capable of satisfying the request based on the actions to be performed during the automated call; initiating execution of the automated call with a given entity of the group of entities by an automated assistant; During execution of said automated call with said given entity, engaging in a conversation with a representative associated with the given entity to perform the action through rendering one or more corresponding instances of synthetic speech related to the request; initiating, by the automated assistant, the execution of additional automated calls with additional entities of the group of entities; generating a notification including an outcome of said conversation with said agent; causing the notification to be rendered via the computing device of the user; A method comprising:

2. determining that the given entity will fulfill the request based on the outcome of the conversation during execution of the automated call; and terminating execution of the additional automated call with the additional entity in response to the given entity determining that the request is fulfilled.

3. determining, upon initiating execution of the automated call with the given entity, that the representative associated with the given entity has placed the automated call on hold; determining that the representative associated with the given entity has resumed the automated call; 3. The method of claim 1 or 2, wherein initiating execution of the additional automated call with the additional entity is in response to the person associated with the given entity determining that they have resumed the automated call.

4. During execution of the corresponding automated call with the given entity, determining a call start time for initiating execution of the additional automated call with the additional entity based on call history data associated with the entity; 3. The method of claim 1 or 2, wherein initiating execution of the additional automated call with the additional entity is in response to determining that a current time matches the call start time.

5. the call history data relating to the additional entity is an indication that the entity is associated with an interactive voice response system and an amount of time to operate the interactive voice response system; an indication of the length of time of a previous automated call with the entity; or An indication of the length of the initial wait time between previous automated calls with said entity. The method of claim 4, comprising one or more of:

6. The method of claim 1 , wherein the action to be performed during the automated call comprises requesting a status update regarding an item of interest associated with the entity of the group.

7. engaging in said conversation with said person associated with said given entity to perform said action; requesting the status update regarding the item of interest from the personnel associated with the given entity; and receiving information regarding the item of interest from the person associated with the given entity in response to the request for a status update.

8. 8. The method of claim 7, wherein the received information regarding the item of interest indicates availability of the item of interest at the given entity, and the notification including the result of the conversation indicates that the item of interest is available at the given entity.

9. in response to determining that the received information regarding the item of interest indicates availability of the item of interest; requesting the personnel associated with the given entity to reserve the item of interest at the given entity for the user; The method of claim 8 , wherein the notification including the result of the conversation further indicates that the item of interest has been reserved for the user at the given entity.

10. in response to determining that the received information regarding the item of interest indicates availability of the item of interest; 10. The method of claim 8, further comprising submitting additional information to the person associated with the given entity to obtain the item of interest on behalf of the user.

11. 11. The method of claim 10, wherein the notification including the result of the conversation further requests approval from the user for submitting the additional information to be submitted to obtain the item of interest.

12. The method of claim 10 , wherein the notification including the result of the conversation further indicates that the additional information has been submitted to obtain the item of interest.

13. The method of claim 8 , wherein the notification further requests the user to join the automated call.

14. The method of claim 8 , wherein the item of interest is specified in the request to initiate performance of the automated call on behalf of the user.

15. 10. The method of claim 8, further comprising the step of terminating execution of the additional automated call with the additional entity in response to determining that the item of interest is available at the given entity.

16. 8. The method of claim 7, wherein the received information regarding the item of interest indicates that the item of interest is not available at the given entity, and the notification including the result of the conversation indicates that the item of interest is not available at the given entity.

17. 17. The method of claim 16, further comprising the step of terminating execution of the automated call with the given entity in response to determining that the item of interest is not available at the given entity.

18. 18. The method of claim 1, wherein the action to be performed during the automated call comprises requesting the availability of the entity of the group for a given period of time.

19. engaging in said conversation with said person associated with said given entity to perform said action; requesting the availability of the given entity for the given time period from the personnel associated with the given entity; and receiving information from the personnel associated with the given entity regarding the availability of the given entity during the given period of time.

20. 20. The method of claim 19, wherein the received information indicates that the given entity is available during the given period of time, and the notification including the result of the conversation indicates that the given entity is available during the given period of time.

21. in response to determining that the given entity is available for the given period of time, requesting the personnel associated with the given entity to keep the given entity available for the user for the given period of time; 21. The method of claim 20, wherein the notification including the result of the conversation further indicates that the given entity will be kept available for the user for the given period of time.

22. The method of claim 20 , wherein the notification further requests the user to join the automated call.

23. 21. The method of claim 20, further comprising the step of terminating execution of the additional automated call with the additional entity in response to determining that the given entity is available for the given period of time.

24. 20. The method of claim 19, wherein the received information indicates that the given entity is not available during the given period of time, and the notification including the result of the conversation indicates that the given entity is not available during the given period of time.

25. 25. The method of claim 24, further comprising the step of terminating execution of the automated call with the given entity in response to determining that the given entity is not available for the given period of time.

26. identifying said group of entities each capable of satisfying said request, determining a location of the computing device of the user; determining that the entity is within a threshold distance from the location of the user's computing device; determining that the entity is capable of satisfying the request; and in response to determining that the entity is within the threshold distance from the location of the computing device and in response to determining that the entity is capable of satisfying the request, including the entity in the group of entities.

27. 27. The method of claim 1, further comprising the step of storing, in association with the given entity, the result of the conversation with the person associated with the given entity.

28. after receiving the request to initiate execution of the automated call on behalf of the user; receiving an additional request from an additional computing device of an additional user to initiate execution of another automated call on behalf of the additional user, the additional request from the additional user being the same as the request from the first user; refraining from initiating the further automated call based on the outcome of the conversation with the representative associated with the given entity or an additional outcome of an additional conversation with an additional representative associated with the additional entity; generating an additional notification including the outcome of the conversation with the representative or an additional outcome of the additional conversation with the additional representative; causing the additional notification to be rendered via the additional computing device of the user; 28. The method of any one of claims 1 to 27, further comprising:

29. receiving the request to initiate execution of the automated call on behalf of the user, receiving touch or typed input at the user's computing device via a search interface or an automated assistant interface; or 29. The method of claim 1, comprising one of the steps of receiving spoken input via one or more microphones of the computing device of the user.

30. 30. The method of claim 1, wherein the person associated with the given entity is one of a human person or an additional automated assistant associated with the given entity.

31. 1. A method implemented by one or more processors, comprising: Identifying an item of interest; identifying a group of entities related to the item of interest; initiating execution of a corresponding automated call with each of the entities of the group by the automated assistant to render one or more corresponding instances of a synthetic utterance requesting a corresponding status update regarding the item of interest; during execution of said corresponding automated call with a given entity of said group of entities; requesting the corresponding status update regarding the item of interest from a person associated with the given entity through the rendering of one or more of the corresponding instances of synthetic speech; receiving information regarding the item of interest from the personnel associated with the given entity in response to the request for the corresponding status update; determining a time delay before initiating execution of a subsequent corresponding automated call with the given entity to request further corresponding status updates regarding the item of interest based on the received information regarding the item of interest; and initiating, by the automated assistant, execution of the subsequent corresponding automated call with the given entity in response to the passage of the time delay.

32. 32. The method of claim 31, wherein the received information regarding the item of interest indicates availability of the item of interest at the given entity and an available quantity of the item of interest at the given entity.

33. determining the time delay based on the received information regarding the item of interest; determining whether the available quantity of the item of interest at the given entity meets a quantity threshold; and assigning a first period of time as the time delay in response to determining that the available quantity of the item of interest at the given entity meets the quantity threshold.

34. 34. The method of claim 33, further comprising the step of assigning a second period of time as the time delay in response to determining that the available quantity of the item of interest at the given entity does not meet the quantity threshold, the second period of time being shorter in length of time than the first period of time and, as a result, elapses before the first period of time.

35. 34. The method of claim 33, wherein a second period of time elapses during the same day that the automated call is initiated and the first period of time elapses during the next day following the same day that the automated call is initiated.

36. 36. The method of any one of claims 32 to 35, further comprising updating search results related to the given entity to indicate one or more of the availability of the item of interest at the given entity or the available quantity of the item of interest at the given entity.

37. analyzing the updated search results related to the given entity and previously updated search results related to the given entity; Based on an analysis of the updated search results and the previously updated search results, predicting future availability of the item of interest at the given entity at a future time; or 37. The method of claim 36, further comprising forecasting a future available quantity of the item of interest at the given entity at the future time.

38. During execution of the corresponding automated call with the given entity, verifying whether the given entity is associated with the item of interest; 38. The method of any one of claims 31 to 37, wherein determining the time delay is responsive to determining that the given entity is associated with the item of interest based on the verification.

39. in response to determining that the given entity is not associated with the item of interest based on the verification, refraining from determining the time delay; and removing the given entity from the group of entities related to the item of interest, wherein removing the given entity from the group of entities related to the item of interest prevents the initiation of any subsequent automated calls with the given entity regarding the item of interest.

40. 40. The method of any one of claims 31 to 39, wherein the person associated with the given entity is one of a human person or an additional automated assistant associated with the given entity.

41. initiating execution of the corresponding automated calls with each of the entities of the group to request the corresponding status updates regarding the item of interest; initiating execution of the corresponding automated call with the given entity; and simultaneously commencing execution of at least additional corresponding automated calls with additional entities of said group of entities.

42. during execution of said additional corresponding automated calls with said additional entities; requesting the corresponding status update regarding the item of interest from an additional person associated with the additional entity through the rendering of the one or more of the corresponding instances of synthetic speech; receiving additional information regarding the item of interest from the additional personnel associated with the additional entity in response to the request for the corresponding status update; determining, based on the received additional information regarding the item of interest, an additional time delay before initiating execution of a subsequent additional corresponding automated call with the additional entity to request the further corresponding status update regarding the item of interest; A first period is assigned to the time delay; a separate second period is allocated to the additional time delay; and and initiating, by the automated assistant, execution of the subsequent additional corresponding automated call with the additional entity in response to the passage of the additional time delay.

43. identifying the group of entities associated with the item of interest, 43. A method according to any one of claims 31 to 42, comprising accessing one or more databases to identify the entities stored in association with the item of interest as the group of entities.

44. 44. The method of claim 43, wherein each of the entities in the identified group is located in a particular geographic region.

45. requesting the status update regarding the item of interest from the personnel associated with the given entity; generating synthetic speech audio data capturing one or more of the corresponding instances of synthetic speech requesting the corresponding status update regarding the item of interest; 45. The method of claim 31, wherein rendering one or more of the corresponding instances of synthetic speech comprises causing the synthetic speech to be rendered on an additional computing device of the person associated with the given entity.

46. At least one processor; and at least one memory storing instructions that, when executed, cause the at least one processor to perform the method of any one of claims 1 to 45.

Citation Information

Patent Citations

  • Vehicle use communication proxy device

    JP2003032329A

  • Escalation to a human operator

    JP2019522914A

  • System and method for making a reservation associated with a calendar appointment

    US20100094668A1