Selectively performing automated phone calls to reduce latency and / or duration of assistant interactions

Abstract

Implementations involve using an assistant to initiate automated telephone calls with entities. Some implementations identify items of interest, identify a group of entities associated with those items, and initiate a call with the entities. During a given call with a given entity, the assistant may request a status update about the item and determine a time delay before initiating another call with the given entity to request further status updates about the item based on information received in response to that request. Other implementations receive a request to perform an action on behalf of a user, identify a group of entities that can perform that action, and initiate a given call with a given entity. During the given call, the assistant may initiate additional calls with additional entities and generate notifications for the user based on the results of the given call and / or the additional calls.

Description

Technical Field Background Technology

[0001] Users can interact with automated assistants via various computing devices, such as smartphones, tablets, wearable devices, automotive systems, and standalone personal assistant devices. The automated assistant receives input from the user (e.g., verbal, tactile, and / or typing) and responds with responsive outputs (e.g., visual and / or audible).

[0002] Users can interact with an automation assistant to perform actions on their behalf. As an example, the automation assistant can make a phone call on behalf of a user to perform a given action and can engage in a conversation with other users to perform that action. For instance, a user can provide input requesting the automation assistant to inquire about a project or service on their behalf by phone. The automation assistant can initiate an automated phone call to a specific entity associated with the project or service and can provide information to additional users associated with that specific entity to make the inquiry. The automation assistant can then notify the user of the result of the automated phone call and / or can update search results associated with that specific entity based on that result.

[0003] However, for some actions performed by automated assistants on behalf of users, a specific entity may not be able to fully fulfill the user's request, resulting in wasted computing and / or network resources used in performing the action. As an example, suppose an automated assistant makes an inquiry on behalf of a user via telephone as described above, and further suppose that an additional user associated with a specific entity indicates that the item or service is unavailable. Some automated assistants can determine that the item or service is unavailable and simply provide the user with a notification reflecting this unavailability. Thus, computing and / or network resources used in performing the automated call are wasted. Furthermore, the user can manually make other phone calls to other entities and / or have the automated assistant make other automated phone calls to other entities to attempt to fulfill the request, further increasing the use of computing and / or network resources. Moreover, other users may make manual and / or automated calls to the specific entity for the same or similar requests that the specific entity cannot fulfill, further increasing the use of computing and / or network resources. Summary of the Invention

[0004] Implementation methods involve using an automation assistant to initiate and execute automated telephone calls. In some implementations, automated telephone calls can be initiated and executed in response to a request to initiate and execute an automated telephone call. The automation assistant can identify a group of entities associated with the request and can initiate a corresponding automated telephone call with one or more entities in that group to perform an action associated with the request. Furthermore, the automation assistant can perform the action associated with the request by rendering an instance of synthesized speech associated with the request. At least some instances of synthesized speech can be generated based on the request and based on processing responses from representatives of the entities participating in the telephone call during the automated telephone call.

[0005] In some implementations, the request to initiate and execute an automated phone call may be based on analyzing the query activity of multiple users. The automation assistant may identify terms corresponding to an item or service of interest that are included in a threshold number of queries (and optionally across a threshold duration) based on the analyzed query activity. In response to the identification that terms corresponding to an item or service of interest are included in a threshold number of queries (and optionally across a threshold duration), the automation assistant may identify groups of entities associated with the item or service of interest in one or more databases. For example, suppose there is a surge in queries for toilet paper. In this example, the request could be for a status update of toilet paper at each entity in the group, and each entity could be identified based on entities stored in association with the toilet paper item. In some versions of those implementations, entities included in the group may be limited to a specific geographic area, from or near that specific geographic area (e.g., within a 10-mile, 50-mile, or other distance threshold), and the analysis of query activity may be limited to the query origin and / or only to one or more entities of a specific type stored in association with the item of interest (e.g., pharmacies, supermarkets, convenience stores, grocery stores, etc.). Continuing the example above, further suppose the recent surge in queries originated in Jefferson County, Kentucky. In this example, entities included in the group can be limited to those physically located in Jefferson County, Kentucky (or within a threshold distance) and stored in association with the toilet paper item. The automation assistant can also intelligently identify entities of corresponding meaningful types associated with items of interest. Continuing the example above, the automation assistant can identify that toilet paper is primarily sold at entities such as supermarkets, convenience stores, grocery stores, etc., and relevant stores in Jefferson County, Kentucky will be included in the group of entities.

[0006] In some versions of those implementations, instances of an automated assistant can be used to call each entity in the group in parallel. During each automated call, an action associated with the request can be performed by rendering an instance of synthesized speech (e.g., requesting a status update), and information can be received from a corresponding representative of the entity in response to the synthesized speech. Furthermore, the automated assistant can determine a time delay before initiating a subsequent additional corresponding automated call with a given entity to perform the action again. Additionally, in response to determining that the time delay has elapsed, the automated assistant can initiate a subsequent additional corresponding automated call with the given entity to perform the action again.

[0007] For example, suppose the request is for an update on the status of toilet paper at each entity in a group, and suppose the representative associated with a given entity indicates that the given entity has enough toilet paper to last for three days. In this example, the automated assistant could assign a 72-hour time period as a time delay based on the information received from the representative. Therefore, the automated assistant will not initiate a subsequent corresponding automated phone call with the given entity until the 72-hour time period has elapsed. Conversely, suppose the representative associated with a given entity indicates that the given entity may not have enough toilet paper to last for the remainder of the day. In this example, the automated assistant could assign a four-hour time period as a time delay based on the information received from the representative. Therefore, the automated assistant will initiate a subsequent corresponding automated phone call with the given entity later on that same day. In these and other ways, the corresponding time delay can be dynamically determined for each entity based on the information received during automated phone calls with the entity. Dynamically determining the time delay in this way balances the technical benefits achieved by maintaining up-to-date information from the entities with the resources used in initiating and executing automated phone calls. In other words, dynamically determining the time delay can prevent another automated phone call to an entity from being executed too quickly, which would lead to unnecessary use of computing and / or network resources—while ensuring that the information from the entity is up-to-date.

[0008] In other implementations, the request to initiate and execute an automated phone call can be based on user input from a user of a computing device associated with an automation assistant. The automation assistant can identify items or services of interest associated with the request based on processing the user input. In response to identifying an item or service of interest in the user input, the automation assistant can identify groups of entities associated with the item or service of interest in one or more databases. For example, suppose a user of the computing device provides the user input "Find me a roofer." In this example, the request could be a request for roofer availability for each entity in the group, and each entity could be identified based on entities stored in association with roofing services. In some versions of those implementations, entities included in the group can be limited to entities within a threshold distance of the user's computing device location. Continuing the example above, entities included in the group could be limited to entities stored in association with roofing services within ten miles of the user's computing device location. If no entity is included in the group, the threshold distance can be increased until one or more entities are included in the group.

[0009] In some versions of those implementations, instances of an automated assistant can be used to call each entity in a group of entities in an interleaved manner. The automated assistant can initiate a first automated phone call with the first entity in the group. Actions associated with requests can be performed by rendering instances of synthesized speech (e.g., requesting status updates, requesting service availability, and / or other actions), and information can be received from a representative associated with the first entity in the group in response to the synthesized speech. Furthermore, the automated assistant can determine the call initiation time associated with each of the other entities in the group. As described in more detail herein (e.g., regarding...) Figure 1 The call initiation time can be a set duration after each automated call is initiated, determined based on historical call data associated with each entity in the group, as well as on the progress of the automated call and / or other data related to the automated call. The automated assistant can continue to initiate automated call calls with entities in the group until the request is fulfilled.

[0010] Furthermore, the automated assistant can generate one or more notifications based on the results of one or more automated phone calls, enabling the rendering of one or more notifications for each automated phone call initiated and executed by the automated assistant. Additionally, the automated assistant can cause the user's computing device to audibly and / or visually render the notifications at the user's computing device. In some further versions of those implementations, the notification may include a prompt requesting user authorization from the computing device to provide user information to a representative associated with a given entity. As described herein, in various implementations, the call initiation time utilized in interleaving automated calls may be determined in consideration of reducing the latency of notification delivery and / or reducing the overall duration of user interaction with the automated assistant. In many of those implementations, the call initiation time is further determined in consideration of mitigating unnecessary calls with one or more later-called entities and / or unnecessarily prolonged call durations (e.g., where an earlier-called entity could fulfill an action). Determining the call initiation time in this manner balances the technical benefits achieved by reducing latency and / or interaction duration with the resources utilized in executing automated phone calls.

[0011] In various implementations, results associated with entities within a group can be updated based on information obtained during automated telephone calls. For example, suppose the request is for a status update on a toilet paper item originating from an information retrieval system derived from a search interface. Further suppose that first information received from a first representative associated with a first entity indicates that the first entity has enough toilet paper to last three days, and second information received from a second representative associated with a second entity indicates that the second entity may not have enough toilet paper to last the remainder of the day. In this example, results associated with the first and second entities can be updated to include corresponding indications of the information received during the automated telephone call. In some versions of those implementations, the updated results may further include a time indication of when the information was received. Providing users with indications of entity information, and optionally time indications of the information, in these and other ways can at least selectively prevent users from manually calling entities to seek such information and / or enable automated calls to entities to seek such information and / or mitigate the risk of users traveling to and from locations associated with entities. At least generally and across large user groups, this can reduce the overall number of telephone calls to entities, thereby saving network and / or computing resources that would otherwise be utilized in such calls.

[0012] The above description is provided only as an overview of some of the embodiments disclosed herein. Those embodiments and other embodiments are described in additional detail herein. Attached Figure Description

[0013] Figure 1 A block diagram depicts an example environment that illustrates various aspects of this disclosure and in which the embodiments disclosed herein can be implemented.

[0014] Figure 2 The flowchart illustrates an example method, according to various implementations, for determining the time delay of a subsequent automated telephone call with a given entity based on a status update of an item of interest received during an automated telephone call with that entity.

[0015] Figure 3 The flowchart illustrates an example method, according to various embodiments, for determining whether to initiate an additional automated telephone call with an additional entity based on information received during an automated telephone call with a given entity.

[0016] Figure 4A , 4B 4C and 4D depict non-limiting examples of determining the time delay of subsequent automated telephone calls with various entities according to various implementations and updating the results associated with each of the various entities based on the results of the automated telephone calls.

[0017] Figure 5A , 5B 5C and 5D depict non-limiting examples of a representative user initiating automated telephone calls with various entities according to various implementations, and rendering notifications associated with each of the various entities based on the results of the automated telephone calls.

[0018] Figure 6A , 6B 6C also describes non-limiting examples of a representative user initiating automated telephone calls with various entities according to various implementations, and such that a notification associated with each of the various entities is rendered based on the results of the automated telephone calls.

[0019] Figure 7 Example architectures of computing devices according to various implementation methods are depicted. Detailed Implementation

[0020] Figure 1 A block diagram illustrating an example environment showcasing various aspects of this disclosure is shown. Client device 110 is illustrated in... Figure 1 In various embodiments, it includes a user input engine 111, a rendering engine 112, a notification engine 113, a speech recognition engine 120A1, a natural language understanding (“NLU”) engine 130A1, and a speech synthesis engine 140A1.

[0021] User input engine 111 can detect user input at client device 110. User input detected at client device 110 may include verbal input detected via the microphone of client device 110 and / or additional verbal input transmitted from an additional user's additional client device to client device 110 (e.g., during an automated telephone call), touch input detected via the user interface of client device 110 and / or typed input detected via the user interface of client device 110. Additional users may be, for example, human representatives associated with an entity, automated assistants associated with an entity, interactive voice response systems (“IVR” systems) associated with an entity, and / or combinations thereof. The automated telephone calls described herein can be performed using various voice communication protocols, including, for example, Voice over Internet Protocol (VoIP), Public Switched Telephone Network (PSTN), and / or other telephone communication protocols. Furthermore, the automated telephone calls described herein can be initiated and performed using automated assistant 115, which can be performed locally at client device 110 (e.g., a local assistant), remotely at one or more servers (e.g., a cloud-based assistant), and / or combinations thereof (e.g., such as...). Figure 1 (Indicated by the dashed line in the middle).

[0022] In various implementations, the automation assistant 115 can use the automatic request system 180 to initiate and execute automated telephone calls. In implementations where the automation assistant 115 executes locally at the client device 110, the automatic request system 180 can also execute locally on the client device 110, allowing the automatic telephone call to be initiated and executed using only the resources of the client device 110. In implementations where the automation assistant 115 executes remotely at one or more servers, the automatic request system 180 can also execute remotely at one or more servers, allowing the automatic telephone call to be initiated and executed using only the resources of the one or more servers. In implementations where the automation assistant 115 executes remotely at one or more servers, the automatic request system 180 can also execute remotely at one or more servers. Figure 1 In the distributed implementation of the automation assistant 115 shown, the automated request system 180 can be remotely executed from one or more servers communicating with the client device 110 via a network 190. The network 190 may include, for example, Wi-Fi, Bluetooth, near-field communication, a local area network, a wide area network, other networks, and / or any combination thereof.

[0023] In various embodiments, the automatic request system 180 includes a speech recognition engine 120A2, an NLU engine 130A2, a speech synthesis engine 140A2, and an automatic telephone calling engine 150. Furthermore, in various embodiments, the automatic telephone calling engine 150 includes a query analysis engine 151, a request processing engine 152, an entity recognition engine 153, a request execution engine 154, an update engine 155, an authorization engine 156, and a time delay engine 157.

[0024] In some implementations, the automation assistant 115 may initiate and execute an automated telephone call using the automated request system 180 in response to detecting user input from a given user on client device 110 via user input engine 111, wherein the user input includes a request to initiate an automated telephone call. User input may include typed or touch input detected via the touchscreen or keyboard of client device 110, and / or verbal input detected via the microphone of client device 110. In implementations where user input engine 111 detects typed or touch input, client device 110 may transmit the indication of the detected user input to the request processing engine 152 of automated telephone call engine 150 via network 190. For example, user input engine 111 may detect typed or touch input via the search interface, automation assistant interface, and / or other interfaces of client device 110, determine that the detected user input relates to initiating and executing an automated telephone call, and transmit the detected user input to the request processing engine 152 of automated telephone call engine 150 for further processing of the detected user input.

[0025] In an implementation where the user input engine 111 detects a given user's verbal input via the microphone of the client device 110, the speech recognition engine 120A1 of the client device 110 can use the speech recognition model 120A to process the audio data generated by the microphone capturing the verbal input to generate recognized text corresponding to the verbal input. Furthermore, the NLU engine 130A1 of the client device 110 can use the NLU model 130A to process the recognized text generated by the speech recognition engine 120A1 to determine the intent included in the verbal input. For example, if the client device 110 detects the verbal input "Find toilet paper near me" from a given user, the client device 110 can use the speech recognition model 120A to process the audio data capturing the verbal input to generate recognized text corresponding to the verbal input, and can use the NLU model 130A to process the recognized text to determine at least the intent to find toilet paper near the given user's current location as an action included in a request to initiate and execute an automated telephone call. In some versions of those implementations, the client device 110 may transmit the identified text corresponding to the verbal input and / or the intent determined based on the verbal input to the request processing engine 152 of the automated telephone call engine 150 for further processing of the identified text and / or intent.

[0026] In other embodiments, when the user input engine 111 detects verbal input from a given user via the microphone of the client device 110, the automation assistant 115 can cause the client device 110 to transmit audio data capturing the verbal input to the automatic request system 180. The speech recognition engine 120A2 and / or NLU engine 130A2 of the assisted call system 180 can process the audio data capturing the verbal input in a similar manner to that described above with respect to the speech recognition engine 120A1 and / or NLU engine 130A1 of the client device 110. Furthermore, the verbal input and / or the intent determined based on the verbal input are sent to the request processing engine 152 of the automatic call engine 150 for further processing to identify the text and / or intent. In some additional and / or alternative embodiments, the speech recognition engine 120A1 and / or NLU engine 130A1 of the client device 110 can be used in a distributed manner in conjunction with the speech recognition engine 120A2 and / or NLU engine 130A2 of the automatic request system 180. In addition, the speech recognition model 120A and / or the NLU model 130A may be stored locally on the client device 110, remotely at the automatic request system 180, and / or remotely at another server that communicates with the client device 110 and / or the automatic request system 180 via the network 190.

[0027] In various implementations, the speech recognition model 120A is an end-to-end speech recognition model, enabling speech recognition engines 120A1 and / or 120A2 to directly use the model to generate recognized text corresponding to spoken input. For example, the speech recognition model 120A can be an end-to-end model for generating recognized text on a character-by-character (or other token-by-token) basis. A non-limiting example of such an end-to-end model for generating recognized text on a character-by-character basis is a recurrent neural network converter (RNN-T) model. An RNN-T model is a form of sequence-to-sequence model that does not employ an attention mechanism. Unlike most sequence-to-sequence models (which typically require processing the entire input sequence (e.g., audio data waveforms or Mel-frequency cepstral coefficients (MFCCs) or other representations) to generate a predicted output), an RNN-T model can be used to process input samples and stream output symbols (e.g., characters of the alphabet). Furthermore, for example, when the speech recognition model is not an end-to-end speech recognition model, speech recognition engines 120A1 and / or 120A2 can instead generate predicted phonemes (and / or other representations). For example, using such a model, the predicted phonemes (and / or other representations) are then used by speech recognition engines 120A1 and / or 120A2 to determine the recognized text that matches the predicted phonemes. In doing so, speech recognition engines 120A1 and / or 120A2 may optionally employ decoding graphs, dictionaries, and / or other resources.

[0028] In some additional and / or alternative implementations, the automation assistant 115 may use the automated request system 180 to initiate and execute automated phone calls without detecting any user input via the user interface input engine 111. In some implementations, the query analysis engine 151 may analyze query activities stored in the query activity database 151A and may transmit a request to initiate and execute an automated phone call in response to determining that one or more conditions are met. The query activities analyzed by the query analysis engine 151 may include queries submitted to a search engine by multiple users using their respective client devices (e.g., via a search interface or automation assistant interface). In some versions of those implementations, one or more conditions may include determining whether multiple users have submitted a threshold number of queries for a given item or service. For example, if the query analysis engine 151 determines, based on analysis of query activities used by multiple users, that multiple users have submitted 30,000 queries for “toilet paper” or its variants (e.g., “TP”, “bathroom tissues”, etc.), exceeding a threshold number of queries for a given item (e.g., toilet paper), the automation assistant 115 may transmit a request to initiate and execute an automated phone call to the request processing engine 152. In some versions of those implementations, query activity may be limited to queries submitted by users within a specific geographic region. For example, query activity may be limited to queries submitted by users within a specific postal code, county, city, town, state, country, and / or other geographic region.

[0029] In some further versions of those implementations, one or more conditions may include determining whether a threshold number of queries for a given item or service have been submitted by multiple users within a given time frame. Continuing the example above, if query analysis engine 151 determines that 30,000 queries for “toilet paper” or its variants have been submitted within a two-week period, the automation assistant may not transmit the request to request processing engine 152. However, if query analysis engine 151 determines that 30,000 queries for “toilet paper” have been submitted within a two-day period, the automation assistant may not transmit the request to request processing engine 152. In this way, query analysis engine 151 can detect whether queries for a given item have increased within a specific time period. Furthermore, in various implementations, query analysis engine 151 may analyze only a subset of queries submitted in query activity database 151A, such as queries attempting to find or retrieve items of interest rather than queries that only mention “toilet paper” or its variants. In this way, query analysis engine 151 can identify a surge in submitted queries from a specific geographic region.

[0030] Request processing engine 152 can process received requests to determine actions associated with the request. Actions associated with a request may include: requesting a status update regarding an item of interest at a given entity, requesting the availability of a given entity associated with a specific service over a period of time, and / or other actions that may be performed by the automated assistant 115 during an automated telephone call. In embodiments where the request is based on user input detected at client device 110 via user input engine 111, request processing engine 152 may determine actions associated with the request based on indications of typed and / or touch input and / or identified text generated from verbal input and / or determined intent. For example, if the user input detected via user input engine 111 is a verbal input of “find an electrician who can work at my house tomorrow at 12:00 noon,” request processing engine 152 may determine actions including calling an electrician to request the electrician’s availability at 12:00 noon tomorrow, based on the generated identified text and / or determined intent. In an implementation where the request is based on analyzing query activity without detecting any user input at the client device 110 via user input engine 111, the request processing engine can determine the action associated with the request based on a given item or service included in the query. For example, if query analysis engine 151 determines that 5,000 queries including the term "hand sanitizer" were submitted by users in Jefferson County, Kentucky, within an hourly timeframe, request processing engine 152 can determine that the action includes calling entities associated with hand sanitizer stored in Jefferson County, Kentucky, to request status updates on the availability of hand sanitizer at each entity.

[0031] Entity recognition engine 153 can identify groups of entities to participate in during an automated telephone call. Entities can be, for example, personal entities, business entities, location entities, and / or other entities. In some embodiments, entity recognition engine 153 can also determine a specific type of entity to be included in the group. For example, one type of personal entity can be a friend entity, a family member entity, a colleague entity, and / or other specific types of personal entities. Furthermore, one type of business entity can be a service provider entity, an airline entity, a hotel entity, a retail store entity, and / or other specific types of business entities. Additionally, one type of location entity can be a school entity, a museum entity, a library entity, and / or other specific types of location entities. In some embodiments, entity recognition engine 153 can also determine a specific entity for the identified entities. For example, the specific entity for a personal entity can be a person's name (e.g., John Doe, example plumber 1, etc.), the specific entity for a business entity can be the name of a business (e.g., example supermarket, imaginary toy store, example plumbing company, etc.), and the specific entity for a location entity can be the name of a location (e.g., imaginary library, example state park, etc.). Although the entities described in this article can be defined at various levels of granularity, for the sake of simplicity, they are collectively referred to as "entities" in this article.

[0032] Each entity identified by the entity recognition engine 153 can be stored in the entity database 153A. Furthermore, each entity can be stored in the entity database 153A in association with its associated items, associated services, associated locations, associated web pages, and / or other data associated with each entity. In some implementations, the entity database 153A can be populated with entities and associated data using a web crawler. For example, if a given entity is a hypothetical toy store, the web crawler can crawl web pages associated with the hypothetical toy store to identify each location of the hypothetical toy store, the available stock of specific items at the hypothetical toy store, and the opening hours of the hypothetical toy store, and store this data in association with the hypothetical toy store in the entity database 153A. As another example, if a given entity is a hypothetical café, the web crawler can crawl web pages associated with the hypothetical café to identify each location of the hypothetical café, the menu items available at the hypothetical café, and the opening hours of the hypothetical café, and store this data in association with the hypothetical café in the entity database 153A. In this way, entity database 153A includes a mapping between each entity stored therein and the data associated with it. In some additional and / or alternative implementations, an entity or a user accessing an entity can provide instructions on items associated with that entity.

[0033] In an implementation where the request is based on received user input detected at client device 110 via user input engine 111, entity recognition engine 153 can identify a group of entities, each capable of fulfilling the request, in response to receiving user input. Entity recognition engine 153 can initially identify each entity in entity database 153A stored in association with a given item or service included in the request as an initial group of entities capable of fulfilling the request. Furthermore, entity recognition engine 153 can identify the location of client device 110 based on location data generated by one or more sensors (e.g., GPS) of client device 110, and can determine which entities in the initial group are within a threshold distance of the location of client device 110. Additionally, entity recognition engine 153 can include entities within the threshold distance of the location of client device 110 and capable of fulfilling the request in the group of entities. For example, if the user input detected at client device 110 is a verbal input of "find tissues", then entity recognition engine 153 can identify the current location of client device 110, identify entities stored in entity database 153A associated with tissues, determine which of the identified entities are within a ten-mile radius of the current location of client device 110, and include each entity associated with tissues and within a ten-mile radius of the current location of client device 110 in a group of entities.

[0034] In some versions of those implementations, the initially identified entities may be limited to those stored in association with a location within a threshold distance of the client device 110's location. Continuing the example above, only entities stored in association with tissues and within a ten-mile radius of the client device 110 may be considered for inclusion in the group. In some versions of those implementations, the entity recognition engine 153 may include only a threshold number of entities in the group. For example, if the number of entities in the group exceeds the threshold number of entities, only entities that meet one or more criteria may be included in the group. One or more criteria may include, for example, location criteria, time criteria, rating criteria, user preference criteria, and / or other criteria. Continuing the example above, if there are ten entities associated with tissues and within a ten-mile radius of the client device 110's current location, then only the five closest entities, the five highest-rated entities, and / or the five currently open entities as indicated by business hours, or the five entities that the user of the client device 110 has previously visited or frequently visits. In some additional and / or alternative versions of those implementations, if no entity in an entity's group satisfies the request, the entity recognition engine 153 may identify additional groups of entities. In response to the determination that no entity in the group can satisfy the request, the threshold distance can be modified. Continuing the example above, if none of the five entities in the group have available tissues, five additional entities within a ten-mile radius can be included in an additional group, or if there are no five additional entities within a ten-mile radius, the ten-mile radius can be expanded to fifteen miles or more. In this way, the entities identified in response to user input are personalized for the user providing the input.

[0035] In implementations where the request is based on query activity analyzed via query analysis engine 151, entity identification engine 153 can identify groups of entities that can satisfy the request in response to receiving a request. Entity identification engine 153 can identify each entity stored in entity database 153A as a group of entities that can satisfy the request. For example, if query analysis engine 151 determines that there has been a recent surge in submitted queries including the term "hot Christmas toys for 2020," then each entity stored as a relation to "hot Christmas toys for 2020" can be included in the group of entities. In some versions of those implementations, entity identification engine 153 can identify a specific geographic region from which queries are submitted. Furthermore, entity identification engine 153 can include entities within a specific geographic region that can satisfy the request in the group of entities. Continuing the example above, if query analysis engine 151 determines that there has been a recent surge in queries for "hot Christmas toys for 2020" originating in San Francisco County, California, then entity identification engine 153 can restrict the entities in the group to entities stored only as a relation to locations within San Francisco, California. In this way, the entities identified in response to the recent surge in submitted queries are personalized for the specific geographic region from which the query originated.

[0036] The request execution engine 154 can initiate and execute automated telephone calls corresponding to entities in a group identified by the entity recognition engine 153. More specifically, each automated telephone call described herein can be initiated and executed using various voice communication protocols, including, for example, Voice over Internet Protocol (VoIP), Public Switched Telephone Network (PSTN), and / or other telephone communication protocols. Furthermore, each automated telephone call described herein can be between an automated assistant 115 and a representative associated with a given entity in the group. The representative can be, for example, a human representative associated with the given entity, an additional automated assistant associated with the given entity, an IVR system associated with the given entity, and / or any combination thereof.

[0037] More specifically, the request execution engine 154 enables the automation assistant 115 to use synthesized speech to engage in a conversation with a representative associated with a given entity during an automated telephone call to perform an action included in the request. The request execution engine 154 can provide text and / or phonemes related to the execution of the request to the speech synthesis engine 140A1 of the client device 110 and / or the speech synthesis engine 140A2 of the auxiliary call system 180 to generate synthesized speech audio data. The synthesized speech audio data can be transmitted to an additional client device of the representative associated with the given entity for audible rendering. The speech synthesis engines 140A1 and / or 140A2 can use the speech synthesis model 140A to generate synthesized speech audio data including synthesized speech corresponding to the text and / or phonemes related to the execution of the request. For example, suppose the request is for a status update regarding “hot Christmas toys for 2020”. In this example, speech synthesis engines 140A1 and / or 140A2 can determine the phoneme sequence corresponding to the status update "Do you have any popular Christmas toys for sale in 2020?", and can process this phoneme sequence using speech synthesis model 140A to generate synthesized speech audio data capturing the request for the status update regarding popular Christmas toys. The synthesized speech audio data can be, for example, in the form of an audio waveform. In determining the phoneme sequence corresponding at least to the phonemes relevant to the execution of the request, speech synthesis engines 140A1 and / or 140A2 can access a token-to-phoneme mapping stored locally at client device 110 or remotely at a server via network 190.

[0038] In various implementations, before executing a request during a corresponding automated call with a given entity, the automated assistant 115 may obtain consent from a representative associated with the given entity to participate in a conversation with the automated assistant 115. For example, the automated assistant 115 may obtain consent after initiating an automated call by identifying itself as the automated assistant 115 and prompting the representative associated with the given entity to consent to participate in a conversation with the automated assistant. If the automated assistant 115 obtains consent from the representative associated with the given entity, the automated assistant 115 may use the automated request system 180 to perform the actions included in the request. Furthermore, once consent has been obtained, the automated assistant may not prompt the representative in any subsequent automated call. However, if the automated assistant 115 does not obtain consent from the representative, the automated assistant 115 may terminate the automated call.

[0039] The request execution engine 154 can participate in a dialogue with a representative associated with a given entity by rendering a corresponding instance of synthesized speech associated with the request. Rendering the corresponding instance of synthesized speech includes rendering the synthesized speech at an additional computing device of the representative associated with the given entity. The corresponding instance of synthesized speech can be generated using the speech synthesis model 140A as described above, and can be generated based on the text and / or phonemes associated with the request. Furthermore, the request execution engine 154 can utilize the speech recognition engines 120A1 and / or 120A2 and the NLU engines 130A1 and / or 130A2 as described above to process any audio data corresponding to information received from the representative associated with the given entity during the given entity interaction. The request execution engine 154 can enable the automation assistant 115 to manipulate the dialogue based on the received information during the dialogue with the representative. For example, the automation assistant 115 can initially verify that the given entity can fulfill the request. If the given entity verifies that it can fulfill the request, the automation assistant 115 can further inquire about fulfilling the request. However, if the given entity indicates that it cannot fulfill the request, the automation assistant 115 can terminate the call. This allows the automation assistant 115 to respond intelligently based on the information received during automated phone calls.

[0040] For example, suppose the request is for a status update regarding the availability of latex gloves. In this example, synthesized speech audio data asking "Do you have any latex gloves available?" can be rendered at the representative's attached client device. The representative can provide information such as "yes" or "no," and the audio data capturing the received information can be processed. Based on the processing of the received audio data, the automation assistant 115 can manipulate the dialogue. For example, if the representative indicates that latex gloves are available at a given entity, the request execution engine 154 can further request the quantity of latex gloves available, whether the given entity expects the latex gloves to be available for the remainder of the day, and so on. However, if the representative indicates that latex gloves are available at a given entity, the request execution engine 154 can cause the automation assistant 115 to terminate the dialogue. This is described in more detail herein (e.g., see references). Figures 4A to 4C (5B to 5D and 6B to 6C) participate in dialogue with representatives associated with a given entity.

[0041] In an implementation where the request is for an update on the corresponding status of a given item, update engine 155 can cause the results associated with the given entity to be updated based on information received during a corresponding automated telephone call with a representative associated with the given entity. It is noteworthy that the results associated with the given entity can be updated regardless of whether the request is based on received user input detected by user input engine 111 or on query activity analyzed by query analysis engine 151. Updated results associated with a given entity may include, for example, an indication of whether a given item is available at the given entity, an indication of the number of items available at the given entity, and / or other information received during a corresponding automated telephone call with a representative associated with the given entity. It is noteworthy that the results (also referred to herein as “search results”) may include any results that can be rendered to a user through various interfaces. For example, results may be presented via an information retrieval system interface, an automation assistant interface, a map or navigation application interface, a customized interface, and / or any other interface used to expose information to the user.

[0042] For example, suppose the request is for a status update regarding toilet paper. In this example, automation assistant 115 can engage in a conversation with a representative associated with a given entity to request a status update regarding toilet paper at the given entity. Further suppose that during the conversation, the representative associated with the given entity provides information indicating that the given entity has available toilet paper, and also provides information about the quantity of toilet paper available (e.g., "We have enough toilet paper for the rest of the day," "We have 100 packs of toilet paper," "We will use it up before noon," etc.). Based on this received information, the update engine can cause the results associated with the given entity to be updated to include the indication that the given entity has available toilet paper and the indication of the quantity of toilet paper available. Conversely, suppose the information received from the representative associated with the given entity indicates that the entity does not sell toilet paper. In this example, the results associated with the given entity can be updated to indicate that the given entity is no longer associated with toilet paper. Furthermore, update engine 155 can transmit an indication that the given entity is no longer associated with toilet paper, and entity recognition engine 153 can remove the association between toilet paper and the given entity in entity database 153A.

[0043] In some versions of those implementations, the result of an update associated with a given entity may also include a time indicator of when information was received from a representative associated with the given entity. The time indicator may include, for example, a specific time the information was received (e.g., 9:00 AM, 2:00 PM), a specific date the information was received (e.g., yesterday, today, 6 / 11 / 2020), a range of dates the information was received (e.g., last week, yesterday morning, this afternoon), and / or any combination thereof. In this document (e.g., referring to…), Figures 4A to 4D This describes in more detail how results associated with multiple entities are updated based on information received during a corresponding automated call. By updating results associated with a given entity, multiple users do not need to call the given entity to request status updates on the item, thus saving computing resources on the client devices of multiple users and the client devices associated with the given entity. In other words, a single automated call and the provision of information to multiple users from that single automated call prevents at least two or more of the multiple users from calling the given entity to seek such information. Furthermore, by updating results associated with a given entity, multiple users do not need to travel to multiple entities to search for the item, thus saving the natural resources consumed by multiple users traveling to multiple entities while searching for the item.

[0044] In some versions of those implementations, update engine 155 may also analyze the results of updates associated with a given entity and the results of previous updates associated with the given entity to predict the future availability of an item at the given entity or the future quantity of an item available at the given entity for a future time. Update engine 155 may use, for example, a machine learning prediction model or rules generated by a machine learning prediction model to predict the future availability of an item at the given entity or the future quantity of an item available at the given entity for a future time. For example, update engine 155 may determine, based on the analysis of the results of updates associated with a given entity, that the given entity has had no toilets available for the past three Thursdays and Fridays, but has had toilet paper available for each of the past three Saturdays. Based on this determination, update engine 155 may predict that the given entity will have no toilet paper available this coming Thursday or Friday, but will have toilet paper available the following Saturday. Therefore, at the beginning of the week, update engine 155 may update the results to include the indication that the given entity is expected to have no toilet paper available on Thursday or Friday, but will have toilet paper available on Saturday.

[0045] In embodiments where the request is based on received user input detected at client device 110 via user input engine 111, request execution engine 154 may initiate automated telephone calls with entities in the group to perform the actions included in the request. Request execution engine 154 may determine the call initiation time for initiating additional automated telephone calls with additional entities in the group. Furthermore, request execution engine 154 may determine each call initiation time before or during the automated telephone calls. In some versions of those embodiments, the call initiation time may be a static threshold duration (e.g., 15 seconds, 30 seconds, 45 seconds, and / or other durations). For example, request execution engine 154 may cause automation assistant 115 to initiate a first automated telephone call with a first entity in the group, a second automated telephone call with a second entity in the group 30 seconds after initiating the first automated telephone call with the first entity, a third automated telephone call with a third entity in the group 30 seconds after initiating the second automated telephone call with the second entity, and so on for each entity in the group.

[0046] In other versions of those implementations, the request execution engine 154 may determine the timing of each call initiation based on historical call data associated with corresponding entities in the group (e.g., stored in a historical call database 154A) and / or the complexity of the request. Historical call data may include, for example, indications of the duration of the corresponding entity's association with and navigation of the IVR system, indications of the duration of previous automated phone calls with the corresponding entity, indications of the duration of the initial hold time during previous automated phone calls with the corresponding entity, and / or other call data derived from previous automated phone calls with each corresponding entity in the group. As a non-limiting example, if historical call data indicates that the first entity in the group is not associated with an IVR system or any hold time, but the second entity in the group is associated with an IVR system that takes 20 seconds to navigate, the request execution engine 154 may cause the automation assistant 115 to simultaneously initiate (e.g., simultaneously or within a threshold time amount (e.g., 500 milliseconds, 1 second, and / or other time amounts)) a first automated phone call with the first entity in the group and a second automated phone call with the second entity in the group. As another non-limiting example, if historical call data indicates that the first entity in the group is associated with an IVR system that takes 10 seconds to navigate and the second entity in the group is associated with an IVR system that takes 30 seconds to navigate, then the request execution engine 154 can cause the automation assistant 115 to initiate a first automated phone call with the first entity in the group and a second automated phone call with the second entity in the group 10 seconds later. As another non-limiting example, if the complexity of the request predicts that it will take 45 seconds to execute, and if historical call data indicates that the first entity in the group is not associated with an IVR system (or any hold time), but the second entity in the group is associated with an IVR system that takes 20 seconds to navigate, then the request execution engine 154 can cause the automation assistant 115 to initiate a second automated phone call with the second entity 25 seconds after initiating the first automated phone call with the first entity. In this example, sufficient time can be provided for the automated assistant to execute the request with the first entity (e.g., 45 seconds) and initiate a second automated phone call with the second entity, such that the representative associated with the second entity should be active for the second call following the execution of the first automated phone call, which is based on a 20-second delay for navigating the IVR and a 25-second delay for call initiation time equal to the 45-second delay for executing the request. Furthermore, this also provides the automated assistant with the opportunity to terminate the second automated phone call with the second entity if the first entity indicates that it can fulfill the request. Conversely, suppose the first entity is associated with an IVR system that takes 20 seconds to navigate, and the second entity is not associated with the IVR system or the hold time.In this example, the call initiation time for the second automated phone call with the second entity could be 65 seconds, allowing the automation assistant 20 seconds to navigate the IVR system and perform actions before initiating the second automated phone call with the second entity. The call initiation time can be determined for each entity in the group in this dynamic manner.

[0047] In various implementations, the call initiation time utilized in enabling automated call interleaving may be determined with consideration of reducing the latency of notification delivery (e.g., as described below with respect to notification engine 113) and / or reducing the overall duration of user interaction with the automation assistant. In many of those implementations, the call initiation time is further determined with consideration of mitigating unnecessary calls to one or more later-called entities and / or unnecessarily prolonged call durations (e.g., where actions could be satisfied by earlier-called entities). Determining the call initiation time in this manner balances the technical benefits achieved by reducing latency and / or the duration of interaction with resources utilized in executing automated phone calls. For example, by enabling automated phone call interleaving, entities can be selectively called until a request included in the user input is satisfied, and there is no need to initiate and execute automated phone calls with each entity in the group. Furthermore, the automation assistant connects to a representative associated with each entity at the appropriate time, so that the automation assistant does not need to wait for the first automated phone call to end to initiate a second automated phone call and remain waiting or navigating the IVR, thereby reducing the interaction duration in executing automated phone calls.

[0048] In embodiments where the request is based on received user input detected at client device 110 via user input engine 111, request execution engine 154 may continue to initiate automated telephone calls with entities in the group in any of the aforementioned ways until one of the entities in the group satisfies the request. In some versions of those embodiments, request execution engine 154 may cause automation assistant 115 to proactively terminate automated telephone calls with other entities once a representative associated with a given entity indicates that the given entity can satisfy the request. For example, suppose a representative associated with a first entity in the group provides information indicating that the first entity can satisfy the request and has initiated a second automated call with a second entity in the group. In this example, request execution engine 154 may cause automation assistant to terminate the second automated call with the second entity because the first entity can satisfy the request.

[0049] In other versions of those implementations, the request execution engine 154 may continue executing the additional automated phone call instead of actively terminating it, even if the representative of the given entity in the automated call indicates that the given entity can fulfill the request. For example, if the additional automated phone call has progressed beyond a threshold stage, the request execution engine 154 may cause the automation assistant 115 to continue the conversation with the additional entity. For example, the request is for an update on the status of hand sanitizer, and it is further assumed that the automation assistant 115 is involved in a conversation with a representative of a first entity associated with the first automated phone call and has initiated a second automated phone call with a second entity in the same group. In this example, if the representative associated with the first entity indicates that the first entity has available hand sanitizer and the automation assistant 115 is maintaining or navigating the IVR system associated with the second entity, the automation assistant 115 may terminate the second automated phone call. However, if the automation assistant is already involved in a conversation with a representative associated with the second entity, the automation assistant 115 may continue executing the second automated phone call. Furthermore, the update engine 155 may cause the search results associated with the second entity to be updated based on information received from the representative associated with the second entity, as described above.

[0050] It is worth noting that automated telephone calls initiated and executed in response to user input detected at client device 110 via user input engine 111 can be initiated and executed by an automation assistant executed locally on client device 110 or via a cloud-based automation assistant. By interleaving the initiation and execution of automated telephone calls in response to requested user input, the techniques described herein can save computational resources of client devices associated with entities. For example, by interleaving automated telephone calls to identify a given entity that can satisfy the request, the techniques described herein avoid simultaneously calling each entity in a group of entities to identify a given entity among those that can satisfy the request. Furthermore, by continuing to execute active automated telephone calls after another entity has indicated that it can satisfy the request, the techniques described herein avoid wasting computational resources consumed by active automated telephone calls compared to simply terminating the active telephone call.

[0051] In embodiments where the request is based on received user input detected at client device 110 via user input engine 111, and in response to determining that a representative associated with a given entity indicates that the given entity can fulfill the request included in the user input, request execution engine 154 may cause synthesized speech including user data (e.g., stored in a user data database) to be rendered at the client device of the representative associated with the given entity. User data may include the name information of the user of client device 110, the address information of the user of client device 110, the contact information of the user of client device 110 (e.g., telephone number, email address, and / or other contact information), the money information of the user of client device 110, and / or other information of the user of client device 110. In some versions of those embodiments, authorization engine 156 may cause a prompt (e.g., using rendering engine 112) to be rendered on the display of client device 100 to authorize the automation assistant 115 to provide user information to the representative. The prompt may also include an optional element that, when selected, allows the user of client device 100 to join an automated telephone call. If the user of client device 110 authorizes the provision of user data to a representative of a given entity, the automation assistant may cause synthesized speech audio data including the user data to be audibly rendered on the client device of the representative associated with the given entity. However, if the user of client device 110 does not authorize the provision of user data, the automation assistant 115 will prevent the synthesized speech audio data including the user data from being audibly rendered on the client device of the representative associated with the given entity. In other versions of those implementations, the user of client device 110 may have previously authorized the provision of user data, and the automation assistant 115 may cause synthesized speech audio data including the user data to be audibly rendered on the client device of the representative associated with the given entity without prompting the user for authorization.

[0052] In some versions of implementations where the user input includes a request for an update on the status of an item and a representative associated with a given entity indicates that the item is available, the automation assistant 115 may request a representative to hold the item for the user of client device 110 at a given entity, or submit monetary information to a representative to retrieve the item on behalf of the user of client device 110. The request execution engine 154 may determine whether to hold or retrieve the item based on user input (e.g., “Find and order disinfectant wipes for me,” “Find and retrieve disinfectant wipes for me,” etc.), or by prompting the user regarding whether the user of client device 110 wants to hold or retrieve the item (e.g., via rendering engine 112).

[0053] For example, suppose the request is for a status update regarding disinfectant wipes, and further suppose that a representative of a given entity indicates that the disinfectant wipes are available at the given entity. Furthermore, in response to determining that the item is available at the given entity and in response to determining that the user of client device 110 wants to obtain the item, the automation assistant 115 may cause synthetic voice audio data to be rendered on the client device of the representative associated with the given entity. This synthetic voice audio data includes the user's name, contact information, and monetary information (if authorized). As another example, suppose the request is for the availability of a pipeline service for a given time and date, and further suppose that a representative of a given entity indicates that the pipeline service is available for the given time and date. Furthermore, in response to determining that the pipeline service is available at the given time and date and in response to determining that the user of client device 110 wants to book the given time and date, the automation assistant 115 may cause synthetic voice audio data to be rendered on the client device of the representative associated with the given entity. This synthetic voice audio data includes the user's name, address, and contact information (if authorized).

[0054] In embodiments where the request is based on received user input detected at client device 110 via user input engine 111, automation assistant 115 may cause notification engine 113 to generate a notification including one or more results of an automated telephone call. The notification generated by notification engine 113 may be visually rendered via the display of client device 110 and / or audibly rendered via the speaker of client device 110 (e.g., using rendering engine 112). The results of the conversation may include, for example, an indication of whether a given entity can fulfill a request, an indication of whether an item or service has been held or booked for the user of client device 110, an indication of whether an item has been obtained for the user of client device 110, an indication that authorization is required to submit user data of the user of client device 110 to obtain an item, an indication requesting the user of client device 110 to join an automated telephone call with a given entity, and / or other results of the conversation. The notification generated by notification engine 113 may be a banner notification, pop-up notification, email notification, text message notification, voice message notification, and / or other types of notification. In this document (e.g., refer to...), Figures 5B-5D 6B and 6C describe in more detail how to generate notifications based on the results of the dialogue.

[0055] In various implementations, rendering engine 112 can render the text of the dialogue between automation assistant 115 and a representative of a given entity on the display of client device 110. Furthermore, the display may include one or more selectable elements that, when selected, allow the user of client device 110 to join a dialogue, end an automated phone call with a given entity, and / or perform other actions. In implementations where automation assistant 115 participates in multiple dialogues with multiple entities, the corresponding text of each dialogue can be rendered on the display of client device 110. In some versions of those implementations, each text may be associated with a tab in the automation assistant interface, allowing the user of client device 110 to switch between texts of multiple dialogues, since client device 110 may have limited screen space. In other implementations, text may not be displayed, and the user of client device 110 can simply receive the aforementioned notifications.

[0056] In embodiments where the request is based on the analysis of query activity via query analysis engine 151, request execution engine 154 can cause automation assistant 115 to initiate corresponding automated phone calls with a given entity in the group to perform the actions included in the request. In some embodiments, request execution engine 154 can cause multiple instances of automation assistant 115 described herein to simultaneously initiate and execute multiple corresponding automated phone calls with corresponding entities in the group. For example, a first instance of automation assistant 115 can initiate and execute a first automated phone call with a first entity in the group, a second instance of automation assistant 115 can simultaneously initiate and execute a second automated phone call with a second entity in the group, and so on for each entity in the group. In other words, instances of automation assistant 115 can initiate and execute each automated phone call in parallel. In other embodiments, corresponding automated phone calls can be initiated and executed in batches (e.g., with a subset of the group of entities) or sequentially. Notably, each instance of automation assistant 115 can be executed remotely at a server (e.g., a cloud-based automation assistant) along with automation request system 180, and client device 110 can be omitted.

[0057] Each instance of the automated assistant can participate in a corresponding dialogue with a corresponding entity in the entity to perform an action included in the request. The time delay engine 157 can determine a time delay before initiating a subsequent corresponding automated phone call with the corresponding entity based on information received during each corresponding dialogue. Furthermore, the time delay engine 157 can assign various time periods as time delays based on information received during the corresponding automated phone call, the hour of the operation associated with the corresponding entity, legal restrictions on whether and / or when an automated phone call can be performed, computational and / or network bandwidth considerations (e.g., “slow” periods for phone calls, such as early morning). Additionally, the time delay engine 157 can store the time periods assigned as time delays in association with the corresponding entity in a time delay database 157A. In an implementation where the request is for a status update regarding an item, the time period assigned to the time delay can be an indication of the available quantity of the item. For example, if the available quantity of the item falls within a first quantity range, a first time period can be designated as a time delay; if the available quantity of the item falls within a second quantity range, a second time period can be assigned as a time delay, and so on for various quantity ranges for the item. As another example, if the number of available items meets the threshold, the first time period can be assigned as a time delay, while if the number of available items fails to meet the threshold, the second time period can be assigned as a time delay.

[0058] For example, suppose the request is for an update on the status of toilet paper, and further suppose that a representative associated with a given entity indicates that toilet paper is available at the given entity. Additionally, suppose that the representative associated with the given entity indicates there is a sufficient quantity of toilet paper to last for the remainder of the day. In this example, the time delay engine 157 can assign a first time period to the time delay associated with the given entity, such that the automation assistant 115 will not initiate a subsequent corresponding automated phone call with the given entity until at least the next day. Conversely, suppose that the representative associated with the given entity indicates the quantity of toilet paper will not last until the end of the day. In this example, the time delay engine 157 can assign a second time period to the time delay associated with the given entity, such that the automation assistant 115 will initiate a subsequent corresponding automated phone call with the given entity later on that same day. It is worth noting that these time delays will be based on the information received during the automated phone call, at different times. Furthermore, search results associated with the given entity can be updated based on information received during the automated phone call, as described above regarding the update engine 155. In these and other ways, a corresponding time delay can be dynamically determined for each entity based on information received during automated phone calls with the entity. Dynamically determining the time delay in this way balances the technological benefits of maintaining up-to-date information from the entity with the resources utilized in executing automated phone calls. In other words, dynamically determining the time delay prevents another automated phone call to the entity from being executed too quickly, which would lead to unnecessary use of computational and / or network resources—while ensuring that the information from the entity is up-to-date.

[0059] Various technical advantages can be achieved by using the techniques described herein. As a non-limiting example, the automated assistant 115 can successfully execute automated phone calls to fulfill requests in a faster and more efficient manner because the automated request system 180 can intelligently initiate automated phone calls. By intelligently initiating automated phone calls, both network and computing resources can be saved because the length and number of conversations can be reduced by using the techniques disclosed herein. As another non-limiting example, search results associated with an entity can be updated based on information received during the conversation. By updating search results associated with an entity, the entity receives fewer phone calls from multiple other users because each other user does not need to initiate a separate phone call with each entity to request status updates on various items and / or inquire about the availability of various services. Furthermore, while this document describes requests for status updates on individual items or services… Figure 1 However, it should be understood that this is for illustrative purposes only and does not imply limitation. For example, Automation Assistant 115 may request status updates on multiple items during an automated phone call with a corresponding entity in a group of entities.

[0060] Figure 2 A flowchart illustrating an example method 300 is provided, illustrating how to determine the time delay of a subsequent automated telephone call with a given entity based on status updates of items of interest received during an automated telephone call with that entity. For convenience, the operation of method 300 is described with reference to a system performing the operation. This system of method 300 includes one or more processors and / or other components of a computing device, such as… Figure 1 Client device 110 Figures 4A to 4D Computing devices 410A, 410B, Figures 5A to 5D Computing device 510, Figures 6A to 6C Computing device 610, Figure 7 The computing device 710, one or more remote servers, and / or other computing devices. Furthermore, although the operations of method 300 are shown in a specific order, this does not imply limitation. One or more operations may be reordered, omitted, or added.

[0061] At box 252, the system identifies items of interest. In some implementations, items of interest can be identified based on analyzing the query activity of multiple users to identify a surge in submitted queries, as described in more detail above (e.g., regarding...). Figure 1 (Query analysis engine 151). In some versions of those implementations, a surge in submitted queries may originate from a specific geographic region. In other implementations, items of interest can be identified based on user input from the user that causes the system to execute method 200. At box 254, the system identifies a group of entities associated with the items of interest. The entities in this group may be stored in association with the items of interest in one or more databases (e.g., Figure 1 The system uses an entity recognition engine 153 and an entity database 153A. Entities can be retrieved from one or more databases. In implementations where a surge in submitted queries may originate from a specific geographic region, the entities in the group can be limited to those located within that specific geographic region.

[0062] At box 256, the system initiates the execution of a corresponding automated phone call to each entity in the group to request a corresponding status update regarding the item of interest. The system can initiate the automated phone call by dialing the automated phone call and rendering one or more corresponding instances of synthesized speech requesting a corresponding status update regarding the item of interest. In some implementations, the system can use a corresponding instance of an automation assistant (e.g., automation assistant 115) to simultaneously initiate the execution of multiple corresponding automated phone calls to each entity in the group. In other implementations, the system can initiate execution only on a subset of the entities, such that automated phone calls to each entity are initiated in batches until automated phone calls to every entity in the group have been executed. In various implementations, the instance of the automation assistant can be an instance of a cloud-based assistant.

[0063] At box 258, during the execution of a corresponding automated telephone call for a given entity in the group of entities, the system: A) requests a corresponding status update from the representative associated with the given entity regarding the item of interest, and B) receives information about the item of interest from the representative. The system can execute the corresponding automated telephone call by engaging in a dialogue with the representative associated with the given entity. Furthermore, the system can request the corresponding status update by rendering one or more corresponding instances of synthesized speech during each automated telephone call. For example, the system can cause the synthesized speech (e.g., “Hello, do you sell toilet paper?”, “Do you have toilet paper available?”, “How much toilet paper do you have available?”) to be rendered at the computing device of the representative associated with the given entity. Additionally, the system can use (e.g., as about…) Figure 1 (A more detailed description of the speech recognition model) The speech recognition model processes audio data corresponding to information received during a corresponding automated telephone call. The information received from the representative can be rendered in response to synthesized speech. As mentioned above regarding Figure 1 In more detail, the system can intelligently respond to representatives during a conversation.

[0064] At box 260, the system updates the search results associated with a given entity to include information about items of interest. In response to receiving corresponding information from each entity in the group, the search results can be updated for each entity in the group. In some implementations, the updated search results may also include an indicator of when the corresponding automated phone call was executed and the time the information was received. For example, suppose the received information indicates that sample store 1 has toilet paper available. The search results associated with sample store 1 can be updated to indicate that toilet paper is available at sample store 1 and the time and date the information was received.

[0065] At box 262, the system determines whether a given entity is still associated with the item of interest. The system can determine this based on information received from box 258 regarding the item of interest. If the system determines in the iteration of box 262 that the given entity is still associated with the item of interest, the system proceeds to box 264. For example, suppose the system renders the synthesized speech “Do you sell toilet paper?” at a computing device associated with the representative to verify that the given entity is still associated with the item of interest. Further suppose the received information indicates that example store 1 sells toilet paper. In this example, the system has already verified that the given entity is still associated with the item of interest.

[0066] At box 264, the system determines a time delay before initiating a corresponding automated telephone call to a given entity, based on received information about the item. Furthermore, the system can assign the determined time delay to a given entity in one or more databases. The determined time delay can be a time period assigned to a given entity based on the availability of the item of interest. For example, suppose a representative associated with example store 1 indicates that there is a sufficient quantity of toilet paper for three days. In this example, the system can assign a three-day time period as a time delay based on the received information indicating that there is a sufficient quantity of toilet paper for three days at example store 1. In contrast, suppose a representative associated with example store 1 expects only a sufficient quantity of toilet paper for the morning. In this example, the system can assign a five-hour time period as a time delay based on the received information indicating that there is a sufficient quantity of toilet paper for three days at example store 1.

[0067] At box 266, the system determines whether the time delay for assigning the given entity, determined in box 264, has elapsed. If, during the iteration of box 266, the system determines that the time delay has not yet elapsed, the system continues to monitor the elapsed time delay for assigning the given entity at box 266. It is worth noting that although the corresponding automated calls can initially be executed in parallel, subsequent automated call calls are initiated based on the determined elapsed time delay. However, if, during the iteration of box 266, the system determines that the time delay determined in box 264 has elapsed, then the system proceeds to box 268.

[0068] At box 268, the system initiates a corresponding follow-up automated call to a given entity to request further status updates regarding the item of interest. The system may request further status updates regarding the item of interest in a similar manner to that described above with respect to box 258. Furthermore, the system may return to box 264 to determine further time delays based on information received during the corresponding follow-up automated call. The system may continue to determine the time delay for the corresponding further automated call until one or more conditions are met. One or more conditions may include, for example, a decrease in the surge of queries submitted for the item of interest, a threshold duration (e.g., one week, two weeks, one month, etc.), and / or the satisfaction of other conditions.

[0069] However, if, during the iteration of box 262, the system determines that an entity is no longer associated with an item of interest, the system can avoid the determination time delay and proceed to box 270 to remove the given entity from the group of entities associated with the item of interest. Therefore, the system will not initiate a subsequent automated call to the given entity to request any further state updates regarding the item of interest. However, the system can still subsequently call the given entity to request state updates regarding other items of interest.

[0070] Although a description of a single entity is given in Figure 2 The time delays described herein are for illustrative purposes only and should not be construed as limitations. As mentioned above, an automated assistant can simultaneously initiate multiple automated phone calls to entities within a group. During each automated phone call, a time delay can be assigned to the corresponding entity within the group, and the time delay for each entity can vary based on information received from the representative associated with the corresponding entity within that entity regarding the item of interest.

[0071] Figure 3 A flowchart illustrating an example method 300 for determining whether to initiate an additional automated telephone call with an additional entity based on information received during an automated telephone call with a given entity. For convenience, the operation of method 300 is described with reference to a system performing the operation. This system of method 300 includes one or more processors and / or other components of a computing device, such as… Figure 1 Client device 110 Figures 4A to 4D Computing devices 410A, 410B, Figures 5A to 5D Computing device 510, Figures 6A to 6C Computing device 610, Figure 7 The computing device 710, one or more remote servers, and / or other computing devices. Furthermore, although the operations of method 300 are shown in a specific order, this does not imply limitation. One or more operations may be reordered, omitted, or added.

[0072] At box 352, the system receives from the user's computing device a request to initiate an automated telephone call on behalf of the user. This request may be based on user input received at the user's computing device. In some embodiments, user input may be touch or typed input from a search interface or an automation assistant interface displayed on the user's computing device. For example, user input may be touch input pointing to a selectable element on the search interface, which, when selected, causes the system to initiate an automated telephone call to execute the request. In other embodiments, user input may be verbal input detected via the microphone of the user's computing device. Verbal input can be processed using a speech recognition model and / or an NLU model to determine if the user input includes a request (e.g., as described above regarding...). Figure 1 (As described).

[0073] At box 354, the system identifies the action to be performed during an automated telephone call based on the request. This action can be identified based on one or more terms included in the user input. The action may include, for example, requesting a status update on an item of interest from a given entity or entities, booking an item of interest at a location associated with a given entity, obtaining an item of interest at a location with a given entity, requesting the availability of services from a given entity or entities, and / or other actions that can be performed using the automated telephone call described herein. For example, the action may include obtaining toilet paper from a specific entity, or the action of obtaining toilet paper normally without specifying any particular entity. As another example, the action may include finding the cheapest electrician available during a given time period. (The above (e.g., regarding...) Figure 1 The request handling engine (152) describes in more detail the identification of actions to be performed.

[0074] At box 356, the system identifies groups of entities that can each fulfill a request based on the actions to be performed during an automated call. Entities in this group may be stored in one or more databases in association with the items or services included in the request (e.g., Figure 1 The system uses an entity recognition engine 153 and an entity database 153A. Entities can be retrieved from one or more databases. Furthermore, entities in a group can be limited to those located within a threshold distance of the user's current location on their computing device. For example, assuming the action included in the request is a request for a status update on a project, the system could include only entities associated with the project within a five-mile radius from the group of entities.

[0075] At box 358, the system initiates the execution of an automated telephone call to a given entity within the group of entities. The system can initiate the automated telephone call by dialing the automated call and rendering one or more corresponding instances of synthesized speech requesting a corresponding status update for the item of interest. In some implementations, the system can use an automation assistant operating locally on the computing device of the user making the request to initiate the execution of the automated telephone call. In other implementations, the system can use an automation assistant operating remotely on a server (e.g., a cloud-based automation assistant) to initiate the execution of the automated telephone call. In still other implementations, the system can use an automation assistant operating partly on the user's computing device and partly on a server to initiate the execution of the automated telephone call (e.g., such as...). Figure 1 The automated assistant described in the document (115).

[0076] At box 360, during the execution of an automated telephone call with a given entity, the system engages in a dialogue with a representative of the given entity to perform an action. During the dialogue, the system may perform the action and receive information about the action from the representative (e.g., information about the availability of a project or service and / or other information described herein). The system may perform the action by rendering one or more corresponding instances of synthesized speech during each automated telephone call. For example, the system may cause synthesized speech (e.g., “Hello, do you sell toilet paper?”, “Do you have toilet paper available?”, “How much toilet paper do you have available?”) to be rendered at an additional computing device of the representative associated with the given entity. Furthermore, the system may use (e.g., information about...) Figure 1 (A more detailed description of the speech recognition model) The speech recognition model processes audio data corresponding to information received during a corresponding automated telephone call. The information received from the representative can be rendered in response to synthesized speech. As mentioned above regarding Figure 1 In more detail, the system can intelligently respond to representatives during a conversation.

[0077] At box 362, the system determines whether to initiate the execution of an additional automatic call to an additional entity within the group. The system may determine whether to initiate the execution of an additional automatic call to an additional entity based on the call initiation time associated with the additional entity. The call initiation time may be specific to the additional entity. In some implementations, this may be based on historical call data stored associated with the entity and / or the additional entity (e.g., Figure 1The call initiation time can be determined using historical call data database 154A. For example, the call initiation time of an additional entity can be determined based on whether the entity and / or additional entity is associated with the IVR system, the average duration of previous automated call to the entity and / or additional entity, the average hold time associated with the entity and / or additional entity, and / or any other call data that can be collected from previous automated call to the entity and / or additional entity. For example, if the determined call initiation time is 30 seconds after the initiation of an automated call to a given entity, it can be based on determining that both the given entity and the additional entity are associated with a 30-second hold time. In these examples, the automated call calls are staggered, allowing the automated assistant to contact a representative associated with each entity 30 seconds apart, rather than contacting two representatives simultaneously. In some additional and / or alternative implementations, the call initiation time can be determined based on the outcome of the automated call to the entity. Continuing the example above, if the system only holds for 20 seconds instead of the desired 30 seconds, the call initiation time can be changed from 30 seconds to 20 seconds, enabling a similar staggering of automated call calls. If, in the iteration of box 362, the system determines not to initiate the execution of an additional automated call, the system proceeds to box 366. It is noteworthy that this indicates the request can be handled by the given entity, and the given entity indicates it can handle the request before the call is initiated. For example, this indicates the given entity to quickly indicate it can fulfill the request. However, if, in the iteration of box 362, the system determines to initiate the execution of an additional automated call with the additional entity, the system can proceed to box 364. As described above with respect to box 358, the system can initiate additional automated telephone calls using an automated assistant that is local to the user's computing device, remotely at a server, and / or a combination thereof.

[0078] At box 364, the system determines whether to proactively terminate the automated call to the given entity and / or the additional automated call to the additional entity. The system may determine whether to proactively terminate the automated call and / or the additional automated call based on information received during each automated call. For example, if the entity or additional entity indicates that a project or service is unavailable, the corresponding automated call may be proactively terminated. In embodiments where the given entity indicates that it can fulfill the request and an additional representative associated with the additional entity has already engaged in a conversation with the automation assistant, the system may not proactively terminate the additional automated call. In some versions of those embodiments, the system may still perform the action and may update the search results associated with the additional entity based on the execution of the action. In this way, the computing resources of the additional computing devices of the representative associated with the additional entity are not wasted by terminating the additional automated call without knowing any information related to the original request. If, in the iteration of box 364, the system does not determine to proactively terminate the automated call to the given entity and / or the additional automated call to the additional entity, the system may monitor one or more signals at box 364 to proactively terminate the automated call to the given entity and / or the additional automated call to the additional entity. However, if in the iteration of box 364, the system determines to actively terminate the automatic call with a given entity and / or the additional automatic call with an additional entity, the system proceeds to box 366.

[0079] At box 366, the system generates a notification based on the outcome of a conversation during an automated call with a given entity and / or an additional automated call with an additional entity. A notification can be generated for each automated call initiated by the system. Furthermore, the notification may include, for example, an indication of the availability of a project or service, whether the system has booked a project or service on behalf of the user, or whether the system has obtained a project and / or other outcome of the conversation on behalf of the user. In some implementations, the notification may include a prompt for the user. The prompt may include, for example, a prompt for the user to join an automated call, or a prompt for the user to authorize the system to provide the user's name, address, contact information, financial information, and / or other user data on the computing device. At box 368, the system causes the notification to be visually and / or audibly rendered via the user's computing device. The notification may be visually rendered via the display of the computing device or another computing device associated with the user and / or audibly rendered via the speaker of the computing device or another computing device associated with the user. In this document (e.g., regarding...), Figure 1 , 5B The generation and rendering of notifications are described in more detail in versions 5D, 6B, and 6C.

[0080] Although this document describes method 300 with respect to a given entity and additional entities, it should be understood that this is for illustrative purposes only and is not intended to be limiting. It should be understood that the call initiation time for each other entity in the group can be determined, and each other entity in the group can be called until the request is satisfied. If no entity in the group can satisfy the request, it can be done as described above (e.g., regarding...). Figure 1 The entity recognition engine 153) identifies additional groups of entities.

[0081] Now for reference Figures 4A to 4C This describes non-limiting examples of determining the time delay of subsequent automated telephone calls with various entities. It is worth noting that... Figures 4A to 4C The text record of the automated telephone call is displayed on the graphical user interface 480A of the computing device 410A. It should be understood that this is for illustrative purposes only and not intended to limit the technology described herein. For example, for the sake of brevity, the automated telephone call described herein could be initiated and executed by an automation assistant (e.g., using...). Figure 1 (Automatic request system 180). The automated assistant may be a cloud-based automated assistant that executes remotely on a server and / or other computing device.

[0082] In some implementations, and as described above (e.g., regarding...), Figure 1 (In more detail, the query analysis engine 151 and request processing engine 152) allow the automation assistant to analyze the query activities of multiple users (e.g., stored in...) Figure 1 The query activity database 151A identifies requests for status updates regarding items or services of interest. An automated assistant can initiate and execute automated phone calls in response to such requests. The automated assistant can identify requests based on, for example, determining whether multiple users have submitted a threshold number of queries for a given item or service. In some versions of those implementations, one or more conditions may include determining whether multiple users have submitted a threshold number of queries for a given item or service within a given time frame, as described above regarding… Figure 1 As described. In some versions of those implementations, query activity may be limited to queries submitted by users within a specific geographic region.

[0083] In some implementations, the automation assistant can further identify groups of entities associated with the identified request, as described herein (e.g., regarding...). Figure 1The entity recognition engine 153 is described in more detail. Groups of entities can be stored (e.g., in an entity database 153A) along with other data associated with each entity (e.g., physical location, web page, and / or other data) in association with the items or services included in the request. A corresponding automatic phone call can be initiated and executed by the automation assistant with a given entity in the group, allowing the automation assistant to initiate and execute a corresponding automatic phone call with each individual entity in the group. In embodiments where the submitted query originates from a specific geographic region, the entities included in the group can also be limited to entities located within that specific geographic region. In response to identifying items or services based on the analysis of query activity and in response to identifying a group of entities associated with the items or services included in the request, the automation assistant can initiate and execute an automatic phone call. Furthermore, the automation assistant can execute a corresponding automatic phone call by rendering an instance of synthesized speech at the computing device of the corresponding representative associated with the corresponding entity in the entity. Additionally, the automation assistant can process audio data corresponding to information received from the corresponding representative during the corresponding automatic phone call. In this way, the automated assistant can participate in a conversation with a representative, and the subsequent synthesized speech can be based on information received from the corresponding representative.

[0084] For example, regarding Figures 4A to 4C Suppose an automated assistant identifies 15,000 submitted queries over a six-hour period, all containing the term "toilet paper" and originating from the geographic region of Jefferson County, Kentucky. In this example, the automated assistant could determine that the 15,000 submitted queries over the six-hour period satisfy one or more conditions to initiate multiple corresponding automated phone calls to groups of entities located in Jefferson County, Kentucky, to request corresponding status updates regarding toilet paper availability. Therefore, the automated assistant can access one or more databases (e.g., ...). Figure 1 The entity database (153A) is used to identify multiple entities stored in Jefferson County, Kentucky, that are associated with a toilet paper project, and the identified entities can be included in groups of entities. Figures 4A to 4C As shown, the group of entities includes at least Example Store 1, Example Store 2, and Example Store 3.

[0085] More specifically, the automated assistant can initiate an automated phone call with sample store 1 to request an update on the status of toilet paper at sample store 1, such as... Figure 4A As shown, an automated phone call can be initiated with Example Store 2 to request an update on the status of toilet paper at Example Store 2, such as... Figure 4B As shown, and the ability to initiate a first automated phone call with Example Store 3 to request an update on the status of toilet paper at Example Store 3, such as Figure 4CAs shown. It is worth noting that it is possible to initiate and execute [the program] in parallel. Figures 4A to 4C Each of the first automated phone calls for each entity in the system (e.g., as instructed by being initiated at 9:00 AM on 6 / 11 / 2020). Furthermore, each of the first automated phone calls can be initiated and executed using a corresponding instance of the automated assistant.

[0086] As described above, an automated assistant can execute each step of a corresponding automated telephone call by rendering an instance of synthesized speech on a computing device representing a given entity. Specifically, the synthesized speech may include requests for updates regarding the status of toilet paper, and optionally other information. For example, such as... Figure 4A As shown, the automated assistant can cause synthesized speech 452A1 to be rendered at the computing device represented by example store 1, which includes a request for an update on the status of toilet paper at example store 1 (e.g., “I’m calling to see whether you have any toilet paper available”) and a request for the quantity of toilet paper available at example store 1 (e.g., “whether you anticipate running out today”). Furthermore, the automated assistant can process audio data corresponding to information 454A1 received in response to the synthesized speech (e.g., “We have some available, but will likely run out this afternoon”) to determine whether toilet paper is available at example store 1 and the quantity available. As another example, such as Figure 4BAs shown, the automated assistant can cause synthesized speech 452B1 to be rendered at the computing device represented by example store 2, which includes a request for an update on the status of toilet paper at example store 2 (e.g., “I’m calling to see whether you have any toilet paper available”). However, the automated assistant can suppress requests for the quantity of toilet paper available at example store 2 until the representative of example store 2 confirms, based on processing audio data (e.g., “Yes”) corresponding to the information received from the representative of example store 2 (454B1). In response to verifying that example store 2 has toilet paper available, the automated assistant can then request an update on the status of the quantity of toilet paper available at example store 2 in further synthesized speech 452B2 (e.g., “Will you run out today?”), and can determine the indication of the quantity of toilet paper available at example store 2 based on processing further received information 454B2 (e.g., “No”, which indicates that example store 2 will not run out of toilet paper today). Furthermore, the automated assistant can process audio data corresponding to information 454A1 received in response to synthesized speech (e.g., "We have some available, but will likely run out this afternoon"). As yet another example, Figure 4C As shown, the automated assistant can cause synthesized speech 452C1 to be rendered at the computing device represented by example store 3. This speech includes a request for an update on the status of toilet paper at example store 3 (e.g., “I’m calling to see whether you have any toilet paper available”) and further includes a request for the quantity of toilet paper available at example store 3 (e.g., “whether you anticipate running out today”). Furthermore, the automated assistant can process audio data corresponding to information 454C1 received in response to the synthesized speech (e.g., “We do not sell toilet paper”) to determine that example store 3 should not be associated with toilet paper storage. In this example, example store 3 can be removed from the group of entities based on the received information 454C1.

[0087] for Figures 4A to 4CIn each of the corresponding automated phone calls, the automation assistant can assign a time period as a time delay to each entity in the group. As the time delay elapses, for the corresponding entity in the group, the automation assistant can initiate and execute a subsequent corresponding automated phone call with that entity to request further status updates regarding that item. The time period assigned as a time delay can be determined based on information received from a representative associated with a given entity (e.g., as described in more detail above with respect to time delay engine 157). In some implementations, the automation assistant can provide an indication of the time period of the time delay assigned to the corresponding entity during each first automated phone call.

[0088] For example, such as Figure 4A As shown, the automated assistant can enable the rendering of further synthesized speech 452A2 at the computing device represented by Example Store 1, indicating a time delay assigned to Example Store 1 (e.g., "I'll call back this afternoon for another update"). In this example, the time period can be only a few hours (e.g., as shown in the image). Figure 4A (as shown in the five hours), because information 454A1 instructs Example Store 1 to "likely run out of toilet paper this afternoon." Therefore, as the time delay elapses (e.g., five hours after the first automated phone call with Example Store 1 (e.g., 2:00 PM on the same day)), the automated assistant can initiate and execute a second automated phone call to request a corresponding further status update regarding toilet paper by: rendering a further synthesized voice 452A3 at the computing device representing Example Store 1, processing audio data corresponding to the further information 454A2 received from the representative of Example Store 1 in response to the further synthesized voice 452A3, and rendering a further synthesized voice 452A4 at the computing device representing Example Store 1, which includes an indication of a further time delay determined based on the further information 452A2 received during the second automated phone call with the representative of Example Store 1. For example, if the representative of Sample Store 1 in Further Information 452A2 instructs Sample Store 1 that toilet paper will not be available until the goods arrive in three days, the period of time assigned as a further time delay could be 72 hours to avoid calling Sample Store 1 within the next three days, since they have already instructed them that they will not have toilet paper until the next goods arrive.

[0089] As another example, such as Figure 4BAs shown, the automated assistant can render further synthesized speech 452B3 (e.g., “I’ll call back tomorrow”) at the computing device represented by example store 2, indicating a time delay assigned to example store 2. In this example, the time period could be a 24-hour period from when message 454B2 indicates that example store 2 will not run out of toilet paper that day. Therefore, as the time delay elapses (e.g., 24 hours after the first automated phone call with Example Store 2 (e.g., at 9:00 AM the next day)), the automated assistant can initiate and execute a second automated phone call to request a corresponding further status update regarding toilet paper by: rendering further synthesized speech 452B4 at the computing device represented by Example Store 2; processing audio data corresponding to further information 454B3 received from the representative of Example Store 2 in response to the further audio data 452B4; and rendering further synthesized speech 452A5 at the computing device represented by Example Store 2, which includes an indication of a further time delay determined based on the further information 452B3 received during the second automated phone call with the representative of Example Store 2. Therefore, as Figure 4A and 4B As indicated by the time delay, the automated assistant can determine the dynamic time period to be assigned to the time delay based on the information received during the automated phone call.

[0090] It is worth noting that, such as Figure 4C As shown, the automated assistant can cause further synthesized speech 452C2 to be rendered on the computing device represented by example store 3, which instructs the automated assistant to avoid future requests for toilet paper from example store 3. In this example, the automated assistant can also avoid determining the time period assigned to example store 3 as a time delay because it has already been removed from the group of entities associated with the item based on received information 454C1, as described above. Therefore, a second automated phone call with example store 3 will not be initiated or executed.

[0091] In various implementations, search results associated with entities in a group can be updated to include information received during the corresponding automated phone call. In some versions of these implementations, search results can also be updated to include a time indicator of when a status update was requested. For example, Figure 4D The graphical user interface 480B of the user's computing device 410B illustrates a diagram based on data from... Figures 4A to 4C The results of automated telephone calls are updated search results associated with each of the various entities. Figure 4DThe graphical user interface 480B further includes: a text response interface element 484 that the user can select to generate user input via a virtual keyboard or other touch and / or typing input; and a voice response interface element 485 that the user can select to generate user input via a microphone of the client device 410. In some embodiments, the user can generate user input via the microphone without selecting the voice response interface element 485. For example, actively monitoring audible user input via the microphone may occur to avoid the need for the user to select the voice response interface element 485. In some and / or other embodiments of those embodiments, the voice response interface element 485 may be omitted. Furthermore, in some embodiments, the text response interface element 484 may be additionally and / or alternatively omitted (e.g., the user can provide only audible user input). Figures 4A to 4D The graphical user interface 480 further includes system interface elements 481, 482, and 483, with which users can interact to cause the computing device 410 to perform one or more actions.

[0092] refer to Figure 4D Assume that the user of computing device 410B is also located in Jefferson County, Kentucky, and further assume that the user of computing device 410B submitted a search query (e.g., spoken or typed via an automated assistant interface or search interface) including the term “toilet paper” (e.g., as indicated by URL 411) after the first automated phone call in 4A to 4C but before the second automated phone call. Further assume that the first search result 420 associated with example store 1, the second search result 430 associated with example store 2, and the third search result 440 associated with example store 3 are all in response to the search query. In this example, the first search result 420 can be updated to include information 421 from the first automated phone call with example store 1, indicating that toilet paper is “available” at example store 1 at “9:00 AM today,” but example store 1 may run out of toilet paper “this afternoon.” Additionally, the second search result 430 can be updated to include information 431 from the first automated phone call with example store 2, indicating that toilet paper was “available” at example store 1 “this morning,” and example store 2 should “not run out today.” Furthermore, the third search result 440 can be updated to include information 441 from the first automated phone call to Example Store 3 indicating that Example Store 3 "does not sell toilet paper." Information 421, 431 of search results 420, 430 can be updated in a similar manner after the second automated phone call and any subsequent automated phone calls. In this way, when the automation assistant performs automated phone calls, the search results associated with each entity in the group can be updated to reflect the availability of various items associated with each corresponding entity.

[0093] Although this article uses requests for state updates about a single item of interest as a description Figures 4A to 4D However, it should be understood that this is for illustrative purposes and not intended to be limiting. For example, if an automation assistant identifies multiple items of interest for which it needs to request status updates, and a given entity in a group is associated with multiple items, the automation assistant can request status updates for each of the multiple items in a first automated call, avoiding the need to initiate and execute multiple automated calls with the given entity to request status updates for each of the multiple items. Furthermore, it should be noted that when an automation assistant requests status updates for multiple items, different time periods can be assigned to the time delay of each of the multiple items. For example, if the automation assistant initially requests status updates for both toilet paper and hand sanitizer, and a representative associated with the given entity provides information indicating that there is sufficient toilet paper for the remainder of the day but no hand sanitizer for the next four days, the automation assistant can assign a 24-hour time period to initiate and execute subsequent automated calls to request further status updates for toilet paper, and a 96-hour time period to initiate and execute subsequent automated calls to request further status updates for hand sanitizer.

[0094] Furthermore, although this article describes the request for status updates regarding items of interest... Figures 4A to 4D However, it should be understood that this is for illustrative purposes and not intended to be limiting. As a non-limiting example, an automated assistant could also request status updates regarding the availability of services. For instance, if an automated assistant determines a surge in queries for body repairs and / or roof repairs (e.g., after a hailstorm), it could initiate and execute automated phone calls in a similar manner, and update search results associated with entities providing these services in a similar way.

[0095] Now for reference Figures 5A to 5D It describes various non-limiting examples of initiating automated phone calls on behalf of users with various entities and rendering notifications associated with each of the various entities based on the results of the automated phone calls. Figures 5A to 5D Each of the above describes a computing device 510 having a graphical user interface 580 associated with a user request that is initiated and / or executed by an automated telephone call on behalf of the user of the computing device 510. An automated assistant (e.g., [missing information]) is associated with the user of the computing device 510. Figure 1 One or more aspects of the automation assistant 115 can be implemented locally on computing device 510 and / or in a distributed manner (e.g., via... Figure 1 The network 190) is implemented on other computing devices that communicate with computing device 510 via the network. For simplicity, Figures 5A to 5D The operations described in this article are performed by an automation assistant (e.g., using...). Figure 1 The automatic request system 180) is executed. Although Figures 5A to 5D The computing device 510 is depicted as a mobile phone, but it should be understood that this is not intended to be limiting. The computing device 510 can be, for example, a server, a standalone speaker, a speaker connected to a graphical user interface, a laptop computer, a desktop computer, and / or any other client device capable of making telephone calls and automating assistant operations.

[0096] Figures 5A to 5D The graphical user interface 580 further includes: a text response interface element 584 that the user can select to generate user input via a virtual keyboard or other touch and / or typing input; and a voice response interface element 585 that the user can select to generate user input via a microphone of the computing device 510. In some embodiments, the user can generate user input via the microphone without selecting the voice response interface element 585. For example, actively monitoring audible user input via the microphone may occur to avoid the need for the user to select the voice response interface element 585. In some and / or other embodiments of those embodiments, the voice response interface element 585 may be omitted. Furthermore, in some embodiments, the text response interface element 584 may be additionally and / or alternatively omitted (e.g., the user can provide only audible user input). Figures 5A to 5D The graphical user interface 580 further includes system interface elements 581, 582, and 583, with which users can interact to enable the computing device 510 to perform one or more actions.

[0097] In various implementations, an automated telephone call can be initiated and executed in response to user input requesting the automation assistant to initiate and execute the automated telephone call. User input can be verbal input, touch input, and / or typed input that includes instructions to initiate and execute the automated telephone call. Furthermore, the automation assistant can perform actions included in the request on behalf of a given user of computing device 510 during the automated telephone call. Figure 5AAs shown, the graphical user interface 580 includes a first search result 520 associated with the entity of imaginary toy store 1 and a second search result 530 associated with the entity of imaginary toy store 2, displayed in response to a query for "hot Christmas toys of 2020" (e.g., as indicated in the text response interface element 584). Furthermore, search results 520 and 530 are displayed in a browser application accessible on a computing device (e.g., as indicated by URL 511 "www.exampleurl0.com / "). Additionally, search results 520 and 530 may also include corresponding information 521 and 531, for example, related to one or more items of interest (e.g., hot Christmas toys). For example, the first information 521 associated with the first search result 520 indicates that the hot Christmas toys of imaginary toy store 1 were out of stock six days ago, while the second information 531 associated with the second search result 530 indicates that the status of the hot Christmas toys of imaginary toy store 2 is unknown and has never been updated.

[0098] In some implementations, search results 520, 530 may be associated with various selectable graphical elements that, when selected, cause computing device 510 to perform a corresponding action. For example, when a call graphical element 522, 532 associated with a given one of search results 520, 530 is selected, user input may indicate that a telephone call should be initiated and executed. As another example, when directional graphical elements 523, 533 are associated with a given one of search results 520, 533, user input may indicate that a navigation action should be initiated and executed. In other implementations, and although not depicted, when a general automatic telephone call graphical element is selected, user input may explicitly indicate that an automatic telephone call should be initiated and executed.

[0099] In some implementations, and as such Figure 5AAs shown, in response to receiving user input initiating and executing a telephone call with any entity (e.g., in response to receiving a selection of call interface elements 522, 532, or a general automated telephone call graphical element), a call details interface 570 can be rendered at computing device 510. In some versions of those embodiments, the call details interface 570 may be rendered at computing device 510 as part of a graphical user interface 580. In some other versions of those embodiments, the call details interface 570 may be an overlay interface separate from the graphical user interface 580 and may include interface element 586 that allows the user to expand the call details interface 570 to display additional call details (e.g., by swiping up on interface element 586) and / or close the call details interface 570 (e.g., by swiping down on interface element 586). In other embodiments, the call details interface 570 can be accessed and closed at any time using interface element 586. Although the call details interface 570 is depicted at the bottom of the graphical user interface 580, it should be understood that this is for illustrative purposes and not intended to be limiting. For example, the call details interface 570 can be rendered on top of the graphical user interface 580, on the side of the graphical user interface 580, or on an interface completely separate from the graphical user interface 580.

[0100] In various implementations, the call details interface 570 may include multiple selectable text segments (or graphical elements). In some implementations, the segments may be selectable, such that when a given segment is selected, the computing device 510 can perform a corresponding action. For example... Figure 5A As shown, the call details interface 570 includes a first selectable text segment 571 for "Automatic Phone Call" and a second selectable text segment 572 for "Regular Call". Furthermore, when selected, the first selectable text segment 571 can provide an instruction to the automation assistant to initiate an automated phone call, and when selected, the second selectable text segment 572 can cause the automation assistant to initiate a regular phone call. It is worth noting that in some versions of those embodiments, the selectable text segments may include other associated selectable text segments. For example, the first selectable text segment 571 for "Automatic Phone Call" may include: a first additional selectable text segment 571A for "Toy Status Update" associated with the action of requesting a status update during an automated phone call; a second additional selectable text segment 571B for "Reserve Toy" associated with the action of keeping a toy for the user of the computing device 510 during an automated phone call; and a third additional selectable text segment 571C for "Purchase Toy" associated with the action of acquiring a toy for the user of the computing device 510 during an automated phone call. Although the text herein refers to... Figures 5A to 5DThe described automated phone call originates from a browser-based application, but it should be understood that this is for illustrative purposes only and not intended to be limiting. For example, automated phone calls can be initiated from various software applications accessible at computing device 510 (e.g., automation assistant applications, contact applications, email applications, text or SMS messaging applications, and / or other software applications), and if the automated phone call is initiated using verbal input, it can be initiated from the home screen of computing device 510, from the lock screen of computing device 510, and / or from other states of computing device 510.

[0101] For example, suppose that user input is detected at computing device 510 to initiate and execute an automated telephone call to the first search result 520 of "imaginary toy store 1". The user input could be verbal input such as "Find popular Christmas toys 2020" or touch input pointing to call interface element 522 (and / or one of selectable text segments 571, 571A to 571C). In various implementations, the automated assistant can identify each item associated with "popular Christmas toys" stored in one or more databases (e.g., [database name missing]) based on the user input indicating "popular Christmas toys". Figure 1 The entity database 153A contains groups of entities. In some implementations, the automation assistant may limit the entities included in a group to those stored in association with popular Christmas toys and within a threshold distance of the location of computing device 510 (e.g., determined based on GPS data generated by the GPS sensor of computing device 510). For example, the automation assistant may initially include all entities stored in association with popular Christmas toys and within four miles of the location of computing device 510 in the group of entities. If not, the automation assistant may increase the threshold distance to include all entities within eight miles of the location of computing device 510, and so on. In some additional and / or alternative implementations, the automation assistant may continue to identify entities to include in the group until a threshold number of entities are identified for inclusion in the group. For example, the automation assistant may continuously increase the threshold distance to identify entities to be included in the group until five entities are included in the group. If the number of entities included in the group exceeds the threshold number, entities that meet one or more criteria may be included in the group (e.g., as described in more detail above with respect to entity recognition engine 153). It is worth noting that even if user input may involve initiating an automated phone call with a specific entity or without specifying an entity, groups of entities associated with popular Christmas toys can still be identified.

[0102] An automation assistant can initiate and execute automated phone calls with one or more entities in a group until the request is satisfied. In some implementations, the automation assistant can initiate a first automated phone call with a first entity in the group in response to receiving user input, and determine the call initiation time for initiating execution of additional automated phone calls with corresponding additional entities in the group. In some versions of those implementations, the call initiation time for each additional automated phone call can be a static threshold duration (e.g., 15 seconds, 30 seconds, 45 seconds, and / or other durations). For example, the automation assistant can initiate a first automated phone call with a first entity in the group, initiate a second automated phone call with a second entity in the group 30 seconds after initiating the first automated phone call with the first entity, initiate a third automated phone call with a third entity in the group 30 seconds after initiating the second automated phone call with the second entity, and so on for each entity in the group. In other versions of those implementations, the automation assistant can determine each call initiation time based on historical call data associated with corresponding entities in the group (e.g., stored in a historical call database 154A). Historical call data may include, for example, indications of the duration of the corresponding entity's association with and navigation of the IVR system, indications of the duration of previous automated telephone calls with the corresponding entity, indications of the duration of the initial hold-up time during previous automated telephone calls with the corresponding entity, and / or other call data derived from previous automated telephone calls with each corresponding entity in the group. (This document, for example, regarding...) Figure 1 The request execution engine 154 describes in more detail the determination of the call initiation time associated with each entity in the group. Furthermore, the automation assistant can enable corresponding notifications (e.g., audibly via the speakerphone of the computing device 510 and / or visually via the graphical user interface 580 of the computing device 510) to be rendered at the computing device 510, including the results of the automated phone call, prompts to the user of the computing device 510, and / or other information related to the automated phone call.

[0103] For example, see specific references Figure 5B Suppose that user input is detected at computing device 510 to initiate an automated telephone call to request a status update on popular Christmas toys (e.g., pointing to...). Figure 5AThe first alternative is touch input of text segment 571A or verbal input of “Find Hot Christmas Toys”. Further assuming the automated assistant recognizes at least hypothetical toy store 1 and hypothetical toy store 2 as entities in the group associated with hot Christmas toys. The automated assistant can initiate an automated phone call to hypothetical toy store 1 and request a status update on hot Christmas toys as an action to be performed based on user input. For example, the automated assistant can cause synthesized speech 552B1 to be rendered at a computing device associated with a representative of hypothetical toy store 1 (e.g., “I’m calling to see whether Hot Christmas Toy is available at your store”), and can process audio data corresponding to information 554B1 received in response to synthesized speech 552B1 (e.g., “Please hold while I go check”).

[0104] Furthermore, the automated assistant can process audio data corresponding to further information 554A2 (e.g., "we are out of stock") when the virtual toy store 1 representative returns to the automated phone call, and can cause further synthesized speech 552B2 (e.g., "thank you for checking") to be rendered on the computing device associated with the virtual toy store 1 representative. Based on the outcome of the dialogue, the automated assistant can cause notification 579B to be visually rendered at a graphical user interface (e.g., via notification interface 590, which can replace call details interface 570, and can be expanded or closed using an element interface 586 similar to call details interface 570). Furthermore, the automated assistant can... Figure 5A The first search result 520 depicted in the image is updated to indicate that popular Christmas toys in the hypothetical toy store 1 are out of stock, and optionally includes a time indicator 554B2 indicating the out-of-stock status of popular Christmas toys (e.g., "12 / 1 / 2020 at 9:00:00 AM", "this morning", etc.). Above (e.g., regarding...) Figure 1 and 4D The search results are updated in more detail based on information received during the automated phone call.

[0105] During an automated phone call with imaginary toy store 1, the automated assistant (or a separate instance thereof) may also initiate an additional automated phone call with imaginary toy store 2 based on a determined call initiation time associated with imaginary toy store 2. In some implementations, the call initiation time associated with imaginary toy store 2 may be zero, allowing the automated assistant to perform automated phone calls in parallel with both imaginary toy stores 1 and 2 (e.g., in accordance with the above regarding initiation and execution). Figures 4A to 4C (Similar to the manner described in the first automated phone call in [the previous implementation]). In other embodiments, additional automated phone calls can be interleaved so that the automated assistant (and / or its individual instances) typically engages with only one entity at a time. In some versions of those embodiments, the call initiation time associated with imaginary toy store 2 can be a set duration associated with each entity in the group, such that each additional automated phone call is initiated 30 seconds after each other. In other versions of those embodiments, the call initiation time associated with imaginary toy store 2 can be based on historical call data associated with imaginary toy store 1 and imaginary toy store 2.

[0106] In some additional and / or alternative versions of those implementations, the call initiation time associated with the virtual toy store 2 can be dynamic and based on the progress of the conversation between the automated assistant and the representative of the virtual toy store 1. For example, suppose the call initiation time associated with the virtual toy store 2, determined according to one or more of the techniques described above, is 30 seconds. However, if the above regarding Figure 5A If the described hold period exceeds the original 30-second call initiation time, the call initiation time associated with the imaginary toy store 2 can be adjusted to a dynamic time when the conversation with the representative of imaginary toy store 1 resumes (e.g., when further information 554B2 is received). In contrast, suppose the representative of imaginary toy store 1 indicates in further information 554B2 (or immediately in information 554B1) that popular Christmas toys are available. In this example, the additional automated call with imaginary toy store 2 can be actively terminated. However, if an additional automated call with imaginary toy store 2 has already been initiated and the automation assistant is engaged in an additional conversation with the representative of imaginary toy store 2, the automation assistant can continue the conversation but change its action to requesting a status update on popular Christmas toys and updating the search results associated with imaginary toy store 2 accordingly. (This text (e.g., regarding...) Figure 1 The request execution engine (154) describes in more detail the determination of the call initiation time associated with the entity of the group.

[0107] For example, see specific references Figure 5C Suppose that user input is detected at computing device 510 to initiate an automated telephone call to retrieve a popular Christmas toy on behalf of the user of computing device 510. The user input could be, for example, pointing to... Figure 5AThe second alternative is text segment 571B, which can be accessed via touch or spoken input: "find Hot Christmas Toy and buy it for me." Further, it is assumed that the result of an automated telephone call represented by a hypothetical toy store 1 is based on... Figure 5B The dialogue depicted indicates that popular Christmas toys are out of stock in hypothetical toy store 1. It is also assumed that an additional automated phone call with hypothetical toy store 2 is initiated 30 seconds after the automated phone call with hypothetical toy store 1. Similar to the reference above. Figure 5B The dialogue described herein allows the automated assistant to render synthesized speech 552C1 on a computing device associated with the representative of the hypothetical toy store 2 (e.g., “I’m calling to see whether Hot Christmas Toy is available at your store”), and to process audio data corresponding to information 554C1 received in response to the synthesized speech (e.g., “yes, we have two more available”).

[0108] Furthermore, the automated assistant can render a further synthesized speech 552C2 on a computing device associated with the representative of the hypothetical toy store 2 (e.g., “May I purchase it on behalf of Jane Doe?”, where Jane Doe is the user of computing device 510), and process audio data corresponding to further information 554C2 when the representative of the hypothetical toy store 2 responds to the further synthesized speech 552C2. Based on the outcome of the dialogue, the automated assistant can render a notification 579C visually on a graphical user interface (e.g., via a notification interface 590, which can replace the call details interface 570, and can be expanded or collapsed using interface elements 586 similar to the call details interface 570).

[0109] It is worth noting that notification 579C can indicate that the imaginary toy store 2 has popular Christmas toys, and may also include a prompt requesting the user of computing device 510 (e.g., Jane Doe) to authorize providing monetary information to the representative of imaginary toy store 2 to obtain the popular Christmas toys on behalf of the user. Furthermore, notification interface 590 may also include selectable graphical elements in response to the prompt, such as a first selectable graphical element 579C1 for authorizing the provision of monetary information, a second selectable graphical element 579C2, and a third selectable element 579C3 for the user to add an additional automated telephone call. In various embodiments, the user of computing device 510 may have already previously authorized the provision of monetary information, and the prompt may be omitted. If the user instructs the automated assistant to provide monetary information (e.g., via selecting the first selectable graphic element 579C1), the user can receive further notifications instructing the user, representing the computing device 510, to acquire a popular Christmas toy in the hypothetical toy store 2, providing directions to the hypothetical toy store 2, creating reminders to select the popular Christmas toy in the hypothetical toy store 2, and / or performing other actions based on the outcome of the conversation during an additional automated phone call with the hypothetical toy store 2. Furthermore, once the automated assistant has acquired the popular Christmas toy, it can proactively terminate other ongoing automated phone calls with other entities in the same group.

[0110] For example, see specific references Figure 5D Suppose that user input is detected at computing device 510 to initiate an automated phone call to keep a popular Christmas toy on behalf of the user of computing device 510 (e.g., with obtaining...). Figure 5C (The opposite of requests for popular Christmas toys). User input could be, for example, pointing to... Figure 5A The third alternative is text segment 571C, which can be entered via touch or voice ("find Hot Christmas Toy and place it on hold for me"). Further assumptions are made regarding the result of an automated telephone call represented by a hypothetical toy store 1. Figure 5B The dialogue depicted indicates that popular Christmas toys are out of stock in imaginary toy store 1. It also assumes that an additional automated call to imaginary toy store 2 is initiated 30 seconds after the automated call to imaginary toy store 1. Similar to the reference above. Figure 5B and 5CThe dialogue described herein allows the automated assistant to render synthesized speech 552D1 on a computing device associated with the representative of the hypothetical toy store 2 (e.g., “I’m calling to see whether Hot Christmas Toy is available at your store”), and to process audio data corresponding to information 554D1 received in response to the synthesized speech (e.g., “yes, we have two more available”).

[0111] Furthermore, the automated assistant can render further synthesized speech 554D2 on the computing device associated with the representative of the imaginary toy store 2 (e.g., “May I purchase it on behalf of Jane Doe,” where Jane Doe is the user of computing device 510), and process audio data corresponding to further information 554D2 when the representative of the imaginary toy store 2 responds to the further synthesized speech 552D2. Based on the outcome of the dialogue, the automated assistant can render a notification 579D visually on the graphical user interface (e.g., via a notification interface 590, which can replace the call details interface 570, and can be expanded or collapsed using interface elements 586 similar to the call details interface 570). Notably, the notification 579D can indicate that the imaginary toy store 2 has popular Christmas toys, and that the automated assistant has pre-ordered it on behalf of the user of computing device 510 at the imaginary toy store 2. Although in Figure 5D Not described in the text, but the automation assistant can further enable the computing device 510 to render navigation information to the imaginary toy store 2, selection instructions when the user of the computing device 510 arrives at the imaginary toy store 2 to select popular Christmas toys, links to web pages associated with the imaginary toy store 2, instructions that have been created for selecting popular Christmas toys in the imaginary toy store 2, and so on.

[0112] In various implementations, automated assistants can enable Figure 5A The first search result 520 depicted in the image is updated to indicate that popular Christmas toys in the hypothetical toy store 1 are out of stock, and optionally includes a time indicator 554B2 indicating the out-of-stock status of popular Christmas toys (e.g., "12 / 1 / 2020 at 9:00:00 AM", "this morning", etc.). Furthermore, the automated assistant can enable... Figure 5AThe second search result 530 depicted in the image is updated to indicate that the popular Christmas toy in the hypothetical toy store 2 is out of stock or has only one left, and optionally includes a time indicator for the information 554C1, 554D1 indicating the availability of the popular Christmas toy (e.g., "12 / 1 / 2020 at 9:00:30 AM", "this morning", etc.). Above (e.g., regarding...) Figure 1 and 4D The search results are updated in more detail based on information received during the automated phone call.

[0113] Furthermore, other automated assistants associated with other users in the same geographic area can utilize information obtained by the automated assistant during automated phone calls to perform further actions. For example, if a user's neighbor provides user input to the corresponding automated assistant at 10:00 AM to retrieve popular Christmas toys on their behalf, the neighbor's corresponding automated assistant can analyze the updated search results and remove imaginary toy store 1 from any identified groups, since popular Christmas toys were unavailable at imaginary toy store 1 at 9:00 AM. The neighbor's corresponding automated assistant can then initiate an automated phone call with imaginary toy store 2 to see if any are still available. By utilizing the information obtained by the automated assistant of computing device 510, other automated assistants can conserve computing resources on their respective computing devices and the computing devices associated with the entity.

[0114] Although Figures 5A to 5D In this document, actions are described as actions performed by a user of computing device 510 on items of interest, but it should be understood that this is merely illustrative and not intended to be limiting. For example, now refer to Figures 6A to 6C The accompanying non-limiting example describes how an automated phone call is initiated on behalf of a user with various entities, and how notifications associated with each of these entities are rendered based on the results of the automated phone call. However, with... Figures 5A to 5D In contrast, this article describes the availability of requested entities within a given time period. Figures 6A to 6C Similar to Figures 5A to 5D , Figures 6A to 6C Each of the above describes a computing device 610 having a graphical user interface 680 associated with a user request that is initiated and / or executed by an automated telephone call on behalf of the user of the computing device 610. An automated assistant (e.g., [missing information]) is associated with the user of the computing device 610. Figure 1 One or more aspects of the automation assistant 115 can be implemented locally on computing device 610 and / or in a distributed manner (e.g., via... Figure 1 The network 190) is implemented on other computing devices that communicate with computing device 610 via the network. For simplicity, Figures 6A to 6CThe operations described in this article are performed by an automation assistant (e.g., using...). Figure 1 The automatic request system 180) is executed. Although Figures 6A to 6C The computing device 610 is depicted as a mobile phone, but it should be understood that this is not intended to be limiting. The computing device 610 can be, for example, a server, a standalone speaker, a speaker connected to a graphical user interface, a laptop computer, a desktop computer, and / or any other client device capable of making telephone calls and running automated assistants.

[0115] Figures 6A to 6C The graphical user interface 680 further includes: a text response interface element 684 that the user can select to generate user input via a virtual keyboard or other touch and / or typing input; and a voice response interface element 685 that the user can select to generate user input via a microphone of the computing device 610. In some embodiments, the user can generate user input via the microphone without selecting the voice response interface element 685. For example, actively monitoring audible user input via the microphone may occur to avoid the need for the user to select the voice response interface element 685. In some and / or other embodiments of those embodiments, the voice response interface element 685 may be omitted. Furthermore, in some embodiments, the text response interface element 684 may be additionally and / or alternatively omitted (e.g., the user can provide only audible user input). Figures 6A to 6C The graphical user interface 680 of D further includes system interface elements 681, 682, 683, with which the user can interact to cause the computing device 610 to perform one or more actions.

[0116] In various implementations, an automated telephone call can be initiated and executed in response to receiving user input that directs to and requests an automation assistant to initiate and execute the automated telephone call. The user input can be verbal, touch, and / or typed input that includes instructions to initiate and execute the automated telephone call. Furthermore, the automation assistant can perform actions included in the request on behalf of a given user of computing device 610 during the automated telephone call. Figure 6AAs shown, the graphical user interface 680 includes a dialogue between an automation assistant and a user of the computing device 610 (e.g., Jane Doe). The dialogue includes initial user input 656A1 (e.g., typed or spoken) including a request (e.g., “Can you find a plumber...this afternoon between 2:00 PM and 4:00 PM”). The automation assistant can provide a response 652A1 (e.g., text or synthesized speech) instructing the automation assistant to “call some local plumbers to see if anyone is available”.

[0117] In some implementations, and as such Figure 6A As shown, a call details interface 670 can be rendered at computing device 610 in response to receiving user input 656A1. In some versions of those embodiments, the call details interface 670 may be rendered at computing device 610 as part of a graphical user interface 680. In some other versions of those embodiments, the call details interface 670 may be an overlay interface separate from the graphical user interface 680 and may include interface element 686 that allows the user to expand the call details interface 670 to display additional call details (e.g., by swiping up on interface element 686) and / or close the call details interface 670 (e.g., by swiping down on interface element 686). In other embodiments, the call details interface 670 may be accessed and closed at any time using interface element 686. Although the call details interface 670 is depicted at the bottom of the graphical user interface 680, it should be understood that this is for illustrative purposes and not intended to be limiting. For example, the call details interface 670 may be rendered at the top of the graphical user interface 680, on the side of the graphical user interface 680, or as a completely separate interface from the graphical user interface 680.

[0118] In various implementations, the call details interface 670 may include multiple selectable text segments (or graphical elements). In some implementations, the segments may be selectable, such that when a given segment is selected, the computing device 610 can perform a corresponding action. For example... Figure 6AAs shown, the call details interface 670 includes a first selectable text field 671 for "Automatic Phone Call" and a second selectable text field 672 for "Regular Call". Furthermore, when selected, the first selectable text field 671 can provide an instruction to the automation assistant to initiate an automated phone call, and when selected, the second selectable text field 672 can cause the automation assistant to initiate a regular phone call. It is worth noting that in some versions of those embodiments, the selectable text fields may include associated information for the automated phone call. For example, the first selectable text segment 671 of "Automatic Telephone Call" may include: a first portion 671A of information associated with "Service: Plumber" as associated with the plumbing service included in user input 656A1; a second portion 671B of information associated with "Time: 2:00 PM – 4:00 PM" as associated with the time during which user input of the plumbing service included in 656A1 is requested during the automatic telephone call; and a third portion 671C of information associated with "Date: 6 / 11 / 2020" as associated with the date during which user input of the plumbing service included in 656A1 is requested during the automatic telephone call. By displaying portions 671A, 671B, and 671C of the information associated with the automatic telephone call, the user of computing device 510 can be provided with information to modify the requested service, the requested time, the requested date, and / or other information related to the service requested by the user of computing device 610. Although this document references Figures 6A to 6C The described automated phone call is initiated based on user input directed to the automation assistant, but it should be understood that this is for illustrative purposes only and not intended to be limiting. For example, automated phone calls can be initiated from various software applications accessible at computing device 610 (e.g., browser-based applications, contact applications, email applications, text or SMS messaging applications, and / or other software applications), and if the automated phone call is initiated using verbal input, it can be initiated from the main screen of computing device 610, from the lock screen of computing device 610, and / or from other states of computing device 610.

[0119] For example, suppose that user input is detected at computing device 610 to initiate and execute an automated telephone call requesting the availability of a pipeline service for a given time and date (e.g., as indicated by information portions 671A, 671B, 671C). In various implementations, the automation assistant can identify each database associated with a "pipeline service" (e.g., ...) based on the user input indicating the "pipeline service". Figure 1The entity database 153A is a group of entities. In some implementations, the automation assistant may limit the entities included in a group to those stored in association with the pipeline service and within a threshold distance of the location of computing device 610 (e.g., determined based on GPS data generated by the GPS sensors of computing device 610). For example, the automation assistant may initially include all entities in the group that are stored in association with the pipeline service and within 10 miles of the location of computing device 610. If not, the automation assistant may increase the threshold distance to include all entities within 20 miles of the location of computing device 610, and so on. In some additional and / or alternative implementations, the automation assistant may continue to identify entities to include in the group until a threshold number of entities are identified for inclusion in the group. For example, the automation assistant may continuously increase the threshold distance to identify entities to be included in the group until five entities are included in the group. If the number of entities included in the group exceeds the threshold number, entities that meet one or more criteria may be included in the group (e.g., as described in more detail above with respect to entity recognition engine 153). It is worth noting that even if user input may involve initiating an automated phone call with a specific entity or without specifying an entity, groups of entities associated with popular Christmas toys can still be identified.

[0120] An automated assistant can initiate and execute automated phone calls to one or more entities within a group until the request is fulfilled. In some implementations, the automated assistant can respond to receiving... Figure 6A The user inputs 656A1 to initiate a first automated telephone call with the first entity in the group, and determines the call initiation time for initiating additional automated telephone calls with the corresponding additional entities in the group. As described above (e.g., regarding...). Figure 1 The request execution engine 154 and Figure 5A In more detail, the call initiation time of each additional automated phone call can be a static threshold duration determined based on historical call data associated with the corresponding entity of the group (e.g., stored in historical call database 154A), and dynamically modified based on the progress of the automated phone call. Furthermore, the automation assistant can cause corresponding notifications of the results of the automated phone call, prompts to the user of computing device 510, and / or other information related to the automated phone call (e.g., audibly via the speakerphone of computing device 510 and / or visually via the graphical user interface 580 of computing device 510) to be rendered at computing device 510.

[0121] For example, see specific references Figure 6B Suppose that user input is detected at computing device 510 to initiate an automated telephone call to request the availability of pipeline services (e.g., pointing to...). Figure 6AThe first selectable graphical element 671 is touch input or verbal input corresponding to user input 656A1. Further assuming the automation assistant recognizes at least Example Plumber 1 and Example Plumber 2 as entities in a group associated with plumbing services. The automation assistant can initiate an automated phone call with Example Plumber 1 and request a status update regarding the availability of plumbing services as an action to be performed based on user input. For example, the automation assistant can cause synthesized speech 652B1 to be rendered at a computing device associated with Example Plumber 1 (e.g., “Are you available to fix a burst pipe in a basement between 2:00 PM and 4:00 PM today?”), and can process audio data corresponding to information 654B1 received in response to synthesized speech 652B1 (e.g., “No, but I am available today at 5:00 PM”).

[0122] Furthermore, in response to receiving information 654B1, the automation assistant can cause further synthesized speech 652B2 to be rendered on the computing device associated with the representative of the example plumber 1 (e.g., “Let me check with Jane Doe to see if 5:00 PM works for her”). Based on the outcome of the dialogue, the automation assistant can cause notification 679B to be visually rendered on the graphical user interface (e.g., via notification interface 690, which can replace call details interface 670 and can be expanded or collapsed using interface elements 686 similar to call details interface 670). Notably, notification 679B may include an indication that the example plumber 1 is unavailable during a given time period (e.g., between 2:00 PM and 4:00 PM) but available during a later time period (e.g., available at 5:00 PM), and may also include a prompt (e.g., “Does that work for you?”). Furthermore, this prompt may cause selectable graphical elements in response to the prompt to be rendered in notification interface 590. Selectable graphical elements may include, for example: a first selectable graphical element 679B1 for authorizing a reservation for 5:00 PM; a second selectable graphical element 679B2 for declining a reservation for 5:00 PM; and a third selectable element 679B3 for the user to add an additional automated phone call. In various embodiments, if the user of computing device 610 selects the first selectable graphical element 679B1 (or otherwise confirms the 5:00 PM time), the automation assistant may further provide the example plumber 1 representative with name, contact, and address information. However, if the user does not respond to the prompt within a threshold duration (e.g., 30 seconds, 45 seconds, and / or other durations), the automation assistant may proactively terminate the automated phone call with the example plumber 1 representative to avoid wasting further computing resources. More specifically, the automation assistant may cause further synthesized speech to be rendered on the computing device of the example plumber 1 representative to indicate the user's response to the prompt (or lack thereof), and the automation assistant may optionally cause further notifications to be rendered via notification interface 590.

[0123] During an automated phone call with Example Plumber 1, the automation assistant (or a separate instance thereof) may also initiate an additional automated phone call with Example Plumber 2 based on a determined call initiation time associated with Example Plumber 2. In some implementations, the call initiation time associated with Example Plumber 2 may be zero, allowing the automation assistant to perform automated phone calls with Example Plumber 1 and Example Plumber 2 in parallel (e.g., as described above regarding initiation and execution). Figures 4A to 4C(Similar to the first automated phone call described in the example). In other embodiments, additional automated phone calls can be interleaved so that the automated assistant (and / or its individual instances) typically engages with only one entity at a time. In some versions of those embodiments, the call initiation time associated with example plumber 2 can be a set duration associated with each entity in the group, such that each additional automated phone call is initiated 30 seconds after each other. In other versions of those embodiments, the call initiation time associated with example plumber 2 can be based on historical call data associated with example plumber 1 and example plumber 2. In some additional and / or alternative versions of those embodiments, the call initiation time associated with example plumber 2 can be dynamic and based on the progress of the conversation between the automated assistant and the representative of example plumber 1, as described above (e.g., regarding...). Figure 5B (More detailed description)

[0124] For example, see specific references Figure 6C Suppose that user input is detected at computing device 610 to initiate an automated telephone call to book plumbing services on behalf of the user of computing device 610. Further assumptions are made regarding the outcome of the automated telephone call represented by example plumber 1, based on... Figure 6B The dialogue described indicates that Example Plumber 2 is unavailable for a given time period. It also assumes that an additional automated call with Example Plumber 2 is initiated 15 seconds after the automated call with Example Plumber 1. Similar to the reference above. Figure 6B The automated assistant can render the synthesized speech 652C1 on the computing device associated with the example plumber 2 (e.g., “Are you available...between 2:00 PM and 4:00 PM today?”), and can process audio data corresponding to the information 554C1 received in response to the synthesized speech (e.g., “Yes, we are available at 3:00 PM, what’s the address?”).

[0125] Furthermore, the automation assistant can cause further synthesized speech 652C2 to be rendered at the computing device associated with the example plumber 2 (e.g., “Jan Doe’s address is 401S.4th St.”, where Jane Doe is the user of computing device 610). Based on the outcome of the dialogue, the automation assistant can cause notification 679C to be visually rendered at the graphical user interface (e.g., via notification interface 690, which can replace call details interface 670 and can be expanded or collapsed using interface elements 686 similar to call details interface 670). Notably, notification 679C can indicate that example plumber 2 is available for a given time period (e.g., 3:00 PM within the desired time range of 2:00 PM to 4:00 PM). Additionally, the automation assistant can reserve time for the user of computing device 610 and respond to... Figure 6B The prompt is rendered before any further user input is received. Figure 6C Notification 679C. In this example, the automation assistant can automatically turn off... Figure 6B The automation assistant can also request available time slots from various entities and reserve a given time slot for the user of computing device 610.

[0126] In various implementations, and although not in Figures 4A to 4C As shown in 5B to 5C, 6B, or 6C, however, after initiating each corresponding automated phone call, the automated assistant may prompt the representative associated with the entity in that group to consent to participate in the conversation with the automated assistant. For example, see reference... Figure 4A The automated assistant can render the synthesized speech "Do you consent to engaging in a conversation with an automated assistant?" at the computing device represented by Example Store 1 before rendering the synthesized speech 452A1. If the automated assistant receives a message in response to a prompt instructing Example Store 1 to agree to participate in the conversation, the automated assistant can then use the above-mentioned... Figure 4A The conversation continues in the manner described. Furthermore, since the Sample Store 1 representative has already agreed to participate in the conversation, the automated assistant may not request the Sample Store 1 representative's consent to participate in the conversation during the second automated call. However, if the automated assistant does not receive a message in response to a prompt instructing the Sample Store 1 representative to consent to participate in the conversation, the automated assistant may terminate the first automated call with the Sample Store 1 representative.

[0127] Figure 7This is a block diagram of an example computing device 710 that can be optionally used to perform one or more aspects of the techniques described herein. In some implementations, one or more of a client device, a cloud-based automation assistant component, and / or other components may include one or more components of the example computing device 710.

[0128] Computing device 710 typically includes at least one processor 714, which communicates with multiple peripheral devices via a bus subsystem 712. These peripheral devices may include: a storage subsystem 724, including, for example, a memory subsystem 727 and a file storage subsystem 726; a user interface output device 720; a user interface input device 722; and a network interface subsystem 716. The input and output devices allow users to interact with computing device 710. The network interface subsystem 716 provides an interface to an external network and couples to corresponding interface devices in other computing devices.

[0129] User interface input device 722 may include a keyboard, pointing devices such as a mouse, trackball, touchpad, or graphics tablet, a scanner, a touchscreen integrated into a display, audio input devices such as a voice recognition system, a microphone, and / or other types of input devices. Generally, the term "input device" is used to encompass all possible types of devices and methods for inputting information into computing device 710 or a communication network.

[0130] User interface output device 720 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display, such as via an audio output device. Generally, the term "output device" is used to encompass all possible types of devices and methods for outputting information from computing device 710 to a user or another machine or computing device.

[0131] Storage subsystem 724 stores programs and data structures that provide the functionality of some or all of the modules described herein. For example, storage subsystem 724 may include selected aspects for performing the methods disclosed herein and implementations. Figure 1 The logic of the various components described in the text.

[0132] These software modules are typically executed by processor 714 alone or in combination with other processors. Memory 725 used in storage subsystem 724 may include multiple memories, including main random access memory (RAM) 730 for storing instructions and data during program execution and read-only memory (ROM) 732 for storing fixed instructions. File storage subsystem 726 can provide permanent storage for program and data files and may include hard disk drives, floppy disk drives with associated removable media, CD-ROM drives, optical drives, or removable media cartridges. Modules implementing the functionality of certain embodiments may be stored by file storage subsystem 726 in storage subsystem 724 or in other machines accessible to processor 714.

[0133] The bus subsystem 712 provides a mechanism for enabling various components and subsystems of the computing device 710 to communicate with each other as intended. Although the bus subsystem 712 is schematically shown as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0134] The computing device 710 can be of various types, including workstations, servers, computing clusters, blade servers, server farms, or any other data processing system or computing device. Due to the constantly evolving nature of computers and networks, Figure 7 The description of the computing device 710 depicted herein is intended only as a specific example for illustrating some implementations. Many other configurations of the computing device 710 may have... Figure 7 The computing device depicted in the text has more or fewer components.

[0135] In situations where the systems described herein collect or otherwise monitor personal information about users, or may use personal and / or monitored information, users may be given the opportunity to control whether a program or feature collects user information (e.g., information about a user's social networks, social behaviors or activities, occupation, user preferences, or the user's current geographic location), or to control whether and / or how content that may be more relevant to the user is received from a content server. Furthermore, some data may be processed in one or more ways before it is stored or used to remove personally identifiable information. For example, a user's identity may be processed so that the user's personally identifiable information cannot be determined, or the user's geographic location may be generalized (e.g., down to the city, zip code, or state level) where geographic location information is available, making it impossible to determine the user's specific geographic location. Therefore, users can control how information about themselves is collected and / or used.

[0136] In some embodiments, a method implemented by one or more processors is provided, the method comprising: identifying an item of interest; identifying a group of entities associated with the item of interest; and initiating, by an automation assistant, the execution of a corresponding automated telephone call with each entity in the group to render one or more corresponding instances of synthesized speech requesting a corresponding status update regarding the item of interest. The method further comprises: during the execution of the corresponding automated telephone call with a given entity in the group: requesting the corresponding status update regarding the item of interest from a representative associated with the given entity by rendering one or more corresponding instances of synthesized speech; and receiving, in response to the request for the corresponding status update, information about the item of interest from the representative associated with the given entity. The method further comprises: determining, based on the received information about the item of interest, a time delay before initiating the execution of a subsequent corresponding automated telephone call with the given entity to request a further corresponding status update regarding the item of interest; and, in response to the elapsed time delay, initiating the execution of the subsequent corresponding automated telephone call with the given entity.

[0137] These and other embodiments of the technology disclosed herein may optionally include one or more of the following features.

[0138] In some implementations, the received information about the item of interest indicates the availability of the item of interest at the given entity and the number of items of interest available at the given entity.

[0139] In some versions of those implementations, determining the time delay based on received information about the item of interest includes: determining whether the available quantity of the item of interest at the given entity meets a quantity threshold; and assigning a first time period as the time delay in response to determining that the available quantity of the item of interest at the given entity meets the quantity threshold. In some further versions of those implementations, the method further includes: assigning a second time period as the time delay in response to determining that the available quantity of the item of interest at the given entity does not meet the quantity threshold. The second time period has a shorter duration than the first time period and therefore elapses before the first time period. In other versions of those implementations, the second time period elapses on the same day the automatic call is initiated, and the first time period elapses during the day following the day the automatic call is initiated.

[0140] In some versions of those implementations, the method further includes: updating search results associated with the given entity to indicate one or more of the following: the availability of the item of interest at the given entity, or the number of items of interest available at the given entity. In some further versions of those implementations, the method further includes: analyzing updated search results associated with the given entity and previously updated search results associated with the given entity; and based on the analysis of the updated search results and the previously updated search results: predicting the future availability of the item of interest at the given entity for a future time, or predicting the future number of items of interest available at the given entity for a future time.

[0141] In some embodiments, the method further includes: during the execution of the corresponding automated telephone call with the given entity: verifying whether the given entity is associated with the item of interest; and wherein determining the time delay is in response to determining that the given entity is associated with the item of interest based on the verification. In some versions of those embodiments, the method further includes: in response to determining that the given entity is not associated with the item of interest based on the verification: avoiding determining the time delay; and removing the given entity from the group of entities associated with the item of interest. Removing the given entity from the group of entities associated with the item of interest prevents the initiation of any subsequent automated calls with the given entity related to the item of interest.

[0142] In some implementations, the representative associated with the given entity is one of the following: a human representative or an additional automated assistant associated with the given entity.

[0143] In some implementations, initiating the execution of the corresponding automated telephone call with each entity in the group to request the corresponding status update regarding the item of interest includes: initiating the execution of the corresponding automated telephone call with the given entity; and simultaneously initiating the execution of at least additional corresponding automated telephone calls with additional entities in the group. In some versions of those implementations, the method further includes: during the execution of the additional corresponding automated telephone call with the additional entity: requesting the corresponding status update regarding the item of interest from an additional representative associated with the additional entity by rendering one or more corresponding instances of synthesized speech; and in response to requesting the corresponding status update, receiving additional information regarding the item of interest from the additional representative associated with the additional entity. The method further includes: determining an additional time delay based on the received additional information regarding the item of interest before initiating the execution of subsequent additional corresponding automated telephone calls with the additional entity to request the further corresponding status update regarding the item of interest. A first time period is assigned to the time delay, and a different second time period is assigned to the additional time delay. The method further includes: in response to the elapsed additional time delay, the automation assistant initiates the execution of the subsequent additional corresponding automated telephone call with the additional entity.

[0144] In some implementations, identifying the group of entities associated with the item of interest includes accessing one or more databases to identify entities stored in association with the item of interest as the group of entities. In some versions of those implementations, each entity in the identified group is located within a specific geographic region.

[0145] In some embodiments, the method of requesting a state update about the item of interest from the representative associated with the given entity further includes: generating synthetic speech audio data of one or more corresponding instances of synthetic speech that captures the corresponding state update about the item of interest; and rendering the one or more corresponding instances of synthetic speech includes rendering the synthetic speech at an additional computing device of the representative associated with the given entity.

[0146] In some embodiments, a method implemented by one or more processors is provided, the method comprising: receiving from a user's computing device a request to initiate the execution of an automated telephone call on behalf of the user; identifying, based on the request, an action to be performed during the automated telephone call; identifying, based on the action to be performed during the automated call, a group of entities, each capable of satisfying the request; and an automation assistant initiating the execution of the automated telephone call with a given entity in the group of entities. The method further comprises: during the execution of the automated telephone call with the given entity: engaging in a conversation with a representative associated with the given entity to perform the action by rendering one or more corresponding instances of synthesized speech associated with the request; and the automation assistant initiating the execution of additional automated telephone calls with additional entities in the group of entities. The method further comprises: generating a notification including the result of the conversation with the representative; and causing the notification to be rendered via the user's computing device.

[0147] These and other embodiments of the technology disclosed herein may optionally include one or more of the following features.

[0148] In some embodiments, the method further includes: determining, based on the outcome of the conversation during the execution of the automated telephone call, that the given entity will fulfill the request; and terminating the execution of the additional automated telephone call with the additional entity in response to determining that the given entity will fulfill the request.

[0149] In some embodiments, the method further includes: after initiating the execution of the automated telephone call with the given entity, determining that the representative associated with the given entity has held the automated telephone call; determining that the representative associated with the given entity has resumed the automated telephone call; and initiating the execution of the additional automated telephone call with the additional entity is in response to determining that the representative associated with the given entity has resumed the automated telephone call.

[0150] In some embodiments, the method further includes: during the execution of a corresponding automated telephone call with the given entity: determining a call initiation time for initiating the execution of the additional automated telephone call with the additional entity based on historical call data associated with the entity; and initiating the execution of the additional automated telephone call with the additional entity in response to determining that the current time matches the call initiation time. In some versions of those embodiments, the historical call data associated with the additional entity includes one or more of the following: an indication of the entity being associated with an interactive voice response system and the duration of navigating the interactive voice response system; an indication of the duration of a previous automated telephone call with the entity; or an indication of the duration of an initial hold time during a previous automated telephone call with the entity.

[0151] In some implementations, the action to be performed during the automated telephone call includes requesting a status update regarding an item of interest associated with the entity of the group. In some implementations, engaging in the conversation with the representative associated with the given entity to perform the action includes: requesting the status update regarding the item of interest from the representative associated with the given entity; and receiving information about the item of interest from the representative associated with the given entity in response to the request for the status update.

[0152] In some further versions of those implementations, the received information about the item of interest indicates the availability of the item of interest at the given entity, and the notification including the result of the dialogue indicates that the item of interest is available at the given entity.

[0153] In other versions of those implementations, the method further includes: in response to determining that received information about the item of interest indicates its availability; requesting the representative associated with the given entity to hold the item of interest for the user at the given entity, and the notification including the result of the dialogue further instructing the user to hold the item of interest at the given entity. In other versions of these implementations, the method further includes: in response to determining that received information about the item of interest indicates its availability; submitting additional information to the representative associated with the given entity and on behalf of the user to obtain the item of interest. In a further version of these implementations, the notification including the result of the dialogue also requests authorization from the user to submit the additional information submitted to obtain the item of interest. In a further version of these implementations, the notification including the result of the dialogue further instructs the submission of the additional information to obtain the item of interest.

[0154] In other versions of those implementations, the notification further requests the user to join the automated phone call. In other versions of those implementations, the item of interest is specified in the request to initiate the execution of the automated phone call on behalf of the user. In other versions of those implementations, the method further includes: terminating the execution of the additional automated phone call with the additional entity in response to determining that the item of interest is available at the given entity.

[0155] In some further versions of those implementations, the received information about the item of interest indicates that the item of interest is unavailable at the given entity, and the notification, including the result of the conversation, indicates that the item of interest is unavailable at the given entity. In other versions of those implementations, the method further includes: terminating the execution of the automated telephone call with the given entity in response to determining that the item of interest is unavailable at the given entity.

[0156] In some implementations, the action to be performed during the automated telephone call includes requesting the availability of the entities in the group during a given time period. In some versions of those implementations, engaging in the dialogue with the representative associated with the given entity to perform the action includes: requesting the availability of the given entity from the representative associated with the given entity during the given time period; and receiving information from the representative associated with the given entity regarding the availability of the given entity during the given time period.

[0157] In some further versions of those embodiments, the received information indicates that the given entity is available during the given time period, and the notification including the result of the conversation indicates that the given entity is available during the given time period. In other versions of those embodiments, the method further includes: in response to determining that the given entity is available during the given time period: requesting the representative associated with the given entity to maintain availability of the given entity for the user for the given time period, and the notification including the result of the conversation further indicates that availability of the given entity for the user is being maintained for the user for the given time period. In other versions of those embodiments, the notification further requests the user to join the automated telephone call. In other versions of those embodiments, the method further includes: in response to determining that the given entity is available during the given time period, terminating the execution of the additional automated telephone call with the additional entity.

[0158] In some further versions of those implementations, the received information indicates that the given entity is unavailable during the given time period, and the notification, including the result of the conversation, indicates that the given entity is unavailable during the given time period. In other versions of those implementations, the method further includes: terminating the execution of the automated telephone call with the given entity in response to determining that the given entity is unavailable during the given time period.

[0159] In some implementations, identifying the group of entities, each capable of fulfilling the request, includes: identifying the location of the user's computing device; determining whether the entity is within a threshold distance of the location of the user's computing device; determining whether the entity can fulfill the request; and, in response to determining that the entity is within the threshold distance of the location of the computing device and in response to determining that the entity can fulfill the request, including the entity in the group of entities. In some versions of those implementations, the method further includes: after receiving the request to initiate the execution of the automated telephone call on behalf of the user, receiving an additional request from an additional user's additional computing device to initiate the execution of a separate automated telephone call on behalf of the additional user. The additional request from the additional user is the same as the request from the user. The method further includes: avoiding initiating the separate automated telephone call based on the result of the conversation with the representative associated with the given entity or the result of an additional conversation with an additional representative associated with the additional entity; generating an additional notification including the result of the conversation with the representative or the result of the additional conversation with the additional representative; and causing the additional notification to be rendered via the user's additional computing device.

[0160] In some implementations, receiving the request to initiate the automated telephone call on behalf of the user includes one of the following: receiving touch or typing input on the user's computing device via a search interface or automation assistant interface; or receiving verbal input via one or more microphones on the user's computing device.

[0161] In some implementations, the representative associated with the given entity is one of the following: a human representative or an additional automated assistant associated with the given entity.

[0162] Furthermore, some embodiments include one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU)) of one or more computing devices, wherein the one or more processors are operable to execute instructions stored in associated memory, and wherein the instructions are configured to cause execution of any of the methods described above. Some embodiments also include one or more non-transitory computer-readable storage media storing computer instructions executable by one or more processors to perform any of the methods described above. Some embodiments also include a computer program product comprising instructions executable by one or more processors to perform any of the methods described above.

[0163] It should be understood that all combinations of the foregoing concepts and the additional concepts described in more detail herein are contemplated as part of the subject matter disclosed herein. For example, all combinations of the claimed concepts appearing at the beginning of this disclosure are contemplated as part of the subject matter disclosed herein.

Claims

1. A method implemented by one or more processors, comprising: Receive a request from the user's computing device to initiate an automated telephone call on behalf of the user; Based on the request, identify the action to be performed during the automated telephone call, the action being performed by rendering one or more corresponding instances of synthesized speech associated with the request; Based on the actions to be performed during the automated telephone call, identify groups of entities that can each satisfy the request; The automated telephone call initiated by the automated assistant to the first entity in the group of the entities is executed during the first automated telephone call. During the execution of the first automated telephone call with the first entity: Using synthesized speech to participate in a dialogue with a representative associated with the first entity to perform the action; and The automated assistant iteratively initiates the execution of corresponding additional automated phone calls to the corresponding additional entities in the group of the entity after the corresponding call initiation time, until the request is satisfied. The action will be performed during each corresponding automated phone call. The initiation time of each corresponding call is determined to stagger the initiation of each automated telephone call; Generate a notification including the results of the dialogue with the representative; as well as This causes the notification to be rendered via the user's computing device.

2. The method according to claim 1, further comprising: The first entity will satisfy the request based on the outcome of the conversation during the execution of the first automated telephone call; as well as In response to determining that the first entity will fulfill the request, each corresponding additional automated telephone call is terminated.

3. The method according to claim 1, further comprising: After initiating the execution of the first automated telephone call with the first entity, it is determined that the representative associated with the first entity has held the first automated telephone call. It was determined that the representative associated with the first entity had resumed the first automated telephone call; as well as The operation of iteratively initiating the execution of the corresponding additional automated telephone call is in response to determining that the representative associated with the first entity has resumed the first automated telephone call.

4. The method according to claim 1, further comprising: During the execution of the first automated telephone call with the first entity: The corresponding call initiation time for initiating the execution of the corresponding additional automated telephone call to the additional entity is determined based on historical call data associated with each additional entity.

5. The method according to claim 4, wherein, The historical call data associated with each additional entity includes one or more of the following: The additional entity is associated with the interactive voice response system and the duration of navigation of the interactive voice response system; An indication of the duration of a previous automated telephone call by the attached entity; or An indication of the duration of the initial hold time during a previous automated telephone call with the attached entity.

6. The method according to claim 1, wherein, The actions to be performed during the automated telephone call include requesting status updates regarding items of interest associated with the entities in the group.

7. The method according to claim 6, wherein, Participating in the dialogue with the representative associated with the first entity to perform the action includes: The representative associated with the first entity requests a status update regarding the item of interest; and In response to the request for the status update, information about the item of interest is received from the representative associated with the first entity.

8. The method according to claim 7, wherein, The received information about the item of interest indicates the availability of the item of interest at the first entity, and wherein the notification, which includes the result of the dialogue, indicates that the item of interest is available at the first entity.

9. The method of claim 8, further comprising: In response to determining the received information about the item of interest, the availability of the item of interest is indicated: Submit additional information to the representative associated with the first entity and on behalf of the user to obtain the items of interest.

10. The method according to claim 9, wherein, The notification, which includes the result of the dialogue, also includes the user's request for authorization to submit additional information in order to obtain the items of interest.

11. The method according to claim 9, wherein, The notification, including the result of the dialogue, further instructs the submission of the additional information to obtain the item of interest.

12. The method according to claim 8, wherein, The notification further requests the user to join the first automated telephone call.

13. The method according to claim 7, wherein, The received information about the item of interest indicates that the item of interest is unavailable at the first entity, and wherein the notification, which includes the result of the dialogue, indicates that the item of interest is unavailable at the first entity.

14. The method of claim 13, further comprising: In response to determining that the item of interest is unavailable at the first entity, the execution of the first automated telephone call to the first entity is terminated.

15. The method according to claim 1, wherein, The actions to be performed during the automated telephone call include requesting the availability of the entities in the group during a given time period.

16. The method according to claim 15, wherein, Participating in the dialogue with the representative associated with the first entity to perform the action includes: The representative associated with the first entity requests the availability of the first entity during the given time period; and Receive information about the availability of the first entity during the given time period from the representative associated with the first entity.

17. The method according to claim 1, wherein, The group that identifies each of the entities capable of satisfying the request includes: Identify the location of the user's computing device; Determine whether the entity is within a threshold distance of the location of the user's computing device; Determine whether the entity can satisfy the request; and In response to determining that the entity is within the threshold distance of the location on the computing device and in response to determining that the entity can satisfy the request, the entity is included in the group of entities.

18. At least one computing device, comprising: At least one processor; as well as At least one memory storing instructions, which, when executed, cause the at least one processor to perform the method according to any one of claims 1 to 17.

19. A method implemented by one or more processors, comprising: Identify items of interest by analyzing query activity from multiple users and / or by detecting user input at the client device; Identifying a group of entities associated with the project of interest, wherein identifying a group of entities associated with the project of interest includes accessing one or more databases to identify entities stored in association with the project of interest as a group of entities; An automated assistant initiates the execution of a corresponding automated phone call to each entity in the group to render one or more corresponding instances, wherein at least a subset of the corresponding automated phone calls to a subset of the entities in the group are initiated and executed in parallel. During the execution of the corresponding automated telephone call to a given entity within the group: By rendering one or more corresponding instances of the synthesized speech, a corresponding state update regarding the item of interest is requested from a representative associated with the given entity; as well as In response to a request for the corresponding status update, information about the item of interest is received from the representative associated with the given entity, wherein the received information about the item of interest indicates the availability of the item of interest at the given entity and the number of items of interest available at the given entity; Based on the received information about the item of interest, a time delay is determined before initiating a subsequent automated telephone call to the given entity to request further status updates regarding the item of interest; and In response to the elapsed time delay, the automated assistant initiates the execution of the subsequent corresponding automated telephone call with the given entity. Determining the time delay based on received information about the project of interest includes: Determine whether the available quantity of the items of interest at the given entity meets a quantity threshold; and In response to determining that the available quantity of the item of interest at the given entity satisfies the quantity threshold, a first time period is assigned as the time delay.

20. The method of claim 19, further comprising: In response to determining that the available number of items of interest at the given entity does not meet the quantity threshold, a second time period is assigned as the time delay, wherein the second time period is a shorter duration than the first time period and therefore elapses before the first time period.

21. The method according to claim 20, wherein, The second time period elapses on the same day the automatic call is initiated, and wherein the first time period elapses on the day following the same day the automatic call is initiated.

22. The method of claim 19, further comprising: Update the search results associated with the given entity to indicate one or more of the following: the availability of the item of interest at the given entity, or the number of items of interest available at the given entity.

23. The method of claim 22, further comprising: Analyze the updated search results associated with the given entity and the previously updated search results associated with the given entity; as well as Based on the analysis of the updated search results and the previously updated search results: To predict the future availability of the project of interest at the given entity for future time, or Predict the future availability of the items of interest at the given entity for the future time.

24. The method of claim 19, further comprising: During the execution of the corresponding automated telephone call with the given entity: Verify whether the given entity is associated with the item of interest; as well as The determination of the time delay is in response to determining that the given entity is associated with the item of interest based on the verification.

25. The method of claim 24, further comprising: In response to determining, based on the verification, that the given entity is not associated with the item of interest: Avoid determining the aforementioned time delay; as well as Remove the given entity from the group of entities associated with the project of interest, wherein removing the given entity from the group of entities associated with the project of interest prevents the initiation of any subsequent automated calls to the given entity related to the project of interest.

26. The method according to claim 19, wherein, The representative associated with the given entity is one of the following: a human representative or an additional automated assistant associated with the given entity.

27. The method according to claim 19, wherein, Initiating the execution of the corresponding automated telephone call to each entity in the group to request the corresponding status update regarding the item of interest includes: Initiating the execution of the corresponding automated telephone call to the given entity; and Simultaneously, initiate the execution of at least additional corresponding automatic telephone calls to additional entities in the group.

28. The method of claim 27, further comprising: During the execution of the additional corresponding automatic telephone call with the additional entity: By rendering one or more corresponding instances of the synthesized speech, an additional representative associated with the additional entity requests an update on the corresponding state of the item of interest. as well as In response to the request for the corresponding status update, additional information about the item of interest is received from the additional representative associated with the additional entity; Based on the received additional information about the item of interest, an additional time delay is determined before initiating a subsequent additional corresponding automated telephone call to the additional entity to request the further corresponding status update about the item of interest. The first time period is assigned to the time delay, and Different second time periods are assigned to the additional time delay; and in response to the elapsed additional time delay, the automation assistant initiates the execution of the subsequent additional corresponding automatic telephone call to the additional entity.

29. The method according to claim 19, wherein, Each entity in the identified group is located within a specific geographic region.

30. The method according to claim 19, wherein, Requesting a status update from the representative associated with the given entity regarding the item of interest includes: Generate synthesized speech audio data, wherein the synthesized speech audio data captures one or more corresponding instances of the corresponding instances of the synthesized speech requesting an update of the corresponding state of the item of interest; and Rendering one or more of the corresponding instances of the synthesized speech includes rendering the synthesized speech at an additional computing device associated with the given entity.

31. At least one computing device, comprising: At least one processor; as well as At least one memory storing instructions, which, when executed, cause the at least one processor to perform the method according to any one of claims 19 to 30.

Images