Automated assistant calls to appropriate agents

Abstract

The present application relates to automated assistant invocation of an appropriate agent. An agent is invoked during a conversation between a user and an automated assistant. Some embodiments are directed to receiving natural language input from a user during a conversation with an automated assistant, the natural language input indicating a desire for an agent to participate, but not indicating a specific agent to participate. Those embodiments are also directed to selecting a specific agent from a plurality of available agents and transmitting an invocation request to the selected specific agent. In some embodiments, an agent selection model, such as a machine learning model, may be utilized in selecting the specific agent. The machine learning model may be trained to generate an output that indicates, for each of a plurality of available agents (and optionally for the intent of those agents), a probability that the available agent (and optionally the intent) will generate appropriate response content.

Description

[0001] Description of the case

[0002] This application is a divisional application of Chinese invention patent application No. 201780028118.7, filed on April 18, 2017. Technical Field

[0003] The present application generally relates to automated assistant invocation of appropriate agents. Background Art

[0004] Users can interact with automated assistants (also referred to as "personal assistants," "mobile assistants," etc.) via various client devices (such as smartphones, tablet computers, wearable devices, automotive systems, standalone personal assistant devices, etc.). The automated assistant receives input from the user (e.g., typed and / or spoken natural language input) and responds with responsive content (e.g., visual and / or audible natural language output). The automated assistant interacted with via the client device can be implemented via the client device itself and / or via one or more remote computing devices (e.g., computing devices in the "cloud") in network communication with the client device. Summary of the Invention

[0005] This specification is generally directed to methods, systems, and computer-readable media for invoking an agent during a conversation between a user and an automated assistant. Invoking the agent can include transmitting (e.g., using an application programming interface (API)) a call request including values ​​for call parameters (e.g., a value for an intent parameter, a value for an intent bit parameter, and / or values ​​for other parameters), and causing the agent to generate content for presentation to the user via one or more user interface output devices (e.g., via one or more of the user interface output devices utilized in the conversation with the automated assistant). Response content generated by the agent can be customized for the call parameters of the call request.

[0006] Some embodiments are directed to receiving natural language input from a user during a conversation between a person and an automated assistant, the natural language input indicating a desire for an agent to engage, but failing to indicate a specific agent to engage. For example, "book me a hotel in Chicago" indicates a desire for an agent to engage with an intent parameter of "hotel reservation" and a location parameter of "Chicago," but fails to specify a specific agent to invoke. Those embodiments are also directed to selecting a specific agent from a plurality of available agents and transmitting a call request to the selected specific agent. For example, a call request may be transmitted to the selected specific agent without transmitting the call request to any other agent among the available agents. In some of those embodiments, a specific agent and a specific intent for the specific agent are selected (e.g., when the specific agent is operable to generate response content in response to any one of a plurality of different intents).

[0007] In some embodiments of selecting a particular agent from a plurality of available agents, an agent selection model is used in selecting the particular agent. In some versions of those embodiments, the agent selection model includes at least one machine learning model, such as a deep neural network model. The machine learning model can be trained to generate an output that indicates, for each of a plurality of available agents (and optionally for the intents of those agents), a probability that the available agent (and optionally the intent) will generate appropriate response content. The generated output is based on input applied to the machine learning model, where the input is based on: the current conversation with the automated assistant, and optionally additional context values. For example, the input based on the current conversation may include various values ​​based on the most recent natural language input provided to the automated assistant in the current conversation and / or past natural language input provided in the current conversation. In addition, the optional additional context values ​​may include, for example, client device context values, such as values ​​based on: historical interactions of a user of the client device, currently rendered and / or recently rendered content on the client device, the location of the client device, the current date and / or time, and the like.

[0008] In some implementations, where the agent selection model includes at least one machine learning model, at least one of the machine learning models may be trained based on training examples based on past interactions with available agents.

[0009] As an example, multiple training instances may each be generated based on a corresponding agent request generated based on natural language input provided by a corresponding person during a conversation with an automated assistant (e.g., natural language input that fails to identify a particular agent). The agent request may be transmitted to each of a plurality of available agents (e.g., to all available agents), and responses may be received from one or more of the available agents to which the agent request was transmitted. Each of the training instances may include a training instance input based on the agent request (e.g., the corresponding natural language input and, optionally, a context value), and a training instance output based on the response. The responses may each indicate the ability of a corresponding one of the agents to parse the agent request. For example, the response from a given agent may be a binary indication (e.g., "parseable," "unparseable," or "response content," "no response content / error"), a non-binary confidence measure (e.g., "70% likely parseable"), the actual response content (or no content / error), etc. Furthermore, for example, receiving a response from a given agent may indicate that it is able to respond, while the absence of a response from the given agent may indicate that it is unable to respond. The agent request may be transmitted to the agent without actively invoking an available agent in the conversation. For example, a proxy request may be similar to a call request, but include a "non-call" flag and / or other indication that the proxy should not be immediately called. Additionally, for example, a response may additionally or alternatively be processed by the automated assistant without providing any corresponding content in the dialog.

[0010] Once trained, such a machine learning model can be used to predict a probability for each of a plurality of agents (and optionally intents) based on the current conversation (e.g., the natural language input of the current conversation and optionally contextual values)—where each of these probabilities indicates a probability (e.g., binary or non-binary) that the agent is capable of appropriately handling a call request based on the conversation. Selection of a particular agent can be based at least in part on such probabilities, and the call request is transmitted to only the particular agent.

[0011] In some embodiments, at least one of the machine learning models may be generated based on natural language input provided to the agent following an invocation of the agent. For example, natural language input provided to the agent immediately following an invocation of the agent may be stored in association with the agent. For example, the natural language input may be that provided to the agent immediately following an invocation that is a "bare" invocation. A bare invocation of an agent is an invocation of the agent based on a call request directed to the agent but that does not include a value for an intent parameter and / or does not include values ​​for any intent bit parameters. For example, in response to a natural language input of "open agent X," a bare invocation of "agent X" may occur in response to a call request sent to "agent X" that does not include any value for the intent parameter and does not include any value for any intent bit parameters. Additionally, for example, in response to a natural language input of "set reminder with agent X," a bare invocation of "agent X" may occur in response to a call request sent to "agent X" that includes a value for the intent parameter of "reminder" but does not include any value for any intent bit parameters. A selection model may be generated that includes a mapping (or other association) between natural language input and the corresponding agent (and optionally, intent) to which the natural language input is provided. In this manner, mapping is based on the initial dialogue provided by the user following a bare call to an agent, which enables the generation of an agent selection model that provides insight into which agents are most likely to be suitable for responding to various natural languages. Additional and / or alternative selection models may be utilized when selecting a particular agent. As one example, a selection model generated based on past explicit selections of agents by various users may additionally or alternatively be utilized when selecting a particular agent.

[0012] In some embodiments, various additional and / or alternative criteria are used in selecting a particular agent (and optionally an intent). As an example, an agent request (as described above) may be transmitted "live" to multiple candidate agents, and the responses from those agents analyzed in determining the particular agent to be selected for invocation. As another example, the additional and / or alternative criteria may include historical interactions of the user of the client device (e.g., how frequently a particular agent has been utilized by the user, how recently a particular agent has been utilized by the user), currently rendered and / or recently rendered content on the client device (e.g., whether the content corresponds to agent characteristics), the location of the client device, the current date and / or time, the ranking of the particular agent (e.g., ranking by user group), the popularity of the particular agent (e.g., popularity among the user group), etc. In embodiments where a machine learning model is utilized in selecting a particular agent, such criteria may be applied as input to the machine learning model and / or considered in conjunction with output generated by the machine learning model.

[0013] The various techniques described above and / or elsewhere herein enable the selection of a specific agent and can increase the likelihood that the selected specific agent will be able to appropriately handle the call request. This can mitigate the risk that the specific agent selected for invocation will be unable to execute the intent of the call request (optionally based on the values ​​of additional parameters of the call request), which can conserve various computing resources. For example, it can conserve network and / or processor resources that might otherwise be consumed by an initial failed attempt to execute the intent using an agent, and then invoke an alternative agent on another subsequent attempt to execute the intent. Furthermore, in embodiments where a specific agent is selected without prompting the user to choose between multiple available agents, it can reduce the number of "turns" required for a person to speak with the automated assistant before the call is made. This can also conserve various network and / or processor resources that would otherwise be consumed by such turns. Additionally, in embodiments where a trained machine learning model is utilized, the trained machine learning model can be utilized to determine the probability that an agent can handle a particular call—without having to consume network resources through "live" interaction with one or more agents to make such a determination. This can also conserve various network and / or processor resources that would otherwise be consumed in such live interactions.

[0014] In some cases, in response to a call to a specific agent according to the techniques disclosed herein, a human-automated assistant conversation can be at least temporarily (actually or effectively) transferred to the specific agent. For example, an output based on the specific agent's response content can be provided to the user to facilitate the conversation, and further user input can be received in response to the output. The further user input (or a transformation thereof) can be provided to the specific agent. The specific agent can utilize its own semantic engine and / or other components in generating further response content, which can be used to generate further output to facilitate the conversation. This general process can continue until, for example, the specific agent provides response content that terminates the specific agent conversation (e.g., an answer or solution rather than a prompt), additional user interface input from the user terminates the specific agent conversation (e.g., alternatively invoking a response from the automated assistant or other agent), etc.

[0015] In some cases, when a conversation is effectively transferred to a particular agent, the automated assistant can still act as an intermediary. For example, when acting as an intermediary where the user's natural language input is voice input, the automated assistant can convert the voice input to text, provide the text (and optionally annotations to the text) to the particular agent, receive response content from the particular agent, and provide output based on the particular response content for presentation to the user. Additionally, for example, when acting as an intermediary, the automated assistant can analyze the user input and / or response content of the particular agent to determine whether the conversation with the particular agent should be terminated, whether the user should be transferred to an alternative agent, whether global parameter values ​​should be updated based on the particular agent conversation, etc. In some cases, the conversation can be actually transferred to the particular agent (once transferred, without the automated assistant acting as an intermediary) and optionally transferred back to the automated assistant upon the occurrence of one or more conditions, such as termination by the particular agent (e.g., in response to completion of an intent via the particular agent).

[0016] Embodiments described herein enable an automated assistant to select an appropriate agent based on a conversation with a user and invoke the agent to implement the user's intent as indicated by the user in the conversation. These embodiments can enable a user to engage an agent via interaction with the automated assistant without requiring the user to know an "invocation phrase" to explicitly trigger the agent and / or without requiring the user to even initially know that the agent exists. Furthermore, embodiments can enable a user to utilize a common automated assistant interface (e.g., an audible / voice-based interface and / or a graphical interface) to invoke any of a plurality of fundamentally different agents that can perform actions across a plurality of fundamentally different intents. For example, a common automated assistant interface can be utilized to engage any of a plurality of agents that handles a "restaurant reservation" intent, any of a plurality of agents that handles a "purchase professional services" intent, any of a plurality of agents that handles a "tell a joke" intent, any of a plurality of agents that handles a "reminder" intent, any of a plurality of agents that handles a "purchase travel services" intent, and / or any of a plurality of agents that handles an "interactive game" intent.

[0017] As used herein, an "agent" refers to one or more computing devices and / or software that is separate from the automated assistant. In some cases, the agent may be a third-party (3P) agent, as it is managed by a party separate from the party that manages the automated assistant. The agent is configured to receive a call request from the automated assistant (e.g., over a network and / or via an API). In response to receiving the call request, the agent generates response content based on the call request and transmits the response content to configure an output based on the response content. For example, the agent may transmit the response content to the automated assistant for the automated assistant to provide output based on the response content. As another example, the agent itself may provide output. For example, a user may interact with the automated assistant via a client device (e.g., the automated assistant may be implemented on the client device and / or in network communication with the client device), and the agent may be an application installed on the client device or an application that is remotely executable from the client device but "streamable" on the client device. When the application is invoked, it may be executed by the client device and / or brought to the forefront by the client device (e.g., its content may take over the display of the client device).

[0018] Various types of input are described herein that can be provided by a user to an automated assistant and / or to an agent via a user interface input device. In some instances, the input can be free-form natural language input, such as textual input based on user interface input generated by a user via one or more user interface input devices (e.g., based on typed input provided via a physical or virtual keyboard or based on voice input provided via a microphone). As used herein, free-form input is input formulated by a user and is not limited to a set of options presented to the user for selection (e.g., not limited to a set of options presented in a drop-down menu).

[0019] In some embodiments, a method performed by one or more processors is provided and includes receiving a natural language input instance generated based on user interface input during a conversation between a person and an automated assistant. The method also includes, before invoking any agent in response to the natural language input instance: generating an agent request based on the natural language input instance; selecting a set of multiple agents from a corpus of available agents for the agent request; transmitting the agent request to each of the multiple agents in the set; receiving corresponding responses to the request from at least a subset of the multiple agents in response to the transmission; determining, based on each of the responses, a relative ability of the agent providing the response to generate response content in response to the agent request; and selecting a specific agent from the multiple agents based on at least one of the responses. The method also includes invoking a specific agent in response to the natural language input and based on selecting the specific agent. Invoking the specific agent causes the response content generated by the specific agent to be provided for presentation via one or more user interface output devices. In some embodiments, only the selected specific agent is invoked in response to receiving the natural language input.

[0020] These and other implementations of the technology disclosed herein can optionally include one or more of the following features.

[0021] In some embodiments, the method further includes: storing in one or more computer-readable media an association of the proxy request with at least one of the agents determined to be capable of responding to the proxy request; and generating an agent selection model based on the stored association between the proxy request and at least one of the agents determined to be capable of responding to the proxy request. In some of those embodiments, the method further includes, subsequent to generating the agent selection model: receiving additional natural language input in an additional human-automated assistant conversation; selecting an additional agent from the plurality of agents based on the additional natural language input and the agent selection model; and in response to the additional natural language input and based on selecting the additional agent: transmitting an additional call request to the additional agent. The additional call request will invoke the additional agent. In response to receiving the additional natural language input, the additional call request is optionally transmitted to only the selected additional agent.

[0022] In some implementations, selecting a particular agent is further based on: the volume of interaction with the particular agent by users participating in the conversation; the recency of interaction with the particular agent by users; and / or the ranking or popularity of the particular agent among the user base.

[0023] In some embodiments, a method performed by one or more processors is provided and includes, for each of a plurality of natural language input instances generated based on user interface input during a conversation between a person and an automated assistant: generating a proxy request based on the natural language input instance; selecting a set of multiple agents from a corpus of available agents for the proxy request; transmitting the proxy request to each of the plurality of agents in the set via one or more application programming interfaces; receiving a corresponding response to the request from each of the plurality of agents in response to the transmitting; and storing one or more associations between the proxy request and the responses to the proxy request in one or more computer-readable media. The responses may each indicate an ability of a corresponding one of the plurality of agents to generate response content in response to the proxy request. The method also includes generating an agent selection model based on the stored associations between the proxy request and its responses. The method also includes, subsequent to generating the agent selection model: receiving subsequent natural language input from the user directed to the automated assistant as part of the conversation between the user and the automated assistant; selecting a particular agent based on the subsequent natural language input and the agent selection model, wherein the particular agent is one of the available agents; and in response to receiving the subsequent natural language input and in response to selecting the particular agent: transmitting a call request to the selected particular agent via one or more of the application programming interfaces. The invocation request invokes a specific agent and causes the specific agent to generate specific response content for presentation to a user via one or more user interface output devices. In some embodiments, in response to receiving subsequent natural language input, the invocation request is transmitted to only the selected specific agent.

[0024] These and other implementations of the technology disclosed herein can optionally include one or more of the following features.

[0025] In some embodiments, for a given natural language input instance in the plurality of natural language input instances, a first subset of responses each indicates an ability to generate responsive content and a second subset of responses each indicates an inability to generate responsive content. In some of those embodiments, the responses of the second subset indicate inability based on a confidence metric that indicates an error or a failure to meet a threshold.

[0026] In some embodiments, the agent selection model is a machine learning model. In some of those embodiments, generating the machine learning model includes generating a plurality of training instances based on the agent requests and their responses, and training the machine learning model based on the training instances. Generating each of the training instances may include: generating a training instance input for the training instance based on a corresponding agent request in the agent requests; and generating a training instance output for the training instance based on the response stored in association with the corresponding agent request. In some of those embodiments, selecting a particular agent based on subsequent natural language input and the agent selection model includes: applying input features based on the subsequent natural language input as input to the machine learning model; generating an output including a value for the particular agent on the machine learning model based on the input; and selecting the particular agent based on the value for the particular agent. In some versions of those embodiments, selecting the particular agent is further based on one or more context values. For example, selecting the particular agent based on the one or more context values ​​may include applying the one or more context values ​​as additional input to the machine learning model.

[0027] In some implementations, the method further includes selecting the plurality of natural language input instances based on determining that the plurality of natural language input instances fail to specify any agent.

[0028] In some embodiments, the method further includes, for a given natural language input instance among multiple natural language input instances: selecting a given agent among a plurality of agents in a set using a response to the agent request; and transmitting the selected call request to the selected given agent, wherein the selected call request is based on the given natural language input instance.

[0029] In some implementations, the set of multiple agents is selected from a corpus of available agents based on the set of multiple agents each being associated with a value for an intent parameter expressed in the natural language input instance.

[0030] In some embodiments, a method performed by one or more processors is provided and includes, for each of a plurality of natural language input instances generated based on user interface input during a conversation between a person and an automated assistant: generating a proxy request based on the natural language input instance; selecting a set of multiple agents from a corpus of available agents for the proxy request; transmitting the proxy request to each of the plurality of agents in the set; and receiving a corresponding response to the request from at least a subset of the plurality of agents in response to the transmitting. The method also includes determining, based on each of the responses, a relative ability of the agent providing the response to generate response content in response to the proxy request. The method also includes storing, in one or more computer-readable media, an association between the proxy request and at least one of the agents determined to be capable of responding to the proxy request. The method also includes generating an agent selection model based on the stored association between the proxy request and the agents determined to be capable of responding to the proxy request. The method also includes, subsequent to generating the agent selection model: receiving subsequent natural language input from the user directed to the automated assistant as part of the conversation between the user and the automated assistant; selecting a particular agent based on the subsequent natural language input and the agent selection model, wherein the particular agent is one of the available agents; and in response to selecting the particular agent: transmitting a call request to the selected particular agent. The invocation request invokes a specific agent and causes the specific agent to generate specific response content for presentation to a user via one or more user interface output devices.

[0031] These and other implementations of the technology disclosed herein can optionally include one or more of the following features.

[0032] In some implementations, selecting a particular agent occurs without providing the user with an output explicitly soliciting the user to select between the particular agent and one or more other agents among the available agents.

[0033] In some embodiments, the agent selection model is a machine learning model. In some of those embodiments, generating the machine learning model includes generating a plurality of training instances based on the agent request and the agents determined to be capable of responding to the agent request; and training the machine learning model based on the training instances. Generating each of the training instances may include generating a training instance input for the training instance based on a corresponding agent request in the agent request; and generating a training instance output for the training instance based on at least one of the agents determined to be capable of responding to the request. In some of those embodiments, selecting a particular agent based on subsequent natural language input and the agent selection model includes applying input features based on the subsequent natural language input as input to the machine learning model; generating an output including a value for the particular agent based on the input to the machine learning model; and selecting the particular agent based on the value for the particular agent.

[0034] In addition, some embodiments include one or more processors of one or more computing devices, wherein the one or more processors are operable to execute instructions stored in an associated memory, and wherein the instructions are configured to cause the performance of any of the foregoing methods. Some embodiments also include one or more non-transitory computer-readable storage media storing computer instructions, wherein the instructions are executable by the one or more processors to perform any of the foregoing methods.

[0035] It should be understood that all combinations of the foregoing concepts and additional concepts described in more detail herein are contemplated as being part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a block diagram of an example environment in which embodiments disclosed herein may be implemented.

[0037] Figure 2 can be included in Figure 1 Examples of various components in the agent selection module.

[0038] Figure 3 Illustrated are examples of how agent requests and responses may be utilized in selecting a particular agent and / or stored in a record database for use in generating an agent selection model.

[0039] Figure 4 is a flow chart illustrating an example method of selecting a single agent to invoke according to implementations disclosed herein.

[0040] Figure 5 is a flowchart illustrating an example method of generating an agent selection model according to implementations disclosed herein.

[0041] Figure 6 is a flow chart illustrating another example method of selecting a single agent to invoke according to implementations disclosed herein.

[0042] Figure 7 is a flow chart illustrating an example method that may be performed by an agent according to implementations disclosed herein.

[0043] Figure 8 and Figure 9 Each illustrates an example conversation between a user; a client device; and the user, an automated assistant associated with the client device, and an agent according to implementations disclosed herein.

[0044] Figure 10Illustrated are additional client devices according to implementations disclosed herein; and example conversations between users of the client devices, automated assistants associated with the client devices, and agents.

[0045] Figure 11 Illustrated is an example architecture of a computing device. DETAILED DESCRIPTION

[0046] In some cases, in order to invoke a specific agent for a specific intent via an automated assistant, the user must provide input that explicitly invokes that specific agent. For example, to invoke an agent named "imaginary agent" for the "restaurant reservation" intent, the user must know to speak an "invocation phrase" to that agent, such as "book a restaurant with imaginary agent." This type of explicit invocation requires the user to know at the outset which agent is most appropriate for the intent, and, through interaction with the agent, sends the user directly to that agent for attempted resolution of the intent.

[0047] However, it may often be the case that the user is unaware of the availability of various agents, and for various automated assistant interfaces, it may be impractical and / or undesirable to explicitly provide the user with a list of available agents and associated functionality within the often constrained automated assistant interface. For example, some automated assistant interfaces are "voice-only," and it may be impractical and / or undesirable for the user to "read a list" of agents and associated functionality. Furthermore, it may even be the case that the automated assistant is unaware of the capabilities of the various available agents.

[0048] Various implementations disclosed herein enable selection and invocation of an appropriate agent in response to "ambiguous" natural language input from a user, such as natural language input indicating a desire to engage an agent but not indicating a specific agent to engage.

[0049] Now turn Figure 1 , illustrates an example environment in which the techniques disclosed herein may be implemented. The example environment includes a client device 106, an automated assistant 110, and a plurality of agents 140A-N. The client device 106 may be, for example, a standalone voice-activated speaker device, a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device in a user's vehicle, and / or a wearable device of a user including a computing device (e.g., a user's watch with a computing device, a user's glasses with a computing device, a virtual or augmented reality computing device). Additional and / or alternative client devices may be provided.

[0050] Although the automated assistant 110 Figure 1106, however, in some embodiments all or aspects of automated assistant 110 can be implemented by client device 106. For example, in some embodiments input processing engine 112 can be implemented by client device 106. In embodiments where one or more (e.g., all) aspects of automated assistant 110 are implemented by one or more computing devices remote from client device 106, client device 106 and those aspects of automated assistant 110 communicate via one or more networks, such as a wide area network (WAN) (e.g., the Internet).

[0051] Although only one client device 106 is illustrated in connection with automated assistant 110, in many embodiments automated assistant 110 can be remote and can interface with each of multiple client devices of multiple users. For example, automated assistant 110 can manage communications with each of the multiple devices via different sessions and can manage multiple sessions in parallel. For example, automated assistant 110 in some embodiments can be implemented as a cloud-based service that employs cloud infrastructure, e.g., using a server farm or a cluster of high-performance computers running software suitable for handling a large number of requests from multiple users. However, for simplicity, many examples herein are described with respect to a single client device 106.

[0052] Automated assistant 110 is separate from agents 140A-N and communicates with agents 140A-N via an API and / or via one or more communication channels (e.g., internal communication channels of client device 106 and / or a network (such as a WAN). In some embodiments, one or more of agents 140A-N are each managed by a respective party that is separate from the party that manages automated assistant 110.

[0053] One or more of the agents 140A-N can each optionally directly or indirectly provide data for storage in the agent database 152. However, it may often be the case that some agents 140A-N do not provide certain data, provide incomplete data, and / or provide inaccurate data. Certain embodiments disclosed herein can mitigate these situations by utilizing various additional techniques when selecting an appropriate agent for ambiguous user input. The data provided for a given agent can define, for example, intents that can be parsed by the given agent. Furthermore, the data provided for a given agent can define, for each intent, the available values ​​that can be processed by the agent for a plurality of intent bit parameters defined for the intent. In some embodiments, the automated assistant 110 and / or other components can define intents and acceptable values ​​that can be defined for each of the intent bit parameters. For example, such criteria can be defined via an API maintained by the automated assistant 110. One or more of the agents 140A-N can then provide (e.g., via a WAN) its intents and available values ​​for its intent bit parameters to the automated assistant 110 and / or other components, which can validate the data and store it in the agent database 152. The agent database 152 may additionally or alternatively store other characteristics of various agents, such as a ranking of the agents, a popularity measure of the agents, and so on.

[0054] Automated assistant 110 includes an input processing engine 112, a local content engine 130, a proxy engine 120, and an output engine 135. In some implementations, one or more engines of automated assistant 110 may be omitted, combined, and / or implemented in a component separate from automated assistant 110. Furthermore, automated assistant 110 may include additional engines that are not illustrated herein for simplicity.

[0055] Automated assistant 110 receives an example of user input from client device 106. For example, automated assistant 110 may receive free-form natural language speech input in the form of a streaming audio recording. The streaming audio recording may be generated by client device 106 in response to a signal received from a microphone of client device 106, which captures the voice input of the user of client device 106. As another example, automated assistant 110 may, in some embodiments, receive free-form natural language typing input and / or even structured (non-free-form) input. In some embodiments, user input may be generated by client device 106 and / or provided to automated assistant 110 in response to an explicit invocation of automated assistant 110 by the user of client device 106. For example, the invocation may be the detection of certain voice input to the user by client device 106 (e.g., an automated assistant 110 hotword / phrase such as "Hey Assistant"), user interaction with a hardware button and / or virtual button (e.g., a tap on a hardware button, a selection of a graphical interface element displayed by client device 106), and / or other specific user interface input.

[0056] Automated assistant 110 provides instances of output in response to receiving instances of user input from client device 106. Instances of output can be, for example, audio to be audibly presented by device 106 (e.g., output via a speaker of client device 106), textual and / or graphical content to be graphically presented by device 106 (e.g., rendered via a display of client device 106), etc. As described herein, some instances of output can be based on local response content generated by automated assistant 110, while other instances of output can be based on response content generated by a selected one of agents 140A-N.

[0057] The input processing engine 112 of the automated assistant 110 processes natural language input and / or other user input received via the client device 106 and generates annotation output for use by one or more other components of the automated assistant 110 (such as the local content engine 130 and / or the agent engine 120). For example, the input processing engine 112 can process natural language free-form input generated by a user via one or more user interface input devices of the client device 106. The generated annotation output includes the natural language input and, optionally, annotations for one or more (e.g., all) terms of the natural language input. As another example, the input processing engine 112 can additionally or alternatively include a speech-to-text module that receives an instance of speech input (e.g., in the form of digital audio data) and converts the speech input into text comprising one or more words or phrases. In some embodiments, the speech-to-text module is a streaming speech-to-text engine. The speech-to-text module can rely on one or more stored speech-to-text models (also referred to as language models), each of which can model the relationship between an audio signal and phonetic units in a language and word sequences in the language.

[0058] In some embodiments, the input processing engine 112 is configured to recognize and annotate various types of grammatical information in the natural language input. For example, the input processing engine 112 may include a part-of-speech tagger configured to annotate terms with their grammatical role. For example, the part-of-speech tagger may tag each term with its part of speech, such as "noun," "verb," ​​"adjective," "pronoun," and the like. Also, for example, in some embodiments the input processing engine 112 may additionally and / or alternatively include a dependency parser configured to determine syntactic relationships between terms in the natural language input. For example, the dependency parser may determine which terms modify other terms, subjects, and verbs of a sentence, etc. (e.g., a parse tree) - and may annotate such dependencies.

[0059] In some embodiments, the input processing engine 112 may additionally and / or alternatively include an entity tagger configured to annotate entity references in one or more segments, such as references to a group of people, organizations, locations, etc. The entity tagger may annotate references to entities at a high level of granularity (e.g., to enable identification of all references to a class of entities such as a group of people) and / or at a lower level of granularity (e.g., to enable identification of all references to a specific entity such as a specific person). The entity tagger may rely on the content of the natural language input to resolve the specific entity and / or may optionally communicate with a knowledge graph or other entity to resolve the specific entity.

[0060] In some implementations, the input processing engine 112 can additionally and / or alternatively include a coreference resolver configured to group or "cluster" references to the same entity based on one or more contextual clues. For example, a coreference resolver can be utilized to resolve the term "it" in an instance of user input that is immediately preceded by "Restaurant A" mentioned previously in a previous instance of user input.

[0061] In some embodiments, one or more components of the input processing engine 112 can rely on annotations from one or more other components of the input processing engine 112. For example, in some embodiments, a named entity tagger can rely on annotations from a coreference resolver and / or a dependency resolver when annotating all mentions of a particular entity. Additionally, for example, in some embodiments, a coreference resolver can rely on annotations from a dependency resolver when clustering references to the same entity. In some embodiments, when processing a particular natural language input, one or more components of the input processing engine 112 can use relevant prior input and / or other relevant data outside of the particular natural language input to determine one or more annotations.

[0062] The input processing engine 112 may attempt to discern the semantics or meaning of the user input and provide an indication of the semantics of the user input to the local content engine 130 and / or the proxy engine 120 for use by those engines. The input processing engine 112 may rely on one or more stored grammar models to map text (or other input) to specific actions and identify properties that constrain the performance of such actions, e.g., input variables for such actions.

[0063] When the user input is associated with a "native action" (as opposed to a proxy action), the local content engine 130 can generate a response to the received user input. In some embodiments, the input processing engine 112 determines whether the user input is associated with a local action or a proxy action. The local content engine 130 can work in conjunction with the input processing engine 112 and perform one or more actions as indicated by the parsed text (e.g., action and action parameters) provided by the input processing engine 112. For local intents, the local content engine 130 can generate local response content and provide the local response content to the output engine 135 to provide corresponding output for presentation to the user via the device 106. The local content engine 130 can utilize one or more stored local content models 154 for generating local content and / or performing other actions. The local content model 154 can, for example, incorporate various rules for creating local response content. In some embodiments, the local content engine 130 can communicate with one or more other "local" components (such as a local dialogue module (which can be the same as or similar to the dialogue module 126)) when generating local response content.

[0064] The output engine 135 provides an example of output to the client device 106. The example of output can be based on local response content (from the local content engine 130) and / or response content from one of the agents 140A-N (when the automated assistant 110 is acting as an intermediary). In some embodiments, the output engine 135 may include a text-to-speech engine that converts the text component of the response content into an audio format, and the output provided by the output engine 135 is in an audio format (e.g., as streaming audio). In some embodiments, the response content may already be in an audio format. In some embodiments, the output engine 135 additionally or alternatively provides text reply content as output (optionally for conversion to audio by the device 106) and / or provides other graphical content as output for graphical display by the client device 106.

[0065] The proxy engine 120 includes a parameter module 122, an agent selection module 124, a dialog module 126, and an invocation module 128. In some implementations, the modules of the proxy engine 120 may be omitted, combined, and / or implemented in components separate from the proxy engine 120. Furthermore, the proxy engine 120 may include additional modules that are not illustrated herein for simplicity.

[0066] Parameter module 122 determines values ​​for parameters such as intent parameters, intent position parameters, context parameters, and the like. Parameter module 122 determines these values ​​based on input provided by the user in the conversation with automated assistant 110 and, optionally, based on the client device context. The value of the intent parameter indicates the intent indicated by the input provided by the user in the conversation and / or indicated by other data. For example, the value of a conversation intent can be one of a plurality of available intents, such as "make a reservation," "book a restaurant reservation," "book a hotel," "purchase professional services," "tell a joke," "reminder," "purchase travel services," and / or other intents. Parameter module 122 can determine the intent based on the most recent natural language input provided to the automated assistant in the conversation and / or past natural language input provided in the conversation.

[0067] The values ​​of the intent bit parameters refer to the values ​​of the finer-grained parameters of the schematic. For example, the "book a restaurant reservation" intent may have intent bit parameters for "number of people", "date", "time", "cuisine type", "specific restaurant", "restaurant area", etc. The parameter module 122 may determine the values ​​of the intent bit parameters based on the input provided by the user in the conversation and / or based on other considerations (e.g., user-set preferences, past user interactions). For example, the values ​​of one or more intent bit parameters of the "book a restaurant reservation" intent may be based on the most recent natural language input provided to the automated assistant in the conversation and / or past natural language input provided in the conversation. For example, the natural language input of "book a restaurant for me tonight at 6:30" may be used by the parameter module 122 to determine the value of the intent bit parameter "date" to be "today's date" and the value of the intent bit parameter "time" to be "18:30". It should be understood that for many conversations, the parameter module 122 may not resolve the values ​​of all (or even any) of the intent bit parameters of the intent. Such values ​​may be resolved (if any) by the agent that is called to satisfy the intent.

[0068] Values ​​for context parameters may include, for example, client device context values, such as values ​​based on historical interactions of a user of the client device, currently rendered and / or recently rendered content on the client device, the location of the client device, the current date and / or time, etc.

[0069] Dialogue module 126 may utilize one or more grammatical models, rules, and / or annotations from input processing engine 112 when interacting with a user via client device 106 to select a particular agent. Parameter module 122 and / or agent selection module 124 may optionally interact with dialogue module 126 to generate prompts for soliciting further user input related to selecting a particular agent. Any prompts generated by dialogue module 126 may be provided for presentation to the user by output engine 135 and for receiving further responsive user input. The further user input may each be analyzed by parameter module 122 (optionally annotated as by input processing engine 112) and / or agent selection module 124 to assist in selecting a particular agent. As an example, three candidate agents may be initially selected as potential agents by agent selection module 124 according to the techniques described herein, and dialogue module 126 may present one or more prompts to the user soliciting the user to select a particular one of the three candidate agents. Agent selection module 124 may then select the particular agent based on the user selection.

[0070] The agent selection module 124 utilizes the values ​​determined by the parameter module 122 to select a particular agent from the agents 140A-N to be invoked. The agent selection module 124 may additionally or alternatively utilize other criteria when selecting a particular agent. For example, the agent selection module 124 may utilize one or more selection models of the selection model database 156 and / or use the agent database 152 when selecting a particular agent.

[0071] refer to Figure 2 , illustrating examples of various components 124A-E that may be included in the agent selection module 124.

[0072] The request component 124A can transmit a "live" proxy request to one or more of the agents 140A-N and utilize the responses (and / or lack of responses) from those agents to determine the ability of each of those agents to generate responsive content in response to the proxy request. For example, the request component 124A can transmit a live proxy request to those agents 140A-N that are associated (e.g., in the agent database 152) with intents that match the values ​​of intent parameters determined by the parameter module 122. As described herein, the proxy request can be based on the values ​​of the parameters determined by the parameter module 122. The proxy request can be similar to (or the same as) an invocation request, but does not result in the immediate invocation of any agent. The response (or lack of response) from the agent in response to the proxy request can directly or indirectly indicate the agent's ability to generate responsive content in response to the proxy request.

[0073] The agent context component 124B determines characteristics of one or more of the agents 140A-N, such as those agents 140A-N that are associated (e.g., in the agent database 152) with intents that conform to values ​​of intent parameters determined by the agent module 122. The agent context component 124B can determine the characteristics from the agent database 152. The characteristics of the agents can include, for example, a stored ranking of a particular agent (e.g., a ranking by user group), the general popularity of the particular agent, the popularity of the particular agent with respect to the intent determined by the parameter module 122, and the like.

[0074] Client context component 124C determines characteristics associated with the user of client device 106 (and / or client device 106 itself), such as characteristics based on the user of the client device's historical interactions with automated assistant 110. For example, the characteristics may include characteristics based on: how frequently each of the various agents has been utilized by the user; how recently each of the various agents has been utilized by the user; currently rendered and / or recently rendered content on the client device (e.g., entities present in the rendered content, recently utilized applications); the location of the client device; the current date and / or time; and the like.

[0075] The selection model component 124D utilizes one or more selection models from the selection model database 156 to determine one or more of the agents 140A-N that may be appropriate for invocation. For example, the selection model component 124D may utilize the selection models to determine, for each of the plurality of agents 140A-N, one or more probabilities or other metrics indicating the appropriateness of invoking the agent. The selection model component 124D may apply one or more of the parameter values ​​determined by the parameter module 122 and / or the values ​​determined by the components 124A, 124B, and / or 124C to each of the selection models.

[0076] The selection component 124E utilizes the output provided by the components 124A, 124B, 124C, and / or 124D in selecting one or more agents 140A-N. In some embodiments and / or scenarios, the selection component 124E selects only a single agent from the agents 140A-N without prompting the user to choose between multiple agents. In some other embodiments and / or scenarios, the selection component 124E may select a subset of the agents 140A-N and provide a prompt to the user (e.g., via a prompt generated in a dialog by the dialog module 126) that solicits the user to provide user interface input to select one of the agents in the subset. The selection component 124E may provide an indication of the selected agent to the invocation module 128.

[0077] Invocation module 128 transmits a call request including parameters determined by parameter module 122 to the agent (among agents 140A-N) selected by agent selection module 124. The transmitted call request invokes the specific agent. As described herein, in some cases, when a specific agent is invoked, automated assistant 110 can still act as an intermediary. For example, when acting as an intermediary where the user's natural language input is voice input: input processing engine 112 of automated assistant 110 can convert the voice input into text; automated assistant 110 can transmit the text (and optionally annotations to the text from input processing engine 112) to the specific agent and receive response content from the specific agent; and output engine 135 can provide output based on the response content for presentation to the user via client device 106. In addition, for example, when acting as an intermediary, automated assistant 110 can additionally or alternatively analyze the user input and / or response content to determine whether the conversation with the agent should be terminated, transferred to an alternative agent, etc. As further described herein, in some cases the conversation may actually be transferred to the agent (once transferred, without the automated assistant 110 acting as an intermediary), and when one or more conditions occur, transferred back to the automated assistant 110. Additionally, as further described herein, in some cases the invoked agent may be executed by the client device 106 and / or brought to the forefront by the client device 106 (e.g., its content may take over the display of the client device 106).

[0078] Each of the agents 140A-N may include a context parameter engine, a content engine, and / or other engines. In addition, in many embodiments, the agent may access various stored models and / or other resources (e.g., its own grammar model and / or content model) when generating response content.

[0079] Figure 1 Also illustrated in FIG. 1 are a model engine 150 and a record database 158. As described in greater detail herein, the record database 158 may include stored information based on interactions between various automated assistants 110 and agents 140A-N. The model engine 150 may utilize this information when generating one or more selection models for the selection model database 156. Additional description is provided herein.

[0080] Now turn Figure 3-10 , will describe the Figure 1 Additional descriptions of the various components of the environment.

[0081] Figure 3 Examples are provided of how agent requests and responses may be utilized in selecting a particular agent and / or stored in the record database 158 for use in generating an agent selection model.

[0082] exist Figure 3 , natural language input 171 is received by input processing engine 112 of automated assistant 110. As a working example, natural language input 171 may be "table for 4, outdoor seating, restaurant A." Input processing engine 112 generates annotated input 172 and provides the annotated input 172 to parameter module 122.

[0083] The parameter module 122 generates values ​​173 for the parameters based on the annotated input 172 and / or based on the client device context. Continuing with the working example, the parameter module 122 can generate a value of "restaurant reservation" for the intent parameter, a value of "outdoors" for the seating preference intent parameter, and a value of "restaurant A" for the restaurant location intent parameter.

[0084] Request component 124A generates a proxy request based on parameter value 173. Figure 3 As indicated by the "AR" directed arrows in FIG. , the request component 124A transmits the proxy request to each of the plurality of agents 140A-D. In some embodiments, the request component 124A may select the agents 140A-D based on determining that they are associated with the "restaurant reservation" intent. In some other embodiments, the request component 124A may transmit the proxy request to the agents 140A-D and / or additional agents regardless of the intent (e.g., it may be transmitted to all agents 140A-N). For example, the intent that can be processed by the various agents may be unknown and / or the intent may not be derived from the conversation.

[0085] If through Figure 3 As indicated by the "R" directed arrows in FIG. 1 , the request component 124A receives responses from each of the proxies 140A-D in response to transmitting the proxy request to the proxies 140A-D. The responses each indicate the ability of the corresponding one of the proxies 140A-D to resolve the proxy request. For example, the response from a given proxy can be a binary indication, a non-binary confidence metric, actual response content (or no content / error), etc. Although in Figure 3 Responses from each of these agents are illustrated in , however in some embodiments or scenarios one or more of the agents 140A-D may not respond - which may indicate that the corresponding agent is unable to respond (e.g., the agent cannot process the agent request and / or is offline). The agent request is transmitted to the agent 140A-D without requiring active invocation by the agent 140A-D.

[0086] Request component 124A stores the responses (and / or determinations based on the responses) and the proxy requests in record database 158. For example, request component 124A can store the proxy requests and an indication of the responses for each of agents 140A-D. For example, request component 124A can store the proxy requests and an indication that agent 140A was unable to respond, agent 140B was able to respond, agent 140C was unable to respond, etc.

[0087] Request component 124A also provides response 174 (and / or a determination based on the response) to selection component 124E.

[0088] The selection component 124E utilizes the responses 174 to select a particular agent 176 and may optionally additionally utilize features provided by the agent context component 124B and / or the client context component 124C. As one example, the selection component 124E may select a particular agent 176 based solely on the responses 174 (e.g., selecting the agent with the response that is most indicative of responsiveness). As another example, the selection component 124E may utilize the responses 174 and a ranking of one or more of the agents 140A-D provided by the agent context component 124B. For example, the selection component 124E may initially select two of the agents 140A-D whose responses are most indicative of responsiveness and then select one of those agents based on the selected agent ranking higher than the unselected agent. As yet another example, the selection component 124E may utilize the responses 174 and historical usage features provided by the client context component 124C. For example, selection component 124E may initially select two of agents 140A-D whose responses are most indicative of responsiveness, and then select one of those based on the selected one being more frequently and / or recently utilized by users interacting with automated assistant 110 .

[0089] The selection component 124E provides the specific agent 176 to the call module 128. The call module 128 can call the specific agent. For example, the call module 128 can call the specific agent by transmitting a call request based on the parameter value 173 to the specific agent. In addition, for example, the call module 128 can use the output engine 135 to provide output based on the response content already received from the specific agent - and can optionally call the specific agent to generate further response content in response to further received user interface input (if any), which is received in response to providing the output.

[0090] Figure 4 is a flowchart illustrating an example method 400 according to embodiments disclosed herein. For convenience, reference is made to the description of the system performing the operations. Figure 4 The operations of the flowchart of method 400 may be performed. This system may include various components of various computer systems, such as one or more components of automated assistant 110. In addition, although the operations of method 400 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0091] At block 450, the system receives user input. In some implementations, the user input received at block 450 is free-form natural language input.

[0092] At block 452, the system determines whether the received user input indicates a proxy action. For example, the system may parse the text of the received natural language input (or text converted from the received audio natural language input) and determine whether the parsed text maps to any proxy actions. For example, the system may determine whether the parsed text maps to any proxy actions based on whether the terms / phrases included in the text match the terms / phrases stored in association with the proxy actions. Additionally, for example, the system may determine whether one or more entities derived from the text match one or more entities stored in association with the proxy actions. As another example, if the system determines that it cannot generate local response content in response to the input, the system may assume that the input maps to a proxy action. Note that in some embodiments, the system may determine that the input maps to a proxy action even when the system is able to generate local response content in response to the input. For example, the system may determine that it is able to generate local response content and that one or more agents can also potentially generate proxy response content. In some of those examples, the system may include a local agent among the agents to be considered in blocks 462, 464, 466, and 468 (described below).

[0093] If the system determines at block 452 that the agent intended action is not indicated, the system proceeds to blocks 454, 456, and 458. At block 454, the system generates local response content without the agent's call. For example, the system can utilize the system's local grammar model and / or local content model to generate local response content. At block 456, the system provides an output based on the local response content. For example, the output can be a conversion of the local response content or the local response content (e.g., text to speech conversion). The output is provided for presentation (e.g., audible or graphical) via a client device. At block 458, the system waits for additional voice input and proceeds back to block 450 when additional voice input is received.

[0094] If the system determines at the iteration of block 452 that an agent action is indicated, the system proceeds to block 460. At block 460, the system determines whether a single agent was specified in the user input of block 450 and / or can otherwise be unambiguously resolved.

[0095] If the system determines at block 460 that a single agent is indicated, the system proceeds to block 474 .

[0096] If the system determines at block 460 that a single agent has not been indicated, the system proceeds to blocks 462, 464, 466, and 468. At block 462, the system generates a proxy request based on the user input in the most recent iteration of block 450 and / or based on prior user input and / or other criteria. At block 464, the system selects a plurality of agents from the corpus of available agents. For example, the system may select all agents in the corpus or a subset of agents (e.g., those with the intent indicated by the proxy request). The agents may include only non-local agents and / or both local and non-local agents. At block 466, the system transmits the proxy request of block 462 to each of the agents selected in block 464.

[0097] At block 468, the system receives one or more responses from the agent in response to the transmission of block 466. At block 470, the system stores an association between the responses received at block 468 (or determinations made based on the responses) and the agent request transmitted at block 466. This stored association can be used in generating an agent selection model (e.g., in Figure 5 At block 472, the system selects a single agent based on the response received at block 468 and / or based on other criteria.

[0098] In some embodiments, the system uses the response received at block 468 to select a subset of agents, provides an indication of the agents in the subset as output for presentation to the user, and utilizes the user selection in response to the output to select a single agent from the agents in the subset at block 472. In some versions of those embodiments, the system may also store the indication of the single agent selected by the user at block 470. The indication of the agent may be, for example, a name or other identifier of the agent and / or an indication of the content of the agent's response included in the response received at block 468.

[0099] At block 474, the system transmits the call request to a single agent. The single agent can be the one selected at block 472 (the determination at block 460 is "no") or the single agent indicated in the user input (the determination at block 460 is "yes"). For example, the system can transmit the call request via one or more communication channels and can optionally utilize an API. In some embodiments, the call request includes values ​​for various call parameters as described herein. In some embodiments, the system can first prompt the user to confirm that the user desires to utilize a single agent before proceeding from block 472 to block 474. In those embodiments, the system can require affirmative user input in response to the prompt before proceeding from block 472 to block 474. In other embodiments, the system can automatically proceed from block 472 to block 474 without first prompting the user for confirmation.

[0100] At optional block 476, the system may update the global parameter value based on the user's interaction with the single agent. For example, further natural language input provided by the user to the single agent may cause the single agent to define a value for a previously undefined global parameter (e.g., a specific intent parameter)—or may cause the single agent to modify a previously defined value for the parameter. The single agent may provide such an updated value to the system and the system may update the global parameter value to reflect the updated value.

[0101] At block 478, the system receives an agent switching input. The agent switching input indicating a desire to switch to a different agent is input from the user (e.g., natural language input). For example, input such as "talk to another agent," "a different agent," "try it with agent X," etc. can be an agent switching input.

[0102] In response to receiving the proxy switch input, the system transmits a call request to an alternative proxy at block 480. The call request to the alternative proxy invokes the alternative proxy in place of the single proxy invoked in block 474. The call request may optionally include the updated global parameter value updated at block 476. In this way, values ​​derived from interactions with the first proxy can be transferred to subsequently invoked proxies, thereby improving the efficiency of interactions with subsequently invoked proxies. In some cases, the selection of which alternative proxy to transmit additional call requests to can be based on the proxy switch input itself (e.g., if it references one of the alternative proxies by name or characteristics) and / or based on other factors (e.g., the proxy request can be transmitted again based on the updated global parameter value and the response utilized to select the alternative proxy). In some embodiments, before transmitting the call request to the alternative proxy at block 480, the system can check to ensure that the alternative proxy is likely to be able to generate response content. For example, the system can transmit the proxy request to the alternative proxy and make such a determination based on the response, and / or make such a determination based on information about the proxy in proxy database 152.

[0103] Figure 5 is a flow chart illustrating another example method 500 according to implementations disclosed herein. Figure 5 The diagram shows an example of generating a selection model in the selection model database 156 based on proxy requests and associated responses. Figure 5 In the example, the selected model is a machine learning model, such as a deep neural network model.

[0104] For convenience, refer to the system description of the operation performed Figure 5 The operations of the flowchart of the method 500 may include various components of various computer systems, such as the model engine 150. In addition, although the operations of the method 500 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0105] At block 552, the system selects a proxy request and associated response. As an example, the proxy request and associated response may be selected from the record database 158. In some embodiments, the selected proxy request and associated response may have been generated by the request component 124A. Figure 3 generated as shown in the figure and / or may have been Figure 4470 of method 400. In some other embodiments, agent requests and / or associated responses generated by other techniques may be utilized. For example, agent requests may be transmitted to an agent and responses received in a "non-live" manner. In other words, agent requests and responses are not necessarily those generated when a particular agent was actively selected during a past human-automated assistant conversation. As a non-limiting example, agent requests may include those based on natural language input provided to an agent immediately following an invocation as a "naked" invocation. For example, natural language input provided to "Agent A" immediately following a naked invocation of "Agent A" may be utilized to generate agent requests that are thereafter transmitted to multiple additional agents.

[0106] As a working example, assume that the agent request selected at the iteration of block 552 includes a value of "restaurant reservation" for the intent parameter, a value of "outdoors" for the seating preference intent parameter, and a value of "restaurant A" for the restaurant location intent parameter. Further assume that the associated response selected includes a binary response of "yes" (response content can be generated) or "no" (response content cannot be generated) to the agent request. In particular, the associated response indicates that agent 1-5 generated a "yes" response and agent 6-200 generated a "no" response.

[0107] At block 554 , the system generates training instances based on the selected proxy requests and associated responses. Block 554 includes sub-blocks 5541 and 5542 .

[0108] At sub-block 5541, the system generates a training instance input for the training instance based on the proxy request and, optionally, based on the additional value. Continuing with the working example, the system may generate a training instance input that includes values ​​for the proxy request for the following: intent parameter, seating preference intent parameter, and restaurant location intent parameter. For dimensions of the training instance input that are not defined by the proxy request, the system may include a "null" value (or other value) in the training instance. For example, if the input dimensions of the machine learning model to be trained include inputs for other intent parameters (same intent and / or other intents), null values ​​may be used for such inputs in the training instance.

[0109] At sub-block 5542, the system generates a training instance output for the training instance based on the response. Continuing with the working example, the system may generate a training instance output that includes a "1" (or other "positive value") for each of the output dimensions corresponding to agents 1-5 (which generated a "yes" response) and a "0" (or other "negative" value) for each of the output dimensions corresponding to agents 6-200 (which generated a "no" response).

[0110] At block 556, the system determines whether there are additional proxy requests and associated responses. If so, the system returns to block 552 and selects another proxy request and associated response, and then generates another training instance based on the selected proxy request and associated response.

[0111] Blocks 558 - 566 may be performed immediately following or in parallel with multiple iterations of blocks 552 , 554 , and 556 .

[0112] At block 558 , the system selects a training instance generated in the iteration of block 554 .

[0113] At block 560, the system applies the training instance as input to the machine learning model. For example, the machine learning model may have input dimensions corresponding to the dimensions of the training instance input generated at block 5541.

[0114] At block 562, the system generates an output on the machine learning model based on the applied training instance input. For example, the machine learning model may have output dimensions corresponding to the dimensions of the training instance output generated at block 5541 (e.g., each dimension of the output may correspond to an agent and / or an agent and an intent).

[0115] At block 564, the system updates the machine learning model based on the generated output and the training instance output. For example, the system may determine an error based on the output generated at block 562 and the training instance output, and backpropagate the error through the machine learning model.

[0116] At block 566, the system determines whether there are one or more additional unprocessed training instances. If so, the system proceeds back to block 558, selects additional training instances, and then performs blocks 560, 562, and 564 based on the additional unprocessed training instances. In some embodiments, at block 566, if one or more training criteria have been met (e.g., a threshold number of epochs have occurred and / or a threshold duration of training has occurred), the system may determine not to process any additional unprocessed training instances. Although method 500 is described with respect to non-batch learning techniques, batch learning may additionally and / or alternatively be used.

[0117] The machine learning model trained according to method 500 can then be used to predict a probability for each of a plurality of available agents (and optionally intents) based on the current conversation, wherein each of these probabilities indicates a probability that the agent is capable of appropriately handling a call request based on the conversation. For example, values ​​based on the current conversation can be applied as input to the trained machine learning model to generate an output on the model, wherein the output includes a plurality of values, each corresponding to an agent, and each of these values ​​indicates a probability (e.g., a value from 0 to 1) that the corresponding agent is capable of generating appropriate response content when called. For example, if 200 available agents are represented by the model, the output can include 200 values, each of which corresponds to one of the agents and indicates a probability that the agent is capable of generating appropriate response content. In this way, the trained machine learning model effectively provides insight into the capabilities of various agents by being trained based on their responses to various real-world agent requests. The trained machine learning model can be used to determine the capabilities of various agents to generate responses to inputs based on the inputs, even when the agent database 152 and / or other resources do not explicitly indicate the capabilities of those agents with respect to the inputs.

[0118] Figure 5 FIGURE 1 illustrates one example of an agent selection model that may be generated and utilized. However, as described herein, additional and / or alternative agent selection models may be utilized in selecting a particular agent. Such additional and / or alternative agent selection models may optionally be based on information related to Figure 5 The machine learning model trained on different training instances is described.

[0119] As an example, a selection model may be generated based on past explicit selections of agents by various users and such a selection model may additionally or alternatively be utilized when selecting a particular agent. Figure 4As described in block 472 of FIG5 , in some embodiments, an indication of multiple agents may be presented to the user and the user's selection of a single agent from the multiple agents may be utilized to select a single agent from the multiple agents. Such explicit selections by multiple users may be utilized to generate a selection model. For example, training instances similar to those described above with respect to method 500 may be generated, but the training instance outputs for each training instance may be generated based on the agent selected by the user. For example, a training instance of "1" (or other "positive value") may be used for the output dimension corresponding to the selected agent, and "0" (or other "negative" value) may be used for each of the output dimensions corresponding to all other agents. Additionally, for example, a training instance of "1" (or other "positive value") may be used for the output dimension corresponding to the selected agent, "0.5" (or other "intermediate value") may be used for the output dimensions corresponding to other agents presented to the user but not selected, and "0" (or other "negative" value) may be used for each of the output dimensions corresponding to all other agents. In this and other ways, explicit user selections of agents may be utilized in generating one or more agent selection models.

[0120] Figure 6 is a flow chart illustrating another example method 600 according to implementations disclosed herein. Figure 6 The diagram shows how to use an agent selection model (such as one based on Figure 5 An example of an agent selection model generated by method 500).

[0121] For convenience, refer to the system description of the operation performed Figure 6 The operations of the flowchart of method 600 may be performed in a manner similar to that of FIG. This system may include various components of various computer systems, such as one or more components of automated assistant 110. Furthermore, although the operations of method 600 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0122] At block 650, the system receives user input. Block 650 may be shared with Figure 4 The blocks 450 have one or more aspects in common.

[0123] At block 652, the system determines whether the received user input indicates an agent action. Block 652 may be shared with Figure 4 The blocks 452 have one or more aspects in common.

[0124] If the system determines at block 652 that the agent intended action is not indicated, the system proceeds to blocks 654, 656, and 658. At block 654, the system generates local response content without invoking the agent. At block 656, the system provides output based on the local response content. At block 658, the system waits for additional voice input and proceeds back to block 650 when additional voice input is received. Blocks 654, 656, and 658 can share the same Figure 4 Blocks 454, 456, and 458 have one or more aspects in common.

[0125] If the system determines at the iteration of block 652 that a proxy action is indicated, the system proceeds to block 660. At block 660, the system determines whether a single proxy is specified in the user input 650 and / or can otherwise be resolved unambiguously. Block 660 may be used with Figure 4 Block 460 shares one or more aspects.

[0126] If the system determines at block 660 that a single agent is indicated, the system proceeds to block 680 .

[0127] If the system determines at block 660 that a single agent has not been indicated, the system proceeds to blocks 672, 674, 676, and 678. At block 672, the system generates input features based on the user input in the most recent iteration of 650 and / or based on prior user input and / or other criteria. For example, the system can generate input features that include values ​​for parameters determined based on the user input, such as values ​​for an intent parameter, a bit of intent parameter, etc. Additionally, for example, the system can generate values ​​based on the current client device context.

[0128] At block 674 , the system applies the input features to the agent selection model.

[0129] At block 676, the system generates a probability for each of the plurality of agents based on the application of the input to the agent selection model. Each of these probabilities indicates the ability of the corresponding agent to generate appropriate response content.

[0130] At block 678, the system selects a single agent based on probability and / or other criteria. In some embodiments, the system selects a single agent based on the highest probability of generating appropriate response content. In some other embodiments, the system selects a single agent based on additional criteria. For example, the system may select an initial subset of agents based on probability, transmit "live" agent requests to the agents in the subset, and utilize the "live" responses to the agent requests when selecting a single agent. As another example, the system may additionally or alternatively select a single agent based on the historical interactions of the user of the client device (e.g., how often the single agent has been utilized by the user, how recently the single agent has been utilized by the user), the currently rendered and / or recently rendered content on the client device, the location of the client device, the current date and / or time, the ranking of the single agent (e.g., ranking by user group), the popularity of the single agent (e.g., popularity among the user group), etc.

[0131] At block 680, the system transmits the call request to a single agent. The single agent may be the one selected at block 678 (where the determination at block 660 is "no"), or the single agent indicated in the user input (where the determination at block 660 is "yes"). Block 680 may share Figure 4 The blocks 474 have one or more aspects in common.

[0132] Figure 7 is a flow chart illustrating another example method 700 according to implementations disclosed herein. Figure 7 Illustrated are examples of methods that may be performed by one or more of the agent selection models.

[0133] For convenience, refer to the system description of the operation performed Figure 7 The operations of the flowchart of method 700 are shown in FIG. This system may include various components of various computer systems, such as one or more components of one of agents 140A-N. Furthermore, while the operations of method 700 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0134] At block 752, the system receives a proxy request from the automated assistant. In some implementations, the proxy request is labeled or otherwise indicated as a proxy request.

[0135] At block 754, the system generates a response based on the proxy request. For example, the system can parse the content of the proxy request, determine whether the system can respond to the request, and generate a response based on whether it can respond to the request. For example, the request may include the value of an intent parameter and the system may determine whether it can respond to the request based on whether it can respond to the intent indicated by the value. For example, if the value of the intent parameter is "booking", but the system can only handle the "game" intent, it may determine that it cannot respond and generate a response indicating that it cannot respond at all (e.g., "0" or other "negative" response value). In addition, for example, the request may include the value of an intent bit parameter and the system may determine whether it can respond to the request based on whether it can respond to the intent bit parameter and / or value. For example, if the system does not support the intent bit parameter, but supports other parameters and values ​​of the proxy request, the system may generate a response indicating that it can handle some but not all of the parameters of the proxy request (e.g., "0.5" or other "partial" response value). As another example, the system may support the intent bit parameter, but the system may not support a specific value. For example, the intent parameter may be a "geographic region" parameter and the value may be a geographic region not served by the system. In such a scenario, the system may generate a response indicating that it is unable to respond or, more specifically, that it is able to handle some but not all of the values ​​of the proxy request.

[0136] At block 756 , the system provides (eg, transmits) the proxy response to the automated assistant without being invoked by the automated assistant.

[0137] At block 758, the system may then receive an invocation request from the automated assistant. In some implementations, the invocation request may indicate that the system should effectively or physically take over the conversation. If the invocation request indicates that the system should physically take over the conversation, the system may establish a direct network communication session with the corresponding client device. If the invocation request indicates that the system should effectively take over the conversation, the system may take over the conversation while still communicating with the component that provided the invocation request and / or related components.

[0138] At block 760 , the system generates response content based on the values ​​of the parameters included in the call request.

[0139] At block 762, the system provides response content. For example, if the call request indicates that the system should effectively take over the conversation and / or should only perform a predetermined action without participating in the conversation, the system can transmit the response content to the component (or related components) that sent the call request. Alternatively, for example, if the call request indicates that the system should actually take over the conversation, the system can transmit the response content to the corresponding client device.

[0140] Blocks 758, 760, and 762 are Figure 7 are illustrated in dashed lines to indicate that they may not be executed in certain situations. For example, as described herein, in some embodiments the system may receive a proxy request without having received a corresponding call request.

[0141] Figure 8 and Figure 9 Each diagram illustrates an example of a user 101, a voice-enabled client device 806, and a conversation that may occur between the user 101, the automated assistant associated with the client device 806, and an agent. The client device 806 includes one or more microphones and one or more speakers. The client device 806 may be implemented on one or more computing devices in network communication with the client device 806. Figure 1 Therefore, for ease of description in Figure 8 and Figure 9 The description references automated assistant 110.

[0142] exist Figure 8 In the example embodiment, a user provides voice input 880A of "Assistant, send flowers to my house today." A speech input corresponding to the voice input is generated by device 806 and provided to automated assistant 110 (e.g., as streaming speech input). Even if voice input 880A does not specify a particular agent, automated assistant 110 can utilize voice input 880A to select a single agent from multiple available agents based on one or more techniques described herein (e.g., based on an agent selection model).

[0143] In response to voice input 880A and selecting a single agent, automated assistant 110 may generate and provide output 882A, “Sure, Agent 1 can handle that.” Alternatively, automated assistant 110 may call “Agent 1,” which then provides agent output 882B, “Hi, this is Agent 1. What kind of flower?”

[0144] In response to agent output 882B, the user provides further voice input 880B of "12 red roses." Speech input corresponding to this voice input is generated by device 806 and provided to automated assistant 110, which forwards the voice input (or a transformation and / or annotation thereof) to "Agent 1." Further voice input 880B specifies a value for the previously unspecified "flower type" intent bit parameter for the "order flowers" intent. Automated assistant 110 can update the global value of the "flower type" intent bit parameter based on further voice input 880B (either directly or based on an indication of that value provided by "Agent 1").

[0145] In response to further voice input 880B, "Agent 1" provides further agent output 882C of "I can deliver them at 5 o'clock for a total of $60. Want to order?"

[0146] In response to further agent output 882C, the user provides further voice input 880C of "Assistant, switch me to another flower agent." Automated assistant 110 may recognize this further voice input as a switch input and select an appropriate alternative agent. For example, automated assistant 110 may determine, based on the agent selection model and / or the "live" agent request, that "agent 2" can handle intents with various values ​​for the intent bit parameter, including a value for "flower type."

[0147] In response to further voice input 880C and selection of an alternative agent, automated assistant 110 may generate and provide output 882D, "Sure, Agent 2 can handle that, too." Furthermore, automated assistant 110 invokes "Agent 2" with a call request that includes the updated global value for the "flower type" intent bit parameter. "Agent 2" then provides agent output 882E, "Hi, this is Agent 2. I can deliver 12 red roses at 5:00 for $50. Order now?" Notably, this output is generated based on the updated global value for the "flower type" intent bit parameter, which was updated in response to the interaction with the previously invoked "Agent 1."

[0148] The user then provides further speech input 880F of "yes" to cause "Agent 2" to satisfy the intent with the specified value for the intent bit parameter.

[0149] exist Figure 9 In the example embodiment, a user provides voice input 980A of "Assistant, table for 2, outdoor seating, tonight at 6pm at an imaginary cafe." A speech input corresponding to the voice input is generated by device 806 and provided to automated assistant 110 (e.g., as streaming speech input). Even though voice input 980A does not specify a particular agent, automated assistant 110 can utilize voice input 980A to select a single agent from multiple available agents based on one or more techniques described herein (e.g., based on an agent selection model).

[0150] In response to voice input 980A and selecting a single agent, automated assistant 110 may invoke "Agent 3." "Agent 3" is able to fulfill the intent with the specified value for the intent bit parameter without engaging in any further conversation with the user. "Agent 3" generates response content to be provided to automated assistant 110 indicating that the appointment is complete and a text confirmation will be sent to user 101. In response, automated assistant 110 provides output 982A that reads, "Appointment made with Agent 3. Text confirmation on the way."

[0151] Figure 10 Another client device 1006 and a display screen 1040 of the client device 1006 are shown. The client device 1006 may include and / or communicate with the automated assistant 110. The display screen 1040 includes a reply interface element 1088 that the user can select to generate user input via a virtual keyboard and a voice reply interface element 1089 that the user can select to generate user input via a microphone. In some embodiments, the user can generate user input via the microphone without selecting the voice reply interface element 1089. For example, during a conversation, active monitoring of the audible user interface for input via the microphone can occur to eliminate the need for the user to select the voice reply interface element 1089. In some of those embodiments and / or in other embodiments, the voice reply interface element 1089 can be omitted. Furthermore, in some embodiments, the reply interface element 1088 can be omitted in addition and / or alternatively (e.g., the user can provide only audible user interface input). Display screen 1040 also includes system interface elements 1081 , 1082 , 1083 that can be interacted with by a user to cause client device 1010 to perform one or more actions.

[0152] exist Figure 10 In the input, enter 1080A and Figure 8 Likewise, outputs 1082A and 1082B are the same as Figure 8 The outputs 882A and 882B are the same. Figure 10 is illustrated as a graphical interaction with automated assistant 110 and / or agent (instead of Figure 8 Although Figure 10 An example of graphical interaction is shown, however different examples are envisioned. For example, instead of Figure 10 As illustrated in FIG, “Agent 1” provides output 1082B via a common interface with the automated assistant—but “Agent 1” may instead be a separate application that is launched and / or brought to the foreground in response to a call. For example, in Figure 10 After output 1082A, the "Agent 1" application interface can be replaced Figure 10 The automated assistant interface then provides an output similar to output 1082B.

[0153] Figure 11 is a block diagram of an example computing device 1110 that can optionally be utilized to perform one or more aspects of the techniques described herein. In some implementations, one or more of the device 106, automated assistant 110, 3P agent, and / or other components can include one or more components of the example computing device 1110.

[0154] The computing device 1110 typically includes at least one processor 1114 that communicates with a number of peripheral devices via a bus subsystem 1112. These peripheral devices may include a storage subsystem 1124 (including, for example, a memory subsystem 1125 and a file storage subsystem 1126), user interface output devices 1120, user interface input devices 1122, and a network interface subsystem 1116. The input and output devices allow a user to interact with the computing device 1110. The network interface subsystem 1116 provides an interface to an external network and couples to corresponding interface devices in other computing devices.

[0155] The user interface input devices 1122 may include a keyboard, a pointing device such as a mouse, a trackball, a touchpad or graphic tablet, a scanner, a touch screen incorporated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and methods for inputting information into the computing device 1110 or onto a communication network.

[0156] The user interface output device 1120 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for generating a visible image. The display subsystem may also provide a non-visual display such as via an audio output device. In general, the use of the term "output device" is intended to include all possible types of devices and methods for outputting information from the computing device 1110 to a user or to another machine or computing device.

[0157] The storage subsystem 1124 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 1124 may include memory devices for executing Figure 4 、 Figure 5 、 Figure 6 and / or Figure 7 The logic of the selected aspects of the method.

[0158] These software modules are generally executed by processor 1114 alone or in combination with other processors. The memory subsystem 1125 used in the storage subsystem 1124 may include a number of memories, including a main random access memory (RAM) 1130 for storing instructions and data during program execution and a read-only memory (ROM) 1132 for storing fixed instructions. The file storage subsystem 1126 may provide persistent storage for program and data files and may include a hard drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of certain embodiments may be stored by the file storage subsystem 1126 in the storage subsystem 1124, or in other machines accessible by the processor 1114.

[0159] The bus subsystem 1112 provides a mechanism for the various components and subsystems of the computing device 1110 to communicate with each other as intended. Although the bus subsystem 1112 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0160] The computing device 1110 may be of varying types, including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, Figure 11 The description of the computing device 1110 depicted in FIG is intended only as a specific example for purposes of illustrating some embodiments. Figure 11 Many other configurations of computing device 1110 are possible having more or fewer components than the computing device depicted in FIG.

[0161] In situations where certain embodiments discussed herein may collect or use personal information about a user (e.g., user data extracted from other electronic communications, information about the user's social network, the user's location, the user's time, the user's biometric information, and the user's activities and demographic information), the user is provided with one or more opportunities to control whether the information is collected, whether the personal information is stored, whether the personal information is used, and how the information is collected about the user, stored, and used. That is, embodiments of the systems and methods discussed herein collect, store, and / or use user personal information only upon receiving explicit authorization from the relevant user to do so. For example, a user is provided with control over whether a program or feature collects user information about that particular user or other users associated with that program or feature. Each user for whom personal information is to be collected is presented with one or more options allowing control over the collection of information related to that user, providing permission or authorization regarding whether information is to be collected and which portions of the information are to be collected. For example, one or more such control options may be provided to the user via a communication network. Furthermore, certain data may be processed in one or more ways before it is stored or used to remove personally identifiable information. As an example, a user's identity may be processed so that personally identifiable information cannot be determined. As another example, the user's geographic location may be generalized to a larger area so that the user's specific location cannot be determined.

Claims

1. A method implemented by one or more processors, the method comprising: receiving voice input from a user during a conversation between the person and the automated assistant; Converting the speech input into text using a speech-to-text module; determining that the text indicates a proxy action and the text fails to explicitly indicate a proxy to be invoked to perform the proxy action; Prior to invoking any agent in response to a natural language input instance generated based on the speech input, and in response to determining that the text fails to explicitly indicate an agent to be invoked to perform the agent action: Generate a proxy request based on the text, wherein the generated proxy request includes: an intent parameter indicating the intent indicated by the text, one or more values ​​for one or more intent bit parameters defined for the intent, and an indication that the proxy request should not immediately invoke a corresponding proxy; selecting a plurality of proxies for the proxy request; transmitting the proxy request to each of the plurality of proxies; receiving, in response to the transmitting, from each of the plurality of agents a corresponding response to the agent request, wherein the plurality of agents generate corresponding responses without invoking the respective agent based on the agent request including the indication that the agent request should not immediately invoke the respective agent, including the indication that the agent request should not immediately invoke the respective agent, and wherein the responses are respectively generated based on one or more values ​​of the intent parameter and the one or more intent bits parameters and respectively indicate an ability of a corresponding one of the plurality of agents to generate response content in response to the agent request; selecting a particular agent from the plurality of agents based on the corresponding responses, wherein selecting the particular agent occurs without providing any user interface output that explicitly solicits the user to select between the particular agent and one or more other agents from the plurality of agents; and In response to receiving the natural language input instance and in response to selecting the particular agent: A call request is transmitted to the specific agent, wherein the call request uses the intention parameter and one or more values ​​of the one or more intention bit parameters to call the specific agent, and causes the specific agent to generate specific response content for presentation to the user via one or more user interface output devices, wherein the specific agent is only called in response to receiving the voice input.

2. The method according to claim 1, wherein Selecting the particular agent is further based on an amount of interaction the user participating in the human-automated assistant conversation has with the particular agent.

3. The method according to claim 1, wherein Selecting the particular agent is further based on a recency of interaction between the user engaging in the human-automated assistant conversation and the particular agent.

4. The method according to claim 1, wherein The selection of the particular agent is further based on: the amount of interaction between the user and the particular agent who participated in the human-automated assistant conversation; and The recency of the user's interaction with the particular agent.

5. The method according to claim 1 , further comprising, after transmitting the call request to the specific agent: receiving the specific response content from the specific agent in response to the transmitting; and An output based on the specific response content is caused to be presented via the one or more user interface output devices.

6. The method according to claim 5, further comprising: receiving, while invoking the particular agent and after causing the output to be presented, additional natural language input instances, the additional natural language input instances being generated based on additional user interface input provided by the user; determining that the additional natural language input instance indicates a desire to switch to an alternative agent among the plurality of agents; as well as In response to determining that the additional natural language input instance indicates a desire to switch to the alternative agent: An additional invocation request is transmitted to the alternative agent, wherein the additional invocation request includes at least one value based on the natural language input instance or another natural language input instance received when invoking the particular agent.

7. A method implemented by one or more processors associated with an automated assistant agent, the method comprising: Receiving a proxy request from an automated assistant, wherein the proxy request is provided by the automated assistant in response to speech input detected by at least one microphone of a client device during a conversation between a human and the automated assistant, and wherein the proxy request comprises: an intent parameter indicating the intent indicated by the speech input, one or more values ​​for one or more intent bit parameters defined for the intent, and an indication that the agent request should not immediately invoke the automated assistant agent; Without invoking the automated assistant agent: generating a response to the proxy request, wherein the response indicates whether the automated assistant agent is capable of responding to the intent indicated by the intent parameter and to one or more values ​​of the one or more intent bit parameters, wherein generating the response is based on parsing the proxy request and determining whether the automated assistant agent is capable of responding to the intent indicated by the intent parameter and to one or more values ​​of the one or more intent bit parameters, and in response to receiving the proxy request, transmitting the response to the automated assistant; receiving an invocation request from the automated assistant in response to transmitting the response; In response to receiving the call request, generating response content based on the intent indicated by the intent parameter and one or more values ​​of the one or more intent bit parameters; and The response content is transmitted.

8. The method according to claim 7, wherein: Transmitting the response content includes transmitting the response content to the automated assistant so that the automated assistant presents the response content via one or more user interface output devices of the client device.

9. The method according to claim 7, wherein: Transmitting the response content includes: transmitting the response content to the client device.

10. The method according to claim 7, wherein: The response is a binary indication of whether the automated assistant agent is able to respond to the intent indicated by the intent parameter and to one or more values ​​of the one or more intent bit parameters.

11. The method according to claim 7, wherein: The response is a non-binary confidence measure of whether the automated assistant agent is able to respond to the intent indicated by the intent parameter and to one or more values ​​of the one or more intent bit parameters.

12. A system for calling an agent, comprising: a memory storing instructions; One or more processors, the one or more processors executing the instructions to cause a method to be performed, the method comprising: receiving voice input from a user during a conversation between the person and the automated assistant; Converting the speech input into text using a speech-to-text module; determining that the text indicates a proxy action and the text fails to explicitly indicate a proxy to be invoked to perform the proxy action; Prior to invoking any agent in response to a natural language input instance generated based on the speech input, and in response to determining that the text fails to explicitly indicate an agent to be invoked to perform the agent action: Generate a proxy request based on the text, wherein the generated proxy request includes: an intent parameter indicating the intent indicated by the text, one or more values ​​for one or more intent bit parameters defined for the intent, and an indication that the proxy request should not immediately invoke a corresponding proxy; selecting a plurality of proxies for the proxy request; transmitting the proxy request to each of the plurality of proxies; receiving, in response to the transmitting, from each of the plurality of agents a corresponding response to the agent request, wherein the plurality of agents generate corresponding responses without invoking the respective agent based on the agent request including the indication that the agent request should not immediately invoke the respective agent, including the indication that the agent request should not immediately invoke the respective agent, and wherein the responses are respectively generated based on one or more values ​​of the intent parameter and the one or more intent bits parameters and respectively indicate an ability of a corresponding one of the plurality of agents to generate response content in response to the agent request; selecting a particular agent from the plurality of agents based on the corresponding responses, wherein selecting the particular agent occurs without providing any user interface output that explicitly solicits the user to select between the particular agent and one or more other agents from the plurality of agents; and In response to receiving the natural language input instance and in response to selecting the particular agent: A call request is transmitted to the specific agent, wherein the call request uses the intention parameter and one or more values ​​of the one or more intention bit parameters to call the specific agent, and causes the specific agent to generate specific response content for presentation to the user via one or more user interface output devices, wherein the specific agent is only called in response to receiving the voice input.

13. The system according to claim 12, wherein: Selecting the particular agent is further based on an amount of interaction the user participating in the human-automated assistant conversation has with the particular agent.

14. The system according to claim 12, wherein: Selecting the particular agent is further based on a recency of interaction between the user engaging in the human-automated assistant conversation and the particular agent.

15. The system according to claim 12, wherein: The selection of the particular agent is further based on: the amount of interaction between the user and the particular agent who participated in the human-automated assistant conversation; and The recency of the user's interaction with the particular agent.

16. The system of claim 12, wherein: The method further includes, after transmitting the call request to the specific agent: receiving the specific response content from the specific agent in response to the transmitting; and An output based on the specific response content is caused to be presented via one or more user interface output devices.

17. The system according to claim 16, wherein: The method further comprises: receiving, while invoking the particular agent and after causing the output to be presented, additional natural language input instances, the additional natural language input instances being generated based on additional user interface input provided by the user; determining that the additional natural language input instance indicates a desire to switch to an alternative agent among the plurality of agents; and In response to determining that the additional natural language input instance indicates a desire to switch to the alternative agent: An additional invocation request is transmitted to the alternative agent, wherein the additional invocation request includes at least one value based on the natural language input instance or another natural language input instance received when invoking the particular agent.

18. A method implemented by one or more processors, comprising: receiving a natural language input instance generated based on user interface input provided by a user during a conversation between the human and the automated assistant, wherein the natural language input instance fails to explicitly indicate an agent to be invoked based on the natural language input instance; Before invoking any agents in response to the natural language input instance: Applying input features based on the natural language input instance as input to a trained machine learning model; generating an output comprising a plurality of values ​​on the machine learning model based on the input, wherein each value in the plurality of values ​​corresponds to a respective agent in a plurality of agents, and wherein each value in the plurality of values ​​indicates an ability of the respective agent in the plurality of agents to generate responsive content for the natural language input instance; selecting a particular agent from the plurality of agents based on the generated value, wherein selecting the particular agent occurs without providing any user interface output explicitly soliciting the user to select between the particular agent and one or more other agents of the plurality of agents; In response to receiving the natural language input instance and in response to selecting the particular agent: transmitting a call request to the specific agent via an application programming interface, wherein the call request invokes the specific agent and causes the specific agent to generate specific response content for presentation to the user via one or more user interface output devices, Wherein only the particular agent is invoked in response to receiving the natural language input instance.

19. The method according to claim 18, further comprising: applying one or more additional input features based on one or more context values ​​as additional inputs to the trained machine learning model, Wherein generating the output on the machine learning model is further based on the additional input.

20. The method according to claim 19, wherein The one or more context values ​​on which at least one of the one or more additional input features is based include a location of a client device via which the user interface input is received.

21. The method according to claim 19, wherein The one or more context values ​​on which at least one of the one or more additional input features is based include a given context value that is based on content currently rendered on a client device via which the user interface input is received.

22. The method according to claim 18, wherein Selecting the particular agent is further based on an amount of interaction the user participating in the human-automated assistant conversation has with the particular agent.

23. The method according to claim 18, wherein Selecting the particular agent is further based on a recency of interaction between the user engaging in the human-automated assistant conversation and the particular agent.

24. The method according to claim 18, wherein The selection of the particular agent is further based on: the amount of interaction between the user and the particular agent who participated in the human-automated assistant conversation; and The recency of the user's interaction with the particular agent.

25. The method according to claim 18, further comprising, after transmitting the call request to the specific agent: receiving the specific response content from the specific agent in response to the transmitting; and An output based on the specific response content is caused to be presented via the one or more user interface output devices.

26. The method of claim 25, further comprising: receiving, while invoking the particular agent and after causing the output to be presented, additional natural language input instances, the additional natural language input instances being generated based on additional user interface input provided by the user; determining that the additional natural language input instance indicates a desire to switch to an alternative agent among the plurality of agents; as well as In response to determining that the additional natural language input instance indicates a desire to switch to the alternative agent: An additional invocation request is transmitted to the alternative agent, wherein the additional invocation request includes at least one value based on the natural language input instance or another natural language input instance received when invoking the particular agent.

27. A method implemented by one or more processors, comprising: receiving a natural language input instance generated based on user interface input provided by a user during a conversation between the human and the automated assistant, wherein the natural language input instance fails to explicitly indicate an agent to be invoked based on the natural language input instance; Before invoking any agents in response to the natural language input instance: Select a specific agent from among multiple agents, The specific agent is selected based on: the natural language input instance and agent selection model, and one or more context values, the one or more context values ​​being external to the natural language input instance and to any other natural language input instances provided by the person during the conversation with the automated assistant, and wherein selecting the particular agent occurs without providing any user interface output explicitly soliciting the user to select between the particular agent and one or more other agents of the plurality of agents; In response to receiving the natural language input instance and in response to selecting the particular agent: transmitting a call request to the specific agent via an application programming interface, wherein the call request invokes the specific agent and causes the specific agent to generate specific response content for presentation to the user via one or more user interface output devices, Wherein only the particular agent is invoked in response to receiving the natural language input instance.

28. The method according to claim 27, wherein The one or more context values ​​include a given context value that is based on content currently rendered on a client device via which the user interface input is received.

29. The method according to claim 27, wherein The one or more context values ​​include a given context value that is based on an amount of interaction between the user engaging in the human-automated assistant conversation and the particular agent.

30. The method of claim 27, wherein: The one or more context values ​​include a given context value that is a recency of an interaction between the user engaging in the human-automated assistant conversation and the particular agent.

31. The method of claim 27, wherein: The one or more context values ​​include: a given context value, the given context value being based on an amount of interaction of the user with the particular agent; and A given context value is appended, the additional given context value being based on the recency of the user's interaction with the particular agent.

32. The method of claim 27, wherein: The one or more context values ​​include a location of a client device via which the user interface input is received.

33. A method implemented by one or more processors, comprising: receiving a natural language input instance generated based on user interface input provided by a user during a conversation between the human and the automated assistant, wherein the natural language input instance fails to explicitly indicate an agent to be invoked based on the natural language input instance; Before invoking any agents in response to the natural language input instance: Applying input features based on the natural language input instance as input to a trained machine learning model; generating an output comprising a plurality of values ​​on the machine learning model based on the input, wherein each value in the plurality of values ​​corresponds to a respective agent in a plurality of agents, and wherein each value in the plurality of values ​​indicates an ability of the respective agent in the plurality of agents to generate responsive content for the natural language input instance; selecting a subset of the plurality of agents based on the generated value; providing an indication of each agent of the selected subset of the plurality of agents for presentation to the user; receiving, in response to providing an indication of each of the plurality of agents of the selected subset, a selection of a particular agent of the plurality of agents of the selected subset, wherein the selection is based on further user interface input provided by the user; and selecting the particular agent based on receiving the selection; In response to receiving the natural language input instance and in response to selecting the particular agent: A call request is transmitted to the specific agent via an application programming interface, wherein the call request calls the specific agent and causes the specific agent to generate specific response content for presentation to the user via one or more user interface output devices.

34. The method according to claim 33, wherein Selecting the subset of the plurality of agents is further based on one or more context values ​​that are external to the natural language input instance and to any other natural language input instances provided during the conversation between the person and the automated assistant.

35. The method according to claim 34, wherein Selecting the subset of the plurality of agents further based on the one or more context values ​​comprises: applying one or more additional input features based on one or more context values ​​as additional inputs to the trained machine learning model; Wherein generating the output on the machine learning model is further based on the additional input.

36. The method of claim 33, wherein: Wherein selecting the subset of the plurality of agents is further based on a context value based on content currently rendered on a client device via which the user interface input is received.

Images