Condition-dependent scalable utility for automated assistants

Abstract

Implementations set forth herein relate to an automated assistant that may employ various utility programs according to a threshold degree of whether certain conditions are met for a given utility program and / or whether a particular input is determined to have a correlation with the given utility program. The automated assistant may be customized by a third-party entity that may make the automated assistant available via a device and / or application. The automated assistant may operate according to certain utility programs that have been allowed by the third party entity. However, when the user input is determined to have a threshold degree of correlation and / or when the use of certain extensible utility has not exceeded a threshold, the third party entity may allow the automated assistant to employ other extensible utility. The utility may refer to a machine learning model and / or other data that may be employed to digest the input.

Description

Background Art

[0001] A human may engage in a human-computer conversation with an interactive software application, referred to herein as an "automated assistant" (also referred to as a "digital agent," "chatbot," "interactive personal assistant," "intelligent personal assistant," "assistant application," "conversational agent," etc.). For example, a human (who may be referred to as a "user" when they interact with an automated assistant) may provide commands and / or requests to the automated assistant using spoken natural language input (i.e., utterances) and / or by providing textual (e.g., typed) natural language input, which in some cases may be converted to text and then processed.

[0002] In many cases, automated assistants are employed through application programming interfaces (APIs) with other applications, thereby allowing certain features of the automated assistant to be accessed via another application. For example, an entity that provides a third-party application for troubleshooting electrical appliances can utilize an assistant API to facilitate interaction between the assistant and a user who is accessing the third-party application. However, as a result of facilitating such interaction, the user may ask the automated assistant for help via the third-party application to receive help on a range of matters outside the scope of the third-party application. For example, a user may ask the automated assistant to read aloud the manual for a specific electrical appliance, and then ask the automated assistant for directions to a nearby electrical appliance store. However, when a third-party entity has not yet enlisted certain services of the automated assistant for providing directions, the automated assistant may respond with an error indication or non-substantial output. Therefore, the user's experience with such third-party applications may be reduced, and, in addition, the entity providing the third-party application may not know the frequency of rendering these error indications. Summary of the invention

[0003] The implementations set forth herein relate to an automated assistant that can provide services (i.e., utilities) according to such extended standards when extended standards are met. In this way, third-party entities seeking to incorporate assistant functionality into their respective devices and / or applications can do so based on the limitations of the resources that can be reserved for those third-party entities. For example, a third-party entity with a public storefront can provide access to a third-party device to the public when the public enters the public storefront. A third-party device (e.g., a third party relative to the party providing the automated assistant) can provide access to the automated assistant based on whether the third-party entity has authorized certain utilities of the automated assistant. For example, the public storefront can be a pet store, and the third-party entity can authorize animal health services, which can allow the automated assistant to answer health questions about a certain animal. The third-party entity can also authorize animal product services, which can allow the automated assistant to order animal products for the user. In some implementations, the third-party entity can choose to allow the automated assistant to answer user queries that may involve one or more other utilities that may not be explicitly registered by the third-party entity. However, these one or more other utilities can be utilized only according to the specified budget and / or other thresholds of the third-party entity. Alternatively or additionally, the one or more other utilities may be utilized solely based on semantic similarity between the content input by the user and utilities registered by third-party entities.Alternatively or additionally, the one or more other utilities may be utilized solely based on semantic similarity between the content input by the user and other data that may be associated with the third-party entity.

[0004] For example, when a user enters a storefront, the user may provide a spoken utterance to the automated assistant via a third-party device. The spoken utterance may be, for example, “Assistant, tell me the best dogfood for a Great Pyrenees”. Because the third-party entity explicitly authorizes the animal health utility (i.e., the plug-in, service, module, and / or other feature of the application) of its automated assistant, the automated assistant may utilize the animal health utility to generate a response (e.g., “A grain-free dogfood may be best for Great Pyrenees”). Thereafter, another user may provide another spoken utterance to the automated assistant, such as, for example, “Assistant, where is the nearest dog park? The automated assistant may determine that the utility currently registered by the third-party entity may not be applicable to generating a response to the other spoken utterance. In some implementations, the automated assistant may determine whether the other spoken utterance is semantically similar to the utility authorized by the third-party entity. Semantic similarity may be determined using one or more heuristic processes and / or one or more trained machine learning models.

[0005] In some implementations, a third-party entity may select a preference for an automated assistant to register an appropriate utility for responding to certain user inputs that are within a threshold degree of similarity to one or more utilities already registered by the third-party entity. Alternatively or in addition, the third-party entity may select another preference for the automated assistant to authorize the appropriate utility for responding to certain user inputs when the appropriate utility can be utilized without exceeding a budget or other threshold specified by the third-party entity. In this way, the utility of an automated assistant customized by a third party can be expanded based on feedback generated from interactions between a user and the customized automated assistant. This can allow any customized automated assistant to adaptively satisfy user queries. This may be particularly beneficial in the context of a customized automated assistant where the developer may not have anticipated that a user would provide queries related to an assistant utility that was not initially authorized by the developer.

[0006] The above description is provided as an overview of some implementations of the present disclosure. Further descriptions of those implementations and other implementations are described in more detail below.

[0007] Other implementations may include a non-transitory computer-readable storage medium storing instructions that are executable by one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU)) to perform methods such as one or more of the methods described above and / or elsewhere herein. Still other implementations may include a system of one or more computers including one or more processors operable to execute stored instructions to perform methods such as one or more of the methods described above and / or elsewhere herein.

[0008] It should be understood that all combinations of the aforementioned concepts and additional concepts described in more detail herein are considered part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are considered part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1A , Figure 1B and Figure 1C A diagram showing a user interacting with an automated assistant that employs an extensible utility based on whether one or more user queries satisfy certain conditions.

[0010] Figure 2A , Figure 2B and Figure 2C A diagram showing a user invoking an automated assistant that can utilize an extensible utility to digest input from the user when one or more conditions are determined to be satisfied.

[0011] Figure 3 A system for providing an automated assistant that can employ extensible utilities depending on whether certain conditions have been met is shown.

[0012] Figure 4 A method for extending the utility of an automated assistant based on whether a user query is semantically similar to an allowed utility and / or whether certain conditions for authorization of the extension of the utility have been met is shown.

[0013] Figure 5 is a block diagram of an example computer system. DETAILED DESCRIPTION

[0014] Figure 1A , Figure 1B and Figure 1CViews 100, 120, and 140 are shown of a user 102 interacting with an automated assistant that employs extensible utilities based on whether one or more user queries satisfy certain conditions. An automated assistant may be provided by an entity that may allow third-party entities to subscribe to certain automated assistant utilities to customize instances of the automated assistant. An automated assistant customized by a third-party entity may be incorporated into a service, device, and / or other interface that is made available by the third-party entity. However, the third-party entity may limit the scope of tasks handled by the automated assistant and / or limit the amount of resources consumed by an instance of the automated assistant.

[0015] For example, user 102 may enter a location 112 associated with a third-party entity, and location 112 may be a store with a computing device 104 for assisting customers entering location 112. In some implementations, a third-party entity (e.g., a deli) may employ an automated assistant with one or more utilities and / or services that allow the automated assistant to resolve certain user queries within a particular scope and / or subject matter. For example, computing device 104 may provide access to an automated assistant that may employ one or more utilities for resolving health and food-related queries. In this manner, when user 102 enters location 112 and has questions about certain products, the automated assistant may easily provide information without having to wait for another agent of the third-party entity to finish assisting another customer.

[0016] As an example, a user 102 may provide a spoken utterance 106, such as "Assistant, how much protein is in a calzone?", and the spoken utterance 106 may be received by the automated assistant via an audio interface (e.g., a microphone) of the computing device 104. In some implementations, input data 108 generated in response to receiving the spoken utterance 106 may be processed to determine whether the spoken utterance 106 is relevant to a utility of the automated assistant. The relevance of the spoken utterance 106 to the utility may be determined using one or more heuristic processes and / or one or more trained machine learning models. For example, the input data 108 representing the spoken utterance 106 may be processed using one or more trained machine learning models to generate input embedding data 110 that may be mapped to a latent space. The input embeddings represented by the input embedding data 110 may then be mapped to a latent space 122 having utility embeddings that may correspond to allowed utility embeddings 124 that may be employed by the automated assistant.

[0017] For example, Figure 1BAs shown, the latent space 122 may include various different embeddings that may correspond to allowed utilities and extensible utilities. In some implementations, the input embedding 126 may be mapped to the latent space 122 for determining whether the distance between the input embedding 126 and one or more allowed utility embeddings 124 in the latent space 122 satisfies an allowed utility threshold 128. In some implementations, the allowed utility threshold 128 may be a value and / or vector that may define an area in which the input embeddings may be compared. When the input embedding 126 is determined to have a distance within the allowed utility threshold 128, the input from the user 102 may be resolved according to the allowed utility of the automated assistant. Otherwise, when the distance does not satisfy the allowed utility threshold 128, a determination may be made as to whether the conditions for employing an extensible utility are met.

[0018] When the input embedding 126 is determined to be within a threshold distance from one or more of the allowed utility embeddings, the spoken utterance 106 from the user 102 can be digested using one or more of the allowed utilities. For example, one or more allowed utilities for digesting health and food related queries can be utilized to generate response data 144 from the spoken utterance 106. The response data 144 can characterize the output 142, such as "A cheese calzone may have atleast 20 grams of protein (cheese pie may contain at least 20 grams of protein)". However, when the input is determined to be not relevant enough to be digested by the allowed utilities, one or more conditions for utilizing the extensible utilities can be determined. In some implementations, the selection of a particular utility for generating the response data can be performed using a hierarchical process, optionally in combination with one or more other processes (e.g., using a heuristic process, one or more machine learning models, etc.). For example, there can be multiple categories of user queries, and each respective category of user queries can be associated with a particular set of utilities that can be used to digest the respective category of queries. When a user query is determined to correspond to a particular category of utilities (e.g., based on a heuristic process and / or a machine learning model), a threshold-based approach can be utilized to select a particular utility within the set of utilities for the particular category. In other words, a non-threshold-based approach can be utilized to determine an appropriate category for the user query, and a threshold-based approach can be utilized to select a particular utility for resolving the user query.

[0019] Figure 2A , Figure 2B and Figure 2CA view 200 is shown of an automated assistant that invokes an automated assistant that can resolve input from the user 202 with extensible utilities when one or more conditions are determined to be satisfied. The automated assistant can be employed by a third-party entity (e.g., a third party relative to a first party that provides the functionality of the automated assistant) that can select certain utilities for the automated assistant to utilize. In some implementations, each utility can be selected based on the budget of the third-party entity and / or the breadth of input topics that the third-party entity wants the automated assistant to handle. Alternatively or in addition, the third-party entity can select whether the automated assistant can employ other utilities for resolving other inputs that certain allowed utilities may not be able to resolve. In some implementations, the third-party entity can select whether these other inputs will be resolved based on the budget of the third-party entity, the relevance of each input to one or more allowed utilities, and / or any other conditions for employing the service.

[0020] For example, user 202 may provide spoken utterance 206 to computing device 204, which provides access to an automated assistant of a third-party entity. The third-party entity may be, for example, a deli with a storefront at location 212 that the user can reach by foot. Spoken utterance 206 may include a query to be resolved, such as "Assistant, how much is a flight to Italy right now?" In response, input data 208 may be generated from the captured input and input embedding data 210 may be generated from the input data 208 using one or more trained machine learning models and / or one or more other processes. Input embedding 226 may be generated from input embedding data 210 and mapped to latent space 222, such as Figure 2B 220. In some implementations, latent space 222 can include a mapping of allowed utility embeddings 224 and / or extensible utility embeddings 244. A distance between input embedding 226 and allowed utility embeddings 224 in the latent space, and / or another distance between input embedding 226 and extensible utility embeddings 244 in the latent space can be determined. For example, when the distance between input embedding 226 and one or more allowed utility embeddings 224 in the latent space satisfies extensible utility threshold 232, spoken utterance 206 from user 202 can be resolved by the automated assistant using extensible utility. Alternatively or in addition, when input embedding 226 is at least a threshold distance from allowed utility threshold 228, spoken utterance 206 from user 202 can be resolved by the automated assistant using extensible utility.

[0021] In some implementations, the selected specific extensible utility can be associated with a specific value for each use. The specific value can be compared to a value threshold associated with an account of a third-party entity that is providing access to the automated assistant at location 212. When the specific value meets the value threshold, the extensible utility can be used by the automated assistant to resolve the input from the user 202. For example, the extensible utility can be used to process the input data 208 to further generate response data that can characterize the output 242 for the automated assistant to render. The extensible utility can be used to generate response data that can be rendered as output 242 and / or to score the response data (e.g., score the relevance to the input). When the response data is considered to be applicable (e.g., having a threshold relevance to the input) to render in response to the spoken utterance 206 from the user 202, the output 242 can be rendered as, for example, "A round trip from Louisville to Rome, Italy is about $1,500 (the round trip cost from Louisville to Rome, Italy is about $1,500)". The output may be generated using a trip planning utility that may be available to a third party entity depending on whether certain conditions are met. The trip planning utility may utilize one or more trained machine learning models that are different from the allowed utilities for resolving health and food related queries. In some implementations, the scalable utility may utilize a trained machine learning model that is trained using training data that is different from other training data used to train another trained machine learning model for the allowed utility. Alternatively or in addition, the scalable utility may utilize a type of machine learning model (e.g., an ASR model) that is different from another type of model (e.g., an NLU model) utilized by the allowed utility.

[0022] Figure 3A system 300 for providing an automated assistant 304 that can employ extensible utilities depending on whether certain conditions have been met is shown. The automated assistant 304 can operate as part of an assistant application provided at one or more computing devices such as a computing device 302 and / or a server device. A user can interact with the automated assistant 304 via an assistant interface 320, which can be a microphone, a camera, a touch screen display, a user interface, and / or any other device capable of providing an interface between a user and an application. For example, a user can initialize the automated assistant 304 by providing speech, text, and / or graphical input to the assistant interface 320 to cause the automated assistant 304 to initiate one or more actions (e.g., provide data, control peripheral devices, access an agent, generate input and / or output, etc.). Alternatively, the automated assistant 304 can be initialized based on processing context data 346 using one or more trained machine learning models. The context data 346 can characterize one or more features of an environment in which the automated assistant 304 is accessible, and / or one or more features of a user who is predicted to want to interact with the automated assistant 304. The computing device 302 may include a display device, which may be a display panel, including a touch interface for receiving touch input and / or gestures to allow a user to control an application 344 of the computing device 302 via the touch interface. In some implementations, the computing device 302 may lack a display device, thereby providing an audible user interface output without providing a graphical user interface output. In addition, the computing device 302 may provide a user interface for receiving oral natural language input from a user, such as a microphone. In some implementations, the computing device 302 may include a touch interface and may not have a camera, but may optionally include one or more other sensors.

[0023] Computing device 302 and / or other third-party client devices may communicate with the server device via a network such as the Internet. In addition, computing device 302 and any other computing devices may communicate with each other via a local area network (LAN) such as a Wi-Fi network. Computing device 302 may offload computing tasks to a server device in order to conserve computing resources at computing device 302. For example, a server device may host automated assistant 304, and / or computing device 302 may transmit input received at one or more assistant interfaces 320 to a server device. However, in some implementations, automated assistant 304 may be hosted at computing device 302, and various processes that may be associated with automated assistant operations may be performed at computing device 302.

[0024] In various implementations, all or less than all aspects of automated assistant 304 may be implemented on computing device 302. In some of those implementations, aspects of automated assistant 304 are implemented via computing device 302 and may interface with a server device, which may implement other aspects of automated assistant 304. The server device may optionally serve multiple users and their associated assistant applications via multiple threads. In implementations in which all or less than all aspects of automated assistant 304 are implemented via computing device 302, automated assistant 304 may be an application separate from (e.g., installed “on top of”) an operating system of computing device 302, or may alternatively be implemented directly by (e.g., treated as an application of, but integral to, the operating system) an operating system of computing device 302.

[0025] In some implementations, automated assistant 304 may include an input processing engine 306 that may employ a plurality of different modules to process input and / or output of computing device 302 and / or server device. For example, input processing engine 306 may include a speech processing engine 308 that may process audio data received at assistant interface 320 to identify text embodied in the audio data. The audio data may be transmitted from, for example, computing device 302 to a server device in order to preserve computing resources at computing device 302. Additionally or alternatively, the audio data may be processed exclusively at computing device 302.

[0026] The process for converting audio data to text may include a speech recognition algorithm that may use a neural network and / or a statistical model to identify grouped audio data corresponding to a word or phrase. The text converted from the audio data may be parsed by a data parsing engine 310 and may be used as text data for an automated assistant 304, which may be used to generate and / or identify command phrases, intentions, actions, slot values, and / or any other content specified by a user. In some implementations, the output data provided by the data parsing engine 310 may be provided to a parameter engine 312 to determine whether the user has provided input corresponding to a specific intention, action, and / or routine that can be performed by the automated assistant 304 and / or an application or agent that can be accessed via the automated assistant 304. For example, assistant data 348 may be stored at a server device and / or a computing device 302, and may include data defining one or more actions that the automated assistant 304 can perform, and parameters necessary for performing these actions. The parameter engine 312 may generate one or more parameters for intentions, actions, and / or slot values, and provide the one or more parameters to an output generation engine 314. Output generation engine 314 may use the one or more parameters to communicate with assistant interface 320 for providing output to a user, and / or to communicate with one or more applications 344 for providing output to one or more applications 344 .

[0027] In some implementations, automated assistant 304 may be an application that may be installed “on top of” an operating system of computing device 302, and / or may itself form part of (or the entirety of) the operating system of computing device 302. The automated assistant application includes and / or has access to on-device speech recognition, on-device natural language understanding, and on-device fulfillment. For example, on-device speech recognition may be performed using an on-device speech recognition module that processes audio data (detected by a microphone) using an end-to-end speech recognition machine learning model stored locally at computing device 302. On-device speech recognition generates recognized text for spoken utterances (if any) present in the audio data. In addition, for example, on-device natural language understanding (NLU) may be performed using an on-device natural language understanding (NLU) module that processes the recognized text generated using on-device speech recognition and optionally context data to generate NLU data.

[0028] The NLU data may include an intent corresponding to the spoken utterance and optionally parameters for the intent (e.g., slot values). On-device fulfillment may be performed using an on-device fulfillment module that utilizes the NLU data (from the on-device NLU) and optionally other local data to determine actions to be taken to resolve the intent (and optionally parameters for the intent) of the spoken utterance. This may include determining local and / or remote responses (e.g., replies) to the spoken utterance, interactions with locally installed applications performed based on the spoken utterance, commands transmitted to an Internet of Things (IoT) device (directly or via a corresponding remote system) based on the spoken utterance, and / or other resolution actions performed based on the spoken utterance. The on-device fulfillment may then initiate local and / or remote execution / execution of the determined actions to resolve the spoken utterance.

[0029] In various implementations, remote speech processing, remote NLU, and / or remote fulfillment may be at least selectively utilized. For example, the identified text may be at least selectively transmitted to a remote automated assistant component for remote NLU and / or remote fulfillment. For example, the identified text may be optionally transmitted for remote execution in parallel with on-device execution or in response to failure of on-device NLU and / or on-device fulfillment. However, on-device speech processing, on-device NLU, on-device fulfillment, and / or on-device execution may be prioritized at least due to the reduced latency they provide when resolving spoken utterances (because resolving spoken utterances does not require client-server round trips). In addition, on-device functionality may be the only functionality available when there is no network connection or when network connection is limited.

[0030] In some implementations, computing device 302 may include one or more applications 344, which may be provided by a third-party entity that is different from the entity that provided computing device 302 and / or automated assistant 304. Automated assistant 304 and / or an application state engine of computing device 302 may access application data 340 to determine one or more actions that one or more applications 344 are capable of performing, as well as a state of each of one or more applications 344 and / or a state of a corresponding device associated with computing device 302. Automated assistant 304 and / or a device state engine of computing device 302 may access device data 342 to determine one or more actions that computing device 302 and / or one or more devices associated with computing device 302 are capable of performing. Additionally, application data 340 and / or any other data (e.g., device data 342) can be accessed by automated assistant 304 to generate context data 346, which can characterize the context in which a particular application 344 and / or device is executing, and / or the context in which a particular user is accessing computing device 302, accessing application 344, and / or any other device or module.

[0031] While one or more applications 344 are executing at computing device 302, device data 342 may characterize the current operational state of each application 344 being executed at computing device 302. Additionally, application data 340 may characterize one or more features of the executing applications 344, such as the content of one or more graphical user interfaces being rendered at the direction of one or more applications 344. Alternatively or additionally, application data 340 may characterize action paradigms based on the current operational state of the respective applications, which action paradigms may be updated by the respective applications and / or by automated assistant 304. Alternatively or additionally, one or more action paradigms of one or more applications 344 may remain static, but may be accessed by the application state engine in order to determine appropriate actions to initiate via automated assistant 304.

[0032] The computing device 302 may further include an assistant invocation engine 322, which may use one or more trained machine learning models to process application data 340, device data 342, context data 346, and / or any other data accessible to the computing device 302. The assistant invocation engine 322 may process the data to determine whether to wait for the user to explicitly say an invocation phrase to invoke the automated assistant 304, or to treat the data as indicating the user's intention to invoke the automated assistant, instead of requiring the user to explicitly say the invocation phrase. For example, one or more trained machine learning models may be trained using instances of training data based on scenarios in which the user is in an environment where multiple devices and / or applications are exhibiting various operating states. Instances of training data may be generated to capture training data that characterizes contexts in which the user invokes the automated assistant and other contexts in which the user does not invoke the automated assistant. When training one or more trained machine learning models based on these instances of training data, the assistant invocation engine 322 may cause the automated assistant 304 to detect or limit detection of spoken invocation phrases from the user based on features of the context and / or environment. Additionally or alternatively, assistant invocation engine 322 can cause automated assistant 304 to detect or limit detection for one or more assistant commands from a user based on characteristics of the context and / or environment. In some implementations, assistant invocation engine 322 can be disabled or limited based on computing device 302 detecting an assistant suppression output from another computing device. In this manner, when computing device 302 is detecting an assistant suppression output, automated assistant 304 will not be invoked based on context data 346, which would otherwise cause automated assistant 304 to be invoked if no assistant suppression output was detected.

[0033] In some implementations, the system 300 may include an input embedding engine 316 that can operate with the input processing engine 306 to generate embeddings from input received by the automated assistant 304. The input can be, for example, a spoken utterance provided by a user, and the input embedding engine 316 can process the input to generate an input embedding. In some implementations, the input embedding can be generated using one or more trained machine learning models and / or one or more heuristic processes to transform the input into data that can be mapped to a latent space.

[0034] In some implementations, the system 300 may include an input relevance engine 318 that can map the input embedding generated by the input embedding engine 316 to a potential space with other embeddings. The other embeddings may correspond to the allowed utilities and extensible utilities of the automated assistant 304. The allowed utilities and extensible utilities may be parts of the data and / or modules that can be employed by the automated assistant 304 to resolve the input. The allowed utilities may be explicitly authorized by the account of the third-party entity that is providing access to the automated assistant 304. The extensible utilities may be authorized by the account of the third-party entity when one or more conditions are met. When the input embedding is generated by the input embedding engine 316, the input relevance engine 318 may determine whether the input is semantically related to the allowed utilities to a threshold degree. In some implementations, the degree of semantic relevance may be based on the distance between the input embedding and the allowed utility embedding in the potential space. When the distance meets the threshold, it may be determined that the input is related to the allowed utility embedding, and therefore, the corresponding allowed utility may be utilized to resolve the input. For example, an allowed utility engine 326 of system 300 may be employed to select an appropriate allowed utility (e.g., an ASR model) to assist in dissolving the input. Allowed utility engine 326 may select an allowed utility based on a distance between an input embedding and / or one or more allowed utility embeddings in a latent space. Alternatively or in addition, allowed utility engine 326 may select an allowed utility to employ based on a frequency of use of the allowed utility, information associated with a user of the automated assistant, features of a third-party entity that has made the automated assistant available, and / or any other information that may be utilized to select a utility for an application.

[0035] When the distance does not meet the threshold, the input relevance engine 318 can provide an indication to the extensible utility engine 324 for determining whether to utilize the extensible utility to resolve the input. In some implementations, the extensible utility engine 324 can determine whether one or more conditions for the third-party entity to adopt the extensible utility are met. One or more conditions can be based on one or more thresholds set by the third-party entity, such as a value threshold and / or a semantic relevance threshold. The value threshold can correspond to the value and / or amount of resources that the third-party entity wants to limit the consumption of the automated assistant. For example, an entity providing an automated assistant application can charge a fee of a specific value for adopting certain utilities, and the third-party entity can limit the fee to a certain threshold within a certain duration. For example, a third-party entity can limit the automated assistant to charge a fee exceeding a specific value threshold (e.g., 100 tokens) within a certain duration (e.g., per month). Alternatively or in addition, the third-party entity can limit the automated assistant to respond to queries that are not relevant to a certain degree. For example, the automated assistant may not attempt to resolve certain input queries that are determined to be not within a threshold degree of relevance to any allowed utility and / or extensible utility. Conversely, in response to receiving input that is determined not to be relevant to an allowed utility to a threshold degree, the automated assistant can cause an indication to be rendered with information for another application and / or other source of help for resolving the input.

[0036] In some implementations, the extensible utility engine 324 can determine the number of user queries that have been provided to the automated assistant associated with the account and / or third-party entity but have not yet been resolved by the automated assistant. In some cases, the user query may not have been resolved because the third-party entity has chosen not to set the threshold accordingly, and / or has not yet provided a budget that will allow the resolution of all queries. In some implementations, if the specific settings and / or thresholds for the automated assistant 304 are adjusted, the extensible utility engine can determine the number of individual user queries that may have been resolved by the automated assistant 304. In some implementations, the individual number can be a total number and / or percentage value that can be rendered at an interface accessible to the agent of the third-party entity. In this way, the third-party entity can receive feedback from its customized automated assistant 304 and adjust the threshold to allow the use of other extensible utilities to resolve user queries that may be frequently provided by the third-party entity's customers. Alternatively or in addition, the third-party entity can allow certain extensible utilities to be utilized by its specific automated assistant 304, thereby designating "extensible" utilities as "allowed" utilities. Thereafter, the newly enabled utility may be employed by automated assistant 304 to resolve additional user input that may not have been previously resolved by automated assistant 304 .

[0037] Figure 4A method 400 for extending the utility of an automated assistant based on whether a user query is semantically similar to an allowed utility and / or whether certain conditions for authorization of the extension of the utility have been met is shown. The method 400 may be performed by one or more computing devices, applications, and / or any other device or module that may be associated with the automated assistant. The method 400 may include an operation 402 of determining whether an assistant input has been received by the automated assistant from a user of the automated assistant. The automated assistant may optionally be an application customized by a third-party entity (e.g., a third party relative to a provider of the automated assistant) to provide certain services to users interacting with the third-party entity. For example, the third-party entity may be an entity that provides a service and / or device, and the third-party entity may choose to incorporate the automated assistant (e.g., via an application programming interface (API), a plug-in, etc.) into its service and / or device. In some implementations, the service may be a third-party application, and the automated assistant may be incorporated into the third-party application, thereby allowing users of the third-party application to interact with the customized automated assistant of the third-party application.

[0038] When one or more assistant inputs are received from one or more users of the automated assistant, third-party application, and / or third-party device, method 400 may proceed from operation 402 to operation 404. Otherwise, method 400 may continue to determine whether an assistant input has been received at operation 402. Operation 404 may include determining whether one or more allowed utilities are capable of resolving inputs from a user. The input data may characterize one or more assistant inputs from one or more users, and one or more allowed utilities may include utilities that a third-party entity has explicitly selected to incorporate into its customized automated assistant. For example, an allowed utility may refer to one or more ASR models, one or more NLU models, one or more other trained machine learning models, one or more heuristic processes, and / or any other features that may be utilized by an automated assistant to process inputs. When the third-party entity is, for example, a computer store, utilities explicitly allowed by the computer store may include language models for responding to questions about computer hardware and / or for processing requests to order computer parts. Thus, when a user (e.g., a customer at a computer store) provides user input querying an automated assistant regarding a particular computer part, that user input may be resolved by an allowed utility, whereas a separate user input regarding a question regarding Newtonian physics may not be resolved by an allowed utility.

[0039] In some implementations, determining the degree of semantic similarity between a user input and a particular assistant utility is performed to determine whether an allowed utility can resolve the user input. Determining the degree of semantic similarity may involve one or more trained machine learning models and / or one or more heuristic processes. For example, one or more trained machine learning models may be utilized to process input data and generate embedding data from the input data. The embedding data may characterize one or more input embeddings that may be mapped to a latent space with utility embeddings. The utility embeddings may represent embeddings, and each respective embedding may correspond to a utility that a particular account and / or entity has authorized and / or otherwise has access to through extensions. The distance between the input embedding and the utility embedding in the latent space may indicate the relevance of the corresponding user input (e.g., a customer asking a question about a computer) to a particular utility (e.g., a utility that an automated assistant employs to resolve computer-related problems).

[0040] When the allowed utility is determined to be unable to resolve the input, the method 400 may proceed from operation 404 to operation 406. Otherwise, the method 400 may proceed from operation 404 to operation 410. Operation 406 may include determining whether one or more conditions for adopting one or more extensible utilities are met. In some implementations, the conditions for adopting the extensible utility may be based on a value or other limit set by a third-party entity for generating a fee that may be associated with utilizing the extensible utility. Alternatively or in addition, the conditions for adopting the extensible utility may be based on the degree of relevance of a particular user input to the allowed utility and / or the extensible utility. In other words, if the input is not at least related to the allowed utility to a threshold degree, the third-party entity may not want to adopt the extensible utility. Alternatively or in addition, if the input is not at least related to the extensible utility (e.g., a utility that the third-party entity may not have explicitly allowed when receiving the user input) to a threshold degree, the third-party entity may not want to adopt the extensible utility. In some implementations, determining whether the condition is met may include determining whether the input data meets a threshold degree of relevance to the allowed utility. For example, the distance from the input embedding to the allowed utility embedding in the latent space may be determined. When the distance from the input embedding to the allowed utility embedding satisfies the distance threshold, the method 400 may proceed from operation 406 to operation 408. Otherwise, the method 400 may continue from operation 408 and return to operation 402, or optionally perform an operation of notifying a user or a third-party entity of the lack of a relevant utility. Alternatively or in addition, the method 400 may optionally include an operation of providing information to the user and / or third-party entity about information sources that can help assist in responding to one or more queries embodied in the input data. For example, the suggestion output may be rendered with an identifier for a separate application and / or a separate information source to solicit a response to the user input.

[0041] In some implementations, operation 410 may include generating response data using one or more allowed utilities, and operation 408 may include generating other response data using one or more extensible utilities. Utilities employed for operation 410 and / or operation 408 may include trained machine learning models, heuristic processes, and / or any other features of applications that can help process data. For example, an allowed utility may include a trained machine learning model trained using data associated with computer science information such as technical specifications. As another example, an extensible utility may include a different trained machine learning model trained using other data associated with physical information such as educational materials for learning physics. Response data generated using a particular utility may embody a response that an automated assistant may provide in response to a user input from a user. For example, a user input such as "Assistant, is there a wireless mouse in stock?" may be processed using an allowed utility for processing computer-related order requests. Alternatively, another user input such as "Assistant, who invented the infrared sensor?" may be processed using an extensible utility for processing history-related computer questions.

[0042] Method 400 may proceed from operation 410 or operation 408 to operation 412 that causes generation of a response output based on the generated response data. For example, when receiving user input about "wireless mouse", the automated assistant may provide a response output, such as "Yes, there are multiple styles available in aisle 9". Alternatively, when receiving other user input about "infrared sensor", the automated assistant may provide different response outputs attributed to different utilities and / or assistants, such as "Herbert Berman may have been the first to invent a type of infrared sensor". However, in some cases, when one or more conditions are not met, the other user input may not be resolved, and data indicating the following may be provided to a third-party entity instead: if certain conditions are modified, the other user input may have been processed. For example, it may be determined that the amount by which a threshold or limit for a particular condition should be changed in order to allow the automated assistant to respond to such user input.

[0043] The third-party entity may then make a determination as to whether to adjust the threshold or limit (e.g., adjust its budget limit, adjust its semantic similarity threshold, etc.) to allow similar user input to be processed using one or more extensible utilities. In some implementations, the data may be presented to the third-party entity at a visual interface of a computing device, thereby allowing the third-party entity to visualize how changes in certain thresholds and / or limits allow its automated assistant to process queries that may be of frequent interest to users (e.g., customers of the third-party entity). For example, the automated assistant may cause the interface of the computing device to render an updated value for the semantic similarity threshold (e.g., 5 points lower than the current value) and an indication that the percentage of user input that will be resolved will increase (e.g., "an increase of 15%") as a result of the third-party entity adjusting the semantic similarity threshold to the updated value. Alternatively or in addition, any unresolved queries may be processed to identify summary queries, which may be presented at the interface with thresholds that may be adjusted to show the third-party entity the types of queries that may be resolved if certain thresholds are adjusted. For example, some users may have provided queries asking for directions to different locations, and the interface may render summaries of these queries, such as "navigation-related queries." The summary may be rendered with a threshold, and when the threshold is adjusted to another limit of "navigation related queries", the automated assistant will be allowed to use utility to resolve subsequent navigation related queries.

[0044] Figure 5 5 is a block diagram 500 of an example computer system 510. The computer system 510 generally includes at least one processor 514 that communicates with a number of peripheral devices via a bus subsystem 512. These peripheral devices may include a storage subsystem 524 (including, for example, a memory 525 and a file storage subsystem 526), ​​a user interface output device 520, a user interface input device 522, and a network interface subsystem 516. The input devices and output devices allow a user to interact with the computer system 510. The network interface subsystem 516 provides an interface to an external network and is coupled to corresponding interface devices in other computer systems.

[0045] The user interface input device 522 may include a keyboard, a pointing device (such as a mouse, a trackball, a touch pad or a graphics tablet, a scanner, a touch screen incorporated into a display), an audio input device (such as a voice recognition system, a microphone), and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways of inputting information into the computer system 510 or onto a communication network.

[0046] The user interface output device 520 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display such as via an audio output device. In general, the use of the term "output device" is intended to include devices and methods of all possible types of information output from the computer system 510 to a user or another machine or computer system.

[0047] Storage subsystem 524 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, storage subsystem 524 may include logic for performing selected aspects of method 300 and / or implementing one or more of system 400, computing device 104, automated assistant, and / or any other application, device, apparatus, and / or module discussed herein.

[0048] These software modules are typically executed by the processor 514 alone or in combination with other processors. The memory 525 used in the storage subsystem 524 may include multiple memories, including a main random access memory (RAM) 540 for storing instructions and data during program execution and a read-only memory (ROM) 542 in which fixed instructions are stored. The file storage subsystem 526 can provide persistent storage for program and data files, and can include a hard disk drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of certain implementations can be stored by the file storage subsystem 526 in the storage subsystem 524, or stored in other machines accessible to the processor 514.

[0049] The bus subsystem 512 provides a mechanism for the various components and subsystems of the computer system 510 to communicate with each other as intended. Although the bus subsystem 512 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0050] Computer system 510 can be of different types, including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, Figure 5 The description of the computer system 510 depicted in FIG. 5 is intended only as a specific example for illustrating some implementations. Many other configurations of the computer system 510 are possible, with similar configurations. Figure 5 The computer systems may have more or fewer components than those depicted.

[0051] In cases where the systems described herein collect personal information about users (or "participants" as often referred to herein) or may make use of personal information, users may be provided with the opportunity to control whether a program or feature collects user information (e.g., information about a user's social network, social behavior or activities, occupation, a user's preferences, or a user's current geographic location), or to control whether and / or how content is received from a content server that may be more relevant to the user. In addition, certain data may be processed in one or more ways before it is stored or used so that personally identifiable information is removed. For example, the identity of a user may be processed so that personally identifiable information about the user cannot be determined, or, in the case of obtaining geographic location information, the geographic location of the user may be generalized (such as to a city, zip code, or state level) so that the specific geographic location of the user cannot be determined. Thus, users may control how information about the user is collected and / or used.

[0052] Although several implementations have been described and illustrated herein, a variety of other means and / or structures for performing functions and / or obtaining results and / or one or more of the advantages described herein may be utilized, and each of such changes and / or modifications is considered to be within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and the actual parameters, dimensions, materials, and / or configurations will depend on one or more specific applications for which this teaching is used. Those skilled in the art will recognize or confirm many equivalents of the implementations described herein using only routine experiments. Therefore, it should be understood that the aforementioned implementations are presented only by way of example, and it should be understood that within the scope of the attached claims and their equivalents, implementations may be practiced in a manner other than specifically described and claimed. The implementations disclosed herein relate to each individual feature, system, article, material, kit, and / or method described herein. In addition, if such features, systems, articles, materials, kits, and / or methods do not contradict each other, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of the present disclosure.

[0053] In some implementations, a method implemented by one or more processors is described as including one or more operations, such as receiving input data indicating that user input has been provided to a computing device that provides access to an automated assistant. The computing device is associated with an account that is authorized to employ one or more allowed utilities for generating response data based on user input, and when one or more conditions for employing one or more extensible utilities are met, the automated assistant operates to employ the one or more extensible utilities. The operation may further include determining, based on the input data and using one or more trained machine learning models, a semantic relevance of the input data to one or more allowed utilities for generating response data. The method may further include determining, based on the semantic relevance, that the input data is not semantically related to one or more allowed utilities for generating response data to a threshold degree. The method may further include determining whether one or more conditions for employing one or more extensible utilities are met based on the input data not being semantically related to one or more allowed utilities to a threshold degree. The method may further include, when it is determined that one or more conditions for employing one or more extensible utilities are met: causing a specific utility in the one or more extensible utilities to be utilized for generating other response data for responding to the user input based on the user input; and causing the computing device or a separate computing device to render a response output using the other response data.

[0054] In some implementations, the method may include causing a suggestion output to be rendered by the computing device or a separate computing device when one or more conditions for employing the one or more extensible utilities are not met. The suggestion output identifies a separate application for soliciting a response to a user input. In some implementations, one or more allowed utilities employ a particular data storage model, and one or more extensible utilities employ a different data storage model. In some implementations, one or more allowed utilities employ a particular natural language understanding model, and one or more extensible utilities employ a different natural language model. In some implementations, one or more conditions for employing the one or more extensible utilities are characterized by conditional data provided by an entity that provides access to the one or more allowed utilities.

[0055] In some implementations, the semantic relevance of the input data to the one or more allowed utilities is based on a distance between the input embedding and the allowed utility embedding in a latent space, and the input embedding is generated using one or more trained machine learning models and the input data. In some implementations, when the distance in the latent space does not meet a relevance threshold for the one or more allowed utilities, the input data is not semantically related to the one or more allowed utilities used to generate the response data to a threshold degree. In some implementations, determining whether one or more conditions for employing the one or more extensible utilities are based on the distance between the input embedding and the allowed utility embedding in the latent space. In some implementations, the method may include, when determining that the one or more conditions for employing the one or more extensible utilities are met: causing the response output to be rendered with an indication that the one or more extensible utilities are utilized to generate the response output. In some implementations, determining whether the one or more conditions for employing the one or more extensible utilities are met includes: determining whether a value metric associated with the one or more extensible utilities meets a value threshold stored in association with the account, wherein when the value metric meets the value threshold, the one or more conditions are met.

[0056] In other implementations, a method implemented by one or more processors is described as including one or more operations, such as processing assistant data generated based on receiving a user query by an automated assistant accessible via one or more computing devices associated with an account. The automated assistant responds to the user query based on whether one or more allowed utilities are authorized by the account. The method may further include determining, based on processing the assistant data, the number of user queries provided to one or more computing devices associated with the account that have been received by the automated assistant but have not yet been resolved by the automated assistant using one or more allowed utilities authorized by the account. The method may further include determining, based on the number of user queries, a separate number of user queries that the automated assistant can resolve using one or more extensible utilities that are not currently authorized by the account. The method may further include causing content to be rendered at an interface of a specific computing device associated with the account based on the separate number of user queries. The content indicates a separate number of user queries that can be resolved by the automated assistant when the account authorizes one or more extensible utilities.

[0057] In some implementations, the method can include determining an extent to which an extensible utility threshold can be adjusted that will allow the automated assistant to resolve the user query using one or more extensible utilities. In some implementations, determining an extent to which a value threshold can be adjusted that will allow the automated assistant to resolve the user query using one or more extensible utilities.

[0058] In yet other implementations, a method implemented by one or more processors is described as including one or more operations, such as receiving input data indicating that user input has been provided to a computing device that provides access to an automated assistant. The computing device is associated with an account that is authorized to employ one or more allowed utilities for generating response data based on the user input, and when one or more conditions for employing one or more extensible utilities are met, the automated assistant operates to employ the one or more extensible utilities. The method may further include determining, based on the input data and using one or more trained machine learning models, a semantic relevance of the input data to one or more allowed utilities for generating response data. The method may further include determining, based on the semantic relevance, whether the input data is semantically related to one or more allowed utilities for generating response data to a threshold degree. The method may further include, when it is determined that the input data is not semantically related to one or more allowed utilities for generating response data to a threshold degree: based on the input data not being semantically related to one or more allowed utilities to a threshold degree, determining whether one or more conditions for employing one or more extensible utilities are met; and when one or more conditions for employing one or more extensible utilities are met, causing a specific utility in the one or more extensible utilities to be utilized for generating a response output.

[0059] In some implementations, the method includes causing, by the computing device or a separate computing device, to render a suggestion output when one or more conditions for employing the one or more extensible utilities are not met. The suggestion output identifies a separate application for soliciting a response to the user input. In some implementations, one or more allowed utilities employ a particular natural language understanding model and one or more extensible utilities employ a different natural language understanding model. In some implementations, the one or more conditions for employing the one or more extensible utilities are characterized by conditional data provided by an entity that provides access to the one or more allowed utilities.

[0060] In some implementations, the semantic relevance of the input data to one or more allowed utilities is based on a distance between an embedding of the input and the embedding of the allowed utilities in a latent space, and the input embedding is generated using one or more trained machine learning models and the input data. In some implementations, when the distance in the latent space does not meet an allowed utility threshold for one or more allowed utilities, the input data is not semantically related to the one or more allowed utilities used to generate the response data to a threshold degree. In some implementations, determining whether one or more conditions for employing one or more extensible utilities are based on a distance between the embedding of the input and the embedding of the allowed utilities in the latent space. In some implementations, the method may further include causing the response output to be rendered with an indication that the one or more extensible utilities were utilized to generate the response output when it is determined that the one or more conditions for employing the one or more extensible utilities are met.

Claims

1. A method implemented by one or more processors, the method comprising: receiving input data indicating that user input has been provided to a computing device providing access to an automated assistant, wherein the computing device is associated with an account that is authorized to employ one or more permitted utilities for generating response data based on user input, and wherein when one or more conditions for employing one or more extensible utilities are met, the automated assistant operates to employ the one or more extensible utilities; determining, based on the input data and using one or more trained machine learning models, a semantic relevance of the input data to the one or more allowed utilities for generating the response data; determining, based on the semantic relevance, that the input data is not semantically related to the one or more allowed utilities for generating the response data to a threshold degree; determining whether the one or more conditions for employing the one or more extensible utilities are satisfied based on the input data not being semantically related to the one or more allowed utilities to the threshold degree; and In response to determining that the one or more conditions for employing the one or more extensible utilities are satisfied: causing a particular utility of the one or more extensible utilities to be utilized for generating, based on the user input, further response data for responding to the user input; and The further response data is caused to be used by the computing device or a separate computing device to render a responsive output.

2. The method of claim 1, further comprising: In response to determining that the one or more conditions for employing the one or more extensible utilities are not satisfied: causing rendering of a suggestion output by the computing device or the separate computing device, The suggestion output identifies a separate application for soliciting a response to the user input.

3. A method as claimed in any preceding claim, wherein: The one or more allowed utilities employ a particular natural language understanding model, and the one or more extensible utilities employ a different natural language understanding model.

4. A method as claimed in any preceding claim, wherein: The one or more conditions for employing the one or more extensible utilities are characterized by condition data provided by an entity providing access to the one or more allowed utilities.

5. A method as claimed in any preceding claim, in, The semantic relevance of the input data to the one or more allowed utilities is based on a distance between an input embedding and an allowed utility embedding in a latent space, and Wherein, the input embedding is generated using the one or more trained machine learning models and the input data.

6. The method of claim 5, wherein: When the distance in the latent space does not satisfy a relevance threshold for the one or more allowed utilities, the input data is not semantically related to the one or more allowed utilities used to generate the response data to the threshold degree.

7. The method of claim 6, wherein: Determining whether the one or more conditions for employing the one or more scalable utilities are satisfied is based on the distance between the input embedding and the allowed utility embeddings in the latent space.

8. The method of any preceding claim, further comprising: When it is determined that the one or more conditions for employing the one or more extensible utilities are met: The responsive output is caused to be rendered with an indication that the one or more extensible utilities were utilized to generate the responsive output.

9. The method according to any one of claims 1 to 6 or 8, wherein Determining whether the one or more conditions for employing the one or more extensible utilities are met includes: determining whether a value metric associated with the one or more extensible utilities satisfies a value threshold stored in association with the account, Wherein, the one or more conditions are satisfied when the value metric satisfies the value threshold.

10. A method implemented by one or more processors, the method comprising: processing assistant data generated based on a user query received by an automated assistant accessible via one or more computing devices associated with the account, wherein the automated assistant responds to the user query based on whether one or more allowed utilities are authorized by the account; determining, based on processing the assistant data, a number of user queries provided to the one or more computing devices associated with the account that have been received by the automated assistant but not yet resolved by the automated assistant using the one or more allowed utilities authorized by the account; determining, based on the number of user queries, a separate number of the user queries that the automated assistant can resolve using one or more extensible utilities that are not currently authorized by the account; and causing content to be rendered at an interface of a particular computing device associated with the account based on the separate number of the user queries, wherein the content indicates the individual number of user queries that can be resolved by the automated assistant when the account authorizes the one or more extensible utilities.

11. The method of claim 10, further comprising: A determination is made as to the extent to which an extensible utility threshold can be adjusted that will allow the automated assistant to resolve the user query using the one or more extensible utilities.

12. The method according to claim 10, A determination is made as to the extent to which a value threshold can be adjusted that will allow the automated assistant to resolve the user query using the one or more extensible utilities.

13. A method implemented by one or more processors, the method comprising: receiving input data indicating that user input has been provided to a computing device providing access to an automated assistant, wherein the computing device is associated with an account that is authorized to employ one or more permitted utilities for generating response data based on user input, and wherein when one or more conditions for employing one or more extensible utilities are met, the automated assistant operates to employ the one or more extensible utilities; determining, based on the input data and using one or more trained machine learning models, a semantic relevance of the input data to the one or more allowed utilities for generating the response data; determining, based on the semantic relevance, whether the input data is semantically related to the one or more allowed utilities for generating the response data to a threshold degree; and In response to determining that the input data is not semantically related to the one or more allowed utilities used to generate the response data to a threshold degree: determining whether the one or more conditions for employing the one or more extensible utilities are satisfied based on the input data not being semantically related to the one or more allowed utilities to the threshold degree; and In response to determining that the one or more conditions for employing the one or more extensible utilities are satisfied, a particular utility of the one or more extensible utilities is caused to be utilized for generating a responsive output.

14. The method of claim 13, further comprising: In response to determining that the one or more conditions for employing the one or more extensible utilities are not satisfied, causing rendering of a suggestion output by the computing device or the separate computing device, The suggestion output identifies a separate application for soliciting a response to the user input.

15. The method according to claim 13 or 14, wherein: The one or more allowed utilities employ a particular natural language understanding model, and the one or more extensible utilities employ a different natural language understanding model.

16. The method according to any one of claims 13 to 15, wherein: The one or more conditions for employing the one or more extensible utilities are characterized by condition data provided by an entity providing access to the one or more allowed utilities.

17. The method according to any one of claims 13 to 15, in, determining the semantic relevance of the input data to the one or more allowed utilities is based on a distance between an input embedding and an allowed utility embedding in a latent space, and Wherein, the input embedding is generated using the one or more trained machine learning models and the input data.

18. The method of claim 17, wherein: When the distance in the latent space does not satisfy an allowed utility threshold for the one or more allowed utilities, the input data is determined to be not semantically related to the one or more allowed utilities for generating the response data to the threshold degree.

19. The method of claim 18, wherein: Determining whether the one or more conditions for employing the one or more scalable utilities are satisfied is based on the distance between the input embedding and the allowed utility embeddings in the latent space.

20. The method of any one of claims 13 to 19, further comprising: When it is determined that the one or more conditions for employing the one or more extensible utilities are satisfied, the responsive output is caused to be rendered with an indication that the one or more extensible utilities were utilized to generate the responsive output.

21. A system comprising: one or more processors; as well as A memory storing instructions which, when executed by the one or more processors, cause the one or more processors to perform operations according to any one of claims 1 to 20.

22. A computer-readable storage medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform the operations of any one of claims 1 to 20.

Images