Machine learned API retrieval with negative example examples

Abstract

A computing system may receive a first request to perform at least one task. The first request may include a natural language description of the task. A computing system may generate a query embedding that includes an embedded description of an action to be performed to perform a task. A computing system may query a data structure that includes an API embedding that includes an embedded description of a function associated with a corresponding application programming interface (API). The data structure may include a false positive example API embedding that includes an embedded description of functionality that is not available via a corresponding API. False positive example API embedding may be associated with instruction content. The computing system may obtain instruction content based on a similarity between the query and the false positive example API embedding. Using the machine-learned sequence processing model, the computing system may operate based on the instruction content.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based upon and claims the benefit of priority from U.S. Provisional Patent Application No. 63 / 642,233, filed on May 3, 2024, the disclosure of which is hereby incorporated by reference in its entirety for all purposes. Technical Field

[0003] The present disclosure generally relates to machine learning processes and machine learning devices and systems. More specifically, the present disclosure relates to machine learning devices and systems configured to retrieve and use application programming interface (API) tools. Background Art

[0004] A computer can receive input. The computer can execute instructions to process the input using a parameterized model to generate an output. The computer can obtain feedback about its performance in generating the output using the model. The computer can generate feedback by evaluating its performance. The computer can receive feedback from an external source. Based on the feedback, the computer can update the parameters of the model to improve its performance. In this way, the computer can iteratively "learn" to generate a desired output. The resulting model is often referred to as a machine-learned model. Summary of the Invention

[0005] Various aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or may be learned from the description, or may be learned through practice of the embodiments.

[0006] An example aspect of the present disclosure provides an example method. In some implementations, the example method may include receiving, by a computing system comprising one or more computing devices, a first request to perform at least one task. In the example method, the first request may include a natural language description of the at least one task. The example method may include generating, by the computing system, a query embedding, the query embedding including an embedded description of an action to be performed to perform the task. The example method may include querying, by the computing system, a data structure including an API embedding using a query. In the example method, the API embedding may include an embedded description of a function associated with a corresponding application programming interface (API). In the example method, the data structure may include a false positive API embedding, the false positive API embedding including an embedded description of a function that is unavailable via the corresponding API. In the example method, the false positive API embedding may be associated with instruction content indicating that the function is unavailable. The example method may include obtaining, by the computing system, instruction content based on a similarity between the query and the false positive API embedding. The example method may include inputting, by the computing system, the instruction content into a machine-learned sequence processing model to generate an intermediate output. The example method may include executing, by the computing system, one or more operations identified in the intermediate output.

[0007] An example method may include generating, by a computing system, a natural language description of actions to be performed to perform a task using a machine-learned sequence processing model. An example method may include embedding, by a computing system, the natural language description of actions to be performed to perform a task to obtain an embedded description of the actions to be performed to perform the task.

[0008] An example method may include receiving, by the computing system, via the one or more APIs, availability data indicating that a first API is currently unavailable. An example method may include updating, by the computing system, a data structure based on the availability data.

[0009] In an example method, updating the data structure may include replacing an API embedding associated with the first API with a false positive API embedding associated with the first API.

[0010] In the example method, the instruction content may include one or more instructions for generating a user message. In the example method, the intermediate output may include a user message indicating that the first request cannot be currently satisfied. In the example method, the operation may include providing the user message to a user associated with the first request.

[0011] In an example method, the instruction content may include one or more instructions for suggesting a substitute action to be performed. In an example method, the user message may include a suggestion to perform a substitute action.

[0012] In an example method, the instruction content may include at least one of: data indicating a time when the function will become available, and one or more instructions for determining the time when the function will become available. In an example method, the replacement action may include scheduling execution of the function at the time when the function becomes available.

[0013] In an example method, the instruction content may include data indicating an API for performing the function. In an example method, the API may be associated with software that is not currently installed on the computing system. In an example method, the replacement action may include installing the software on the computing system.

[0014] An example method may include receiving, by a computing system, a user request to install software on the computing system. An example method may include installing, by the computing system, the software on the computing system in response to the user request. An example method may include updating, by the computing system, a data structure to include an API embedding indicating that an API associated with the software is currently available.

[0015] In an example method, the replacement action may include a replacement function available via a corresponding API. The example method may include receiving, by a computing system, a request to perform the replacement action. The example method may include generating, by the computing system, instructions for causing the API to perform the replacement function based on an API embedding associated with the replacement function. The example method may include providing, by the computing system, the instructions to the API.

[0016] In an example method, the instruction content may include data indicating an API for which the computing device lacks authentication data. The example method may include requesting, by the computing system, authentication data for accessing the API. The example method may include receiving, by the computing system, authentication data for accessing the API. The example method may include updating, by the computing system, a data structure in response to receiving the authentication data to include an API embedding indicating that the API is currently available. The example method may include generating, by a machine-learned sequence processing model, instructions for causing the API to perform an action to be performed to perform a task based at least in part on the authentication data. The example method may include providing, by the computing system, the instructions to the API.

[0017] In an example method, a data structure may include device-specific API information indicating one or more APIs accessible to the computing device.

[0018] In an example approach, the device-specific API information may include data indicative of software installed on one or more computing devices.

[0019] In an example method, the device-specific API information may include data indicating an API for which one or more computing devices have access authorization data.

[0020] In an example method, the computing system may include a mobile phone. In an example method, the machine-learned sequence processing model may be executed locally on the mobile phone. In an example method, the data structure may be stored locally on the mobile phone.

[0021] An example method may include receiving, by a computing system, a second request. The second request may include a natural language description of at least one second task. The example method may include generating, by the computing system, a query embedding including an embedded description of a second action to be performed to perform the second task. The example method may include querying, by the computing system, a data structure based on the query embedding. The example method may include obtaining, by the computing system, an API embedding. The API embedding may include one or more instructions for causing the API to perform the second action. The example method may include retrieving, from a second data structure including user-specific knowledge associated with a user associated with the second request, at least one data record. The at least one data record may include user-specific knowledge associated with the second action. The example method may include inputting, by the computing system, the user-specific knowledge associated with the second action and the one or more instructions into a machine-learned sequence processing model to generate an intermediate output. The example method may include performing, by the computing system, one or more operations identified in the intermediate output.

[0022] In an example method, the API embedding may include data indicating one or more functional categories associated with a function associated with a corresponding application programming interface (API). In an example method, the false positive API embedding may include data indicating one or more false positive functional categories. In an example method, the instruction content may be obtained based at least in part on the false positive functional categories.

[0023] In an example method, obtaining the instruction content may include selecting, by the computing system, a false positive function category using a machine-learned sequence processing model. In an example method, obtaining the instruction content may include obtaining, by the computing system, the instruction content based on the selected false positive category and a similarity between the query and the false positive API embedding.

[0024] In an example method, obtaining the instruction content may include selecting, by the computing system, the false positive category based on a similarity between the query and the false positive category. In an example method, obtaining the instruction content may include obtaining, by the computing system, the instruction content based on the false positive category.

[0025] In an example method, the first request may be provided in a format including audio data. In an example method, generating the query embedding may include providing at least a portion of the audio data to a machine-learned sequence processing model. In an example method, the machine-learned sequence processing model may be a multimodal machine-learned sequence processing model.

[0026] Example aspects of the present disclosure provide one or more example non-transitory computer-readable media storing instructions executable by one or more processors to cause a computing system to perform example operations. In some implementations, the example operations may include receiving a first request to perform at least one task. In the example operations, the first request may include a natural language description of the at least one task. The example operations may include generating a query embedding comprising an embedded description of an action to be performed to perform the task. The example operations may include using a query to query a data structure comprising API embeddings. In the example method, the API embedding may include an embedded description of functionality associated with a corresponding application programming interface (API). In the example operations, the data structure may include a false positive API embedding, the false positive API embedding comprising an embedded description of functionality that is unavailable via the corresponding API. In the example operations, the false positive API embedding may be associated with instruction content indicating that the functionality is unavailable. The example operations may include obtaining instruction content based on a similarity between the query and the false positive API embedding. The example operations may include inputting the instruction content into a machine-learned sequence processing model to generate an intermediate output. The example operations may include executing one or more operations identified in the intermediate output.

[0027] Example aspects of the present disclosure provide an example computing system comprising one or more processors and one or more example non-transitory computer-readable media storing instructions executable by the one or more processors to cause the computing system to perform example operations. In some implementations, the example operations may include receiving a first request to perform at least one task. In the example operations, the first request may include a natural language description of the at least one task. The example operations may include generating a query embedding comprising an embedded description of an action to be performed to perform the task. The example operations may include using a query to query a data structure comprising API embeddings. In the example method, the API embedding may include an embedded description of functionality associated with a corresponding application programming interface (API). In the example operations, the data structure may include a false positive API embedding comprising an embedded description of functionality that is unavailable via the corresponding API. In the example operations, the false positive API embedding may be associated with instruction content indicating that the functionality is unavailable. The example operations may include obtaining instruction content based on a similarity between the query and the false positive API embedding. The example operations may include inputting the instruction content into a machine-learned sequence processing model to generate an intermediate output. The example operations may include executing one or more operations identified in the intermediate output.

[0028] Other exemplary aspects of the present disclosure relate to other systems, methods, devices, tangible non-transitory computer-readable media, and apparatuses for performing the functions described herein. These and other features, aspects, and advantages of various implementations will be better understood with reference to the following description and appended claims. The accompanying drawings, which are incorporated into and constitute a part of this specification, illustrate implementations of the present disclosure and, together with the description, help explain the relevant principles. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a block diagram of an example system for performing API retrieval according to an example implementation of the present disclosure;

[0030] Figure 2 is a block diagram of an example system for performing API retrieval according to an example implementation of the present disclosure;

[0031] Figure 3 is a block diagram of an example system for performing API retrieval according to an example implementation of the present disclosure;

[0032] Figure 4 is a flowchart illustrating an example method according to an example aspect of the present disclosure;

[0033] Figure 5 is a flowchart illustrating an example method according to an example aspect of the present disclosure;

[0034] Figure 6 is a flowchart illustrating an example method according to an example aspect of the present disclosure;

[0035] Figure 7 is a flowchart illustrating an example method for training a machine-learned model according to an example implementation of aspects of the present disclosure;

[0036] Figure 8 is a block diagram of an example process flow for processing an input to generate an output using a machine-learned model according to an example implementation of aspects of the present disclosure;

[0037] Figure 9 is a block diagram of an example sequence processing model according to an example implementation of aspects of the present disclosure;

[0038] Figure 10 is a block diagram of an example technique for populating an example input sequence for processing by a sequence processing model according to an example implementation of aspects of the present disclosure;

[0039] Figure 11 is a block diagram of an example model development platform according to an example implementation of aspects of the present disclosure;

[0040] Figure 12is a block diagram of an example training workflow for training a machine-learned model according to an example implementation of aspects of the present disclosure;

[0041] Figure 13 is a block diagram of an inference system for operating one or more machine-learned models for inference according to an example implementation of aspects of the present disclosure;

[0042] Figure 14 is a block diagram of an example networked computing system according to an example implementation of aspects of the present disclosure;

[0043] Figure 15 is a block diagram of an example computing device according to an example implementation of aspects of the present disclosure; and

[0044] Figure 16 is a block diagram of an example computing device according to an example implementation of aspects of the present disclosure. DETAILED DESCRIPTION

[0045] In general, the present disclosure relates to machine-learned devices and systems configured to retrieve and use application programming interface (API) tools. More specifically, the present disclosure relates to API retrieval, where negative examples indicate that no API tool is currently available to perform a specific function. For example, a computing system may receive a request to perform a task in a natural language (e.g., English, etc.). The task may include one or more individual actions. Based on the request, the computing system may query a database of tools (e.g., software applications, hardware devices, web-based tools, etc.) that can be called via an API. The database may include "positive" data entries indicating a specific API currently accessible to the computing device, and "negative" or "false positive" data entries indicating that the API tool is currently unavailable or cannot be confidently selected based on the received request. If the query returns data entries for available API tools, the computing system may use a machine-learned sequence processing model (e.g., a language model) to generate instructions for calling the API to cause the tool to operate. If the query returns data entries indicating that the API tool for performing the requested action is currently unavailable, the computing device may take another action, such as returning an error message or suggesting an alternative action.

[0046] Some existing techniques for retrieving API information using database lookups can result in errors. For example, if the database does not contain any APIs that provide the desired functionality, a simple search technique (e.g., a best K search) that returns the most similar API entries may return irrelevant results. For example, a request may include a request for "confirm that my prescriptions are ready at the drugstore." But if the database does not include any pharmacy APIs, the most relevant results may be related to glasses or a more general shopping app. That is, the best result may have some similarity to the request but may not provide the desired functionality. According to traditional methods, avoiding returning incorrect APIs in such scenarios may require carefully tuning the similarity threshold, and even a carefully tuned similarity threshold may only result in a partial reduction in API retrieval errors.

[0047] Advantageously, according to example implementations of the present disclosure, augmenting the database of an API tool with false positives can facilitate accurate and high-confidence identification of unavailable functionality. For example, continuing with the above example, the API database can contain a false positive entry for pharmacy services. This can provide a strong match for a request. However, rather than containing API information, the entry can contain instructional content indicating the unavailability of the pharmacy services API. For example, in this way, the unavailability of the pharmacy services API can be determined with high confidence by providing a strong false positive match.

[0048] In some instances, querying the database can include generating a machine-learned embedding based on a request and querying based on the machine-learned embedding. For example, in some instances, a request can be provided to a first machine-learned sequence processing model (e.g., a language model) that acts as a machine-learned agent. Based on the request, the first machine-learned sequence processing model can identify one or more actions required to perform a task and can generate a natural language query based on each action. As an illustrative example, a request to send an email can include information about the content, recipient, time, and other information associated with the desired email. Based on such a request, the first machine-learned sequence processing model can generate a natural language query that can describe a type of action to be performed (e.g., in just a few words), such as "sendemail". In some instances, the first or second machine-learned sequence processing model can generate a machine-learned embedding based on the natural language query or based on the request. The machine-learned embedding can be, for example, a vector of numbers output by the second machine-learned model after being provided with the natural language query as input.

[0049] The database of API tools may include, for example, multiple API embeddings indicating multiple functions that the corresponding API tool is available to the computing system, and multiple false positive API embeddings indicating functions that the corresponding API tool is unavailable. Each API embedding and false positive embedding may include, for example: instruction content configured to be provided as input to a machine-learned sequence processing model; one or more machine-learned embeddings; and other appropriate data. Each API embedding may include, for example, one or more embedded descriptions of functions that can be performed by the corresponding API tool. Similarly, each false positive API embedding may include, for example, one or more embedded descriptions of functions that the corresponding API tool is unavailable. As an illustrative example, the database may have multiple API embeddings associated with various functions of the Gmail API, wherein the multiple API embeddings may include machine-learned embeddings of phrases such as "send an email," "search inbox for email," "display email," "schedule Gmail push notification," "edit email signature," and the like.

[0050] The query API tool's database may include, for example, retrieving one or more data entries (e.g., API embeddings, false positive API embeddings, etc.) based on a comparison between a machine-learned embedding of a natural language query and a machine-learned embedding associated with the data entry. For example, the query API tool's database may include retrieving the top n (e.g., one, three, five, etc.) data entries whose embeddings are closest to the query embedding based on a distance metric (e.g., cosine distance, Euclidean distance, etc.). In some instances, the query API tool database may include hierarchical queries, such as identifying a category or type of action to be performed and querying based on the category or type. The category or type of action may be identified, for example, by a sequence processing model (e.g., outputting a category name, etc.); by comparison between machine-learned embeddings (e.g., based on a distance metric); or other methods. In some instances, all or part of one or more (e.g., the top n, etc.) data entries may be provided to a machine-learned model, which may perform additional actions based on the provided data.

[0051] In some instances, if the query returns an API embedding of an available API tool that is capable of performing the requested action, the computing device may use the API tool to perform the action. Using the API tool may include, for example, providing the instruction content of the API embedding as input to a machine-learned sequence processing model; generating an intermediate output using the machine-learned sequence processing model based on the instruction content; and performing one or more actions identified in the intermediate output. For example, the instruction content may include an input context for generating instructions for calling the API tool. For example, the retrieved API embedding may include instruction content containing API documentation, such as an API signature describing the method name, syntax, and one or more parameters for calling the API tool (e.g., alarm.set(hours, mins, message), etc.). In some instances, the instruction content of the retrieved API embedding may include other input context for prompting the sequence processing model, such as a few-shot example illustrating example instructions for calling the API tool (e.g., alarm.set(13, 30, Make coffee), etc.). In some instances, the instruction content may be provided as input context to the machine-learned sequence processing model, and the sequence processing model may generate API instructions based on the input context. In some instances, the input context may include system prompts, such as instructions for generating an API call based on the received request. In some instances, the input context may include all or part of the received request.

[0052] If the query returns a false positive API embedding, the computing device may perform another operation, such as returning a message indicating that the requested action cannot be performed; scheduling the requested action for a time when a suitable API becomes available; suggesting an alternative action or alternative API; requesting additional information about the request (e.g., from a user); requesting additional information necessary to perform the requested action (e.g., access authentication data, such as a username and password, an API key, etc.); querying another database; or other operations. In some instances, performing the other operation may include generating, by the machine-learned sequence processing model, an output identifying the operation based on the instruction content of the retrieved false positive API embedding; and performing the operation by the computing device. For example, the instruction content of the false positive API embedding may include an input context to be provided to the machine-learned sequence processing model; the machine-learned sequence processing model may generate an output (e.g., an API call, an error message, etc.) based on the input context; and the computing device may perform the operation associated with the output.

[0053] In some instances, the actions associated with a false positive API embedding may include requesting iterative feedback to further clarify the request (e.g., to help identify an available API for making the request) or to gather information necessary to make the request. For example, a false positive API embedding may indicate that the computing system lacks authentication data (e.g., a username, password, other login data, API key, security certificate data, etc.), and the actions associated with the false positive API embedding may include requesting authentication data (e.g., from the user, from another computing device, etc.). As another example, a false positive API embedding may indicate that the request is ambiguous, or that multiple API tools can perform a particular action. For example, a "send email" request may be executed by more than one email API, or an email may be sent from more than one account associated with the same email API. As another example, a "send message" request (or a request to "tell" someone something, etc.) may be executed via an email API, a text messaging API, an internet-based instant messaging API, a voice messaging API, etc. In such instances, the negative example may include instructions to request additional information (e.g., from the user) to clarify the request.

[0054] In some instances, the API tool database may include device-specific or user-specific availability data. For example, the API tool database may include false positive API embeddings for API tools associated with software that has not yet been installed on a particular computing device; false positive API embeddings for API tools for which the user does not yet have access authorization (e.g., user account, API key, login data, etc.); or false positive API embeddings for API tools that are otherwise unavailable to the device or user. In some instances, the API tool database may be updated based on new availability information. For example, the API tool database may be updated when a software tool is installed or uninstalled; when access authorizations are created, added, deleted, edited, etc.; or when an API tool otherwise becomes available or unavailable (e.g., due to server downtime, power outage, etc.).

[0055] In some instances, the systems and methods of the present disclosure can be used in the context of a mobile digital assistant. For example, some example methods according to aspects of the present disclosure can be implemented on a mobile device such as a smartphone. In some instances, the mobile device can run one or more machine-learned models locally on the device. In some instances, the mobile device can store an API tool database locally on the device. In this way, for example, some requested tasks can be performed entirely on the device, and the privacy of the request data and API tool data can be protected.

[0056] In some instances, the API tool database may include API tools for reading and writing data (e.g., to computer memory). For example, in some instances, the API tool may retrieve custom context data associated with a particular device or user, such as a contact list stored on the device; a contact list associated with a user account; or other user-specific or device-specific context. As an illustrative example, a mobile digital assistant may receive a natural language request for "call Dad"; determine, based on user permission data, that the user has authorized the mobile digital assistant to access the user's contacts; retrieve a phone number associated with "Dad" from the user's contacts; and use the API tool to call the phone number. In some instances, the custom context may include custom contexts saved based on previous user interactions with the user's permission (e.g., by a machine-learned sequence processing model, by the mobile digital assistant, etc.). As an illustrative example, a mobile digital assistant may receive a request to “schedule an appointment with my accountant”; request clarification from the user to identify “my accountant”; ask the user whether they want the mobile digital assistant to remember the identity of “my accountant” in the future; and, in response to the user request to remember the information, write a custom contextual data entry relating “my accountant” to the identified person or business.

[0057] In some instances, a computing device may receive a multi-action or "multi-hop" request that may require multiple actions to fulfill the request. For example, in some instances, the request may require a first action for stating the request and a second action for performing the request. In some instances, fulfilling the request may require calling a first API to perform the first action (e.g., purchasing concert tickets online) and calling a second API to perform the second action (e.g., adding the concert date to a calendar, etc.). In some instances, fulfilling the multi-hop request may include identifying multiple subtasks by a machine-learned sequence processing model; and for each subtask, performing one or more of the operations described above (e.g., querying an API database; generating and executing API instructions; etc.).

[0058] The systems and methods of the present disclosure provide a variety of technical effects and benefits. For example, some implementations according to aspects of the present disclosure can provide improved technical performance compared to alternative systems and methods. As another example, some implementations according to aspects of the present disclosure can provide similar (e.g., identical) technical performance at a reduced computational cost (e.g., power usage, memory usage, processor usage, etc.) compared to alternative systems and methods.

[0059] In some examples, the provided systems and methods can provide increased reliability and reduced errors compared to alternative systems and methods. For example, the provided systems and methods can more accurately identify requests for which no API is available than alternative systems and methods (e.g., systems that rely on confidence thresholds, heuristics, etc.). For example, when no API is available to perform the requested task, some alternative implementations can populate the context window of a sequence processing model with the "nearest" unsuitable or unavailable API and rely on the sequence processing model itself to identify that no API can perform the requested action. However, when provided with such context, some sequence processing models may erroneously generate instructions that attempt to call unavailable or inappropriate APIs. For example, the risk of such errors may be higher for low-computational cost sequence processing models (such as models with low parameter counts or small context windows, or models configured to run locally on mobile devices (e.g., smartphones)). Additionally, for models with limited context windows, populating the context window with irrelevant API information may prevent the computing device from adding other, more relevant information to the context window. Increasing the relevance of the information in the context window may increase the technical performance of the system (e.g., inference accuracy, etc.) or enable additional actions. For example, instead of populating the context window with information about irrelevant or unavailable APIs, example implementations of the present disclosure may populate the context window with instructions for graceful failure, such as instructions for generating an error message; requesting clarification; suggesting an alternative action; or other actions. For example, in this way, the technical performance of an agent using a machine-learned API tool may be improved (e.g., the percentage of requests successfully fulfilled, etc.). Similarly, the number of failed requests or operations may be reduced (e.g., the number of requests to unavailable APIs, the number of operations performed that do not match the requested action, the number of invalid API requests to available APIs, the number of erroneous user messages provided to users, etc.).

[0060] Additionally, some example implementations can provide similar (e.g., identical) technical performance at a reduced computational cost compared to alternative implementations. For example, in some implementations, the computational cost of machine-learned sequence generation can increase with the size of the input context provided. For example, in some instances, the cost of processing the input context can increase quadratically with the size of the input context. In some instances, the systems and methods of the present disclosure can avoid populating the context window of a machine-learned model with irrelevant information, thereby providing similar technical performance at a reduced computational cost compared to alternative implementations.

[0061] Additionally, in some instances, systems and methods that can provide improved technical performance can be adapted to provide similar technical performance at a reduced computational cost compared to alternative systems and methods. For example, in some instances, increasing the size of a machine-learned model (e.g., the number of parameters, etc.) can increase the computational cost of the model (e.g., power cost, memory usage, processor usage, etc.) and improve the technical performance of the model. Similarly, reducing the size can reduce the cost and technical performance of the model. Therefore, systems and methods that can provide improved technical performance with models of similar size compared to alternative methods can be adapted to provide similar technical performance with smaller models compared to alternative methods, thereby reducing the computational cost of retrieval via machine learning API tools. In this way, for example, the provided systems and methods can improve the functionality of the computing device itself.

[0062] Additionally, by enabling smaller machine-learned models to perform well on API retrieval tasks, the provided systems and methods can enable the models to run on resource-limited client hardware (e.g., mobile devices such as smartphones), or on server hardware with reduced computational cost or lower latency. Running on resource-limited hardware can provide additional technical effects and benefits, such as reduced data transmission costs (e.g., by processing requests locally on the requester's device), increased privacy, reduced hardware costs (e.g., processor costs, memory costs, etc.), reduced computational costs (e.g., electricity costs from running on a processor device with lower power usage), and other benefits. Similarly, running on similar hardware with reduced latency can provide additional technical effects and benefits, such as reduced hardware costs and computational costs (e.g., electricity costs, processor usage, memory usage, etc.). For example, techniques that implement computations with reduced latency on a particular computing system can also implement computations with similar latency on less powerful computing systems (e.g., with fewer processor devices, less memory, etc.), thereby helping to reduce hardware costs and computational costs for some API retrieval tasks (e.g., API retrieval tasks with latency targets or maximum acceptable latency, etc.).

[0063] Additionally, the improved energy efficiency of example implementations of the present disclosure can reduce the amount of pollution or other waste associated with implementing machine learning models and systems, thereby advancing the fields of machine learning and artificial intelligence as a whole. The amount of pollution can be reduced overall (e.g., its absolute amount) or reduced on a normalized basis (e.g., energy per task, per model size, etc.). For example, the amount of CO2 released (e.g., by power sources) associated with training and executing machine learning models can be reduced by enabling more energy-efficient training or inference operations. The amount of thermal pollution in the environment (e.g., by processors / storage locations) can be reduced by enabling more energy-efficient training or inference operations.

[0064] Various example implementations are described herein with reference to the accompanying figures.

[0065] Figure 1 1 is a block diagram of an example system for performing API retrieval. A machine-learned agent 104 may receive a request 102 to perform one or more operations. Based on the request 102, the machine-learned agent 104 may generate one or more API queries 106 to retrieve one or more data records 110 from an API data structure 108 that includes data indicating a plurality of APIs. In some instances, the data records 110 may include data indicating currently available API tools 114, and in some instances, the machine-learned agent may generate one or more API instructions 112 to cause the one or more API tools 114 to perform one or more actions 116. In some instances, the data records 110 may include data indicating that a suitable API tool 114 is currently unavailable, and the machine-learned agent may perform one or more other actions based on the data records 110. In some instances, the machine-learned agent may provide one or more success or failure messages 118 (e.g., to a requester associated with the request 102, etc.) based on the data records 110 or the actions 116.

[0066] Request 102 may include, for example, various types of data, such as text data, audio data, image data, audiovisual data (e.g., video), binary data, numerical data, etc. In some instances, request 102 may include one or more types of data. Request 102 may include, for example, a request to perform a task or cause a task to be performed. A task may include one or more actions. A task may include an action to be performed by: a computing device; a processor device, an intelligent device (e.g., a smart appliance, a robotic device, an automated hardware device, etc.); a machine-learned agent 104; a software application; or any other device, software, or component thereof. Request 102 may include a natural language portion, such as a task description or an action description provided in a natural language format (e.g., a text-based natural language format, an audio-based natural language format, an image- or video-based natural language format, etc.). In some instances, request 102 may include additional components in addition to the natural language portion, such as metadata (e.g., in a structured format such as JSON, XML, etc.), supplemental data (e.g., image data, audio data, video data, etc.), contextual data, prompt data (e.g., few-shot prompts, thought chain prompts, system prompt data, etc.), or any other appropriate data. In some instances, request 102 may include multimodal data in more than one format (e.g., audio and text, image and audio, etc.). For example, a user may request "Find a website where I can buy a chair like this" in audio format, accompanied by, for example, an image of the chair (e.g., a mobile phone camera image, etc.).

[0067] The machine learning agent 104 may include one or more machine learning models. The machine learning agent 104 may include various model architectures. Example model architectures of the machine learning agent 104 may include sequence processing model architectures (e.g., transformers, selectively structured state space models, etc.). For example, the machine learning agent 104 may be configured to receive an input sequence and generate an output sequence. For example, the machine learning agent 104 may be configured to generate an output sequence, wherein elements of the output sequence are predicted based on elements of the input sequence. In some instances, the machine learning agent 104 may include an attention mechanism (e.g., self-attention, etc.). In some instances, the machine learning agent 104 may include a pre-trained sequence processing model (e.g., a pre-trained base model trained on a large corpus of natural language data, etc.). In some instances, the machine learning agent 104 may include a fine-tuned sequence processing model. For example, the machine learning agent 104 may include a sequence processing model that has been fine-tuned on API retrieval data. In some instances, the machine-learned agent 104 may include a sequence processing model that has been trained for API retrieval, such as via supervised learning, reinforcement learning (e.g., reinforcement learning based on human feedback, etc.), imitation learning, or other appropriate training methods. In some instances, the machine-learned agent 104 may include multiple machine-learned models (e.g., a first model including a large language model such as a transformer, a second model including a sentence encoder for generating sentence embeddings, etc.).

[0068] In some instances, the machine-learned agent 104 may include models with multiple parameters or context window sizes that are relatively small compared to larger machine-learned sequence processing models. For example, the machine-learned agent 104 may have multiple parameters or context window sizes to facilitate local operation on mobile devices (e.g., smartphones, etc.) that have relatively limited computing resources compared to enterprise-class machines. For example, in some instances, the machine-learned agent 104 may have multiple parameters or context window sizes such that the amount of memory (e.g., RAM) required to perform all or part of the inference computation using the machine-learned agent 104 is less than the amount of memory available on the mobile device (e.g., on-chip memory associated with a particular processor device or group of processor devices; memory of the entire mobile device; etc.). For example, the context window size may be less than or equal to 10,000 tokens; less than or equal to 5,000 tokens; less than or equal to 4,000 tokens; less than approximately 3,000 tokens; less than or equal to 2,000 tokens; less than or equal to 1,000 tokens; less than or equal to 500 tokens; etc. As another example, the number of parameters of the machine-learned model can be less than or equal to 100 billion; less than or equal to 50 billion; less than or equal to 25 billion; less than or equal to 20 billion; less than or equal to 15 billion; less than or equal to 10 billion; less than or equal to 5 billion; less than or equal to 3 billion; less than or equal to 1 billion; less than or equal to 500 million; less than or equal to 100 million; etc. As another example, the quantization level can be configured to reduce the memory usage of the model. For example, the number of bits used to represent the parameters of the model can be less than or equal to 32 bits; less than or equal to 16 bits; less than or equal to 8 bits; less than or equal to 4 bits; less than or equal to 2 bits; or 1 bit.

[0069] The API query 106 can include various types of data, such as machine-learned embedding data (e.g., vector, matrix, or tensor data; numeric or binary embedded data; etc.), natural language data, computer-readable instructions for accessing the API data structure 108, and the like. For example, in some instances, the API query 106 can include machine-learned embeddings that are configured to be compared (e.g., based on a distance metric) with machine-learned embeddings contained in the API data structure 108. For example, in some instances, the API query 106 can include machine-learned embeddings computed using a machine-learned model that is similar to (e.g., the same) as the machine-learned model used to compute the machine-learned embeddings of the API data structure 108. In this way, for example, the machine-learned embeddings of the API query 106 can share a semantic space with the machine-learned embeddings of the API data structure, and the distance (e.g., vector distance, etc.) between the machine-learned embeddings in the shared semantic space can indicate the similarity of semantic meaning between the embeddings.

[0070] The API query 106 can be generated, for example, by one or more machine-learned sequence processing models (e.g., machine-learned agents 104, etc.) based on the request 102. For example, in some instances, a first machine-learned sequence processing model can generate a natural language query based on the request 102, and a second machine-learned sequence processing model (e.g., a sentence encoder, etc.) can generate a machine-learned embedding based on the natural language query. The natural language query can be, for example, a word, a phrase, or a sentence (e.g., a short sentence with less than ten words, etc.). In some instances, the second machine-learned sequence processing model can include a sentence encoder configured to receive a plurality of tokens (e.g., text tokens, etc.) as input and generate a machine-learned embedding (e.g., a vector of numbers calculated based on the input tokens, etc.) as output. However, a dual-model or two-step implementation is not required. For example, in some instances, the request 102 can be provided directly to an encoder-based sequence processing model (e.g., a sentence encoder, etc.), and the encoder-based sequence processing model can generate a machine-learned embedding directly based on the request 102. In another example, an input context can be provided to a first sequence processing model configured to generate a natural language query based on request 102, and the machine-learned embedding can be extracted directly from an intermediate or hidden layer (e.g., the last layer before the output layer, etc.) of the sequence processing model. Other implementations are possible.

[0071] Generating a natural language query using the first sequence processing model (e.g., via the machine learning agent 104 or a component thereof) can include, for example, providing an input context including the request 102 and additional context to the first sequence processing model. In some instances, the additional context can include one or more instructions (e.g., natural language instructions), such as instructions to generate a natural language query. In some instances, the additional context can include a few-shot prompt. For example, the additional context can include a small number (e.g., 3, 5, 10, etc.) of example request-query pairs that illustrate examples of queries associated with the example requests. In some instances, the additional context can include thought chaining prompts, such as a step-by-step inference chain based on one or more examples of the one or more example requests.

[0072] In some instances, request 102 may include a request to perform multiple actions, and generating a natural language query based on request 102 may include identifying, by machine-learned agent 104, a first action to perform; and generating a natural language query associated with the action. In some instances, identifying the first action to perform may include providing request 102 and additional input context to machine-learned agent 104. In some instances, the additional input context may include one or more instructions (e.g., natural language instructions), such as instructions to decompose request 102 into its component actions, or instructions to select a first action to take to fulfill request 102. In some instances, the additional input context may include a few-shot prompt or a thought chaining prompt.

[0073] The API data structure 108 can be, for example, a data structure (e.g., a database, such as a vector database, a file, a file system, a collection of data objects, a hash table, a dictionary, etc.) that holds multiple data entries indicating one or more API tools. In some instances, each data entry can include a machine-learned embedding component and a corresponding input context component. The input context component can include, for example, data in a format to be provided to the machine-learned agent 104 as input context (e.g., sequence data, such as natural language data, etc.). The machine-learned embedding component can include, for example, machine-learned embedded data indicating one or more attributes of the data entry (e.g., a vector of numbers, a vector of quantized data (such as binary data), etc.). For example, the machine-learned embedding component can include a machine-learned embedding of a natural language description of a function, action, category, or other attribute associated with the data entry. In some instances, each data entry can indicate one or more functions associated with the API tool. For example, in some instances, for a single API tool, the API data structure 108 can include multiple data entries associated with multiple functions of the API tool. However, this is not required. For example, in some instances, the API data structure 108 can include data entries indicating multiple functions of the API tool. In some instances, the API data structure 108 may include data entries indicating multiple API tools. In some instances, the API data structure 108 may also maintain multiple negative data entries (also referred to as false positive data entries) indicating that no API is available to perform a particular function. Example implementation details of data entries (including negative data entries) are further described below with respect to data records 110 and negative data records 210. In some instances, the API data structure 108 may include a hierarchical structure (e.g., tree-based, etc.) or hierarchical data. For example, in some instances, each data record 110 of the API data structure 108 may belong to one or more categories (e.g., API categories, functional categories, etc.), subcategories, or other divisions of the API data structure 108.

[0074] In some examples, the API data structure 108 may be stored locally on a device running the machine-learning agent 104 (e.g., a mobile device such as a smartphone), or may be stored remotely (e.g., on a cloud computing device, etc.). The API data structure 108 may be accessed by the machine-learning agent 104 in a variety of ways. For example, the API data structure 108 may include a data structure accessible to the machine-learning agent 104 via an API (e.g., by generating API instructions 112, etc.). In some examples, the API data structure 108 may include a data structure accessible using system code or glue code (such as computer code that receives output (e.g., API query 106) from the machine-learning agent 104 and queries the API data structure 108 based on the output). In some examples, the API data structure 108 may include a data structure internal to the machine-learning agent 104 (e.g., a machine-learning agent 104 with memory read / write capabilities (e.g., a memory-augmented language model, a neural Turing machine, etc.)) or otherwise managed by the machine-learning agent 104. In some instances, the API data structure 108 (e.g., a locally stored data structure, etc.) may include device-specific or user-specific API availability information, such as data indicating that the API tool 114 is not installed on the device; data indicating that the device lacks authentication data necessary to use the API tool 114 (e.g., login data, API key, security certificate data, etc.); or other device-specific or user-specific availability data.

[0075] In some instances, querying the API data structure 108 using the API query 106 can include providing the API query 106 to a database service (e.g., a vector database service). In some instances, the database service can perform a search based on the query and return one or more results (e.g., data records 110). In some instances, performing the search can include performing a similarity search based on a comparison between a machine-learned embedding associated with the API query 106 and a plurality of corresponding machine-learned embeddings of a plurality of corresponding data entries of the API data structure 108. For example, in some instances, the machine-learned agent 104 can generate a machine-learned embedding to provide to the vector database service as the API query 106 or a component thereof. In some instances, the machine-learned agent 104 can provide a text-based query (e.g., in natural language) to a machine-learned embedding tool (e.g., a sentence encoder, a vector database service including a machine-learned embedding tool, etc.) to generate the machine-learned embedding. In some instances, the vector database service can search the API data structure 108 based on the embedded query. In some examples, the vector database service can compare the vector-based machine-learned embedding of the API query 106 with a plurality of vector-based machine-learned embeddings of a plurality of corresponding data entries of the API data structure 108. In some examples, the vector database service can identify one or more data entries having machine-learned embeddings that are close (e.g., top-n closest embeddings, etc.) to the machine-learned embedding of the API query 106 based on a distance metric (e.g., cosine distance, Euclidean distance, etc.). In some examples, the vector database service can provide the one or more data entries as output (e.g., to the machine-learned agent 104; to a computing device running or controlling the machine-learned agent 104; etc.).

[0076] In some instances, querying the API data structure 108 using the API query 106 can include a hierarchical or multi-step (e.g., two-step, three-step, etc.) query process. For example, in some instances, querying the API data structure 108 can include executing a first query to retrieve data indicating a plurality of data records 110; and executing a second query or second action to select one or more data records 110 from the plurality of data records 110. For example, in some instances, querying the API data structure 108 can include executing a first query based on a functional category or API category associated with the request 102. After retrieving the category or a plurality of data records 110 associated with the category, the computing device can perform a second action to select one or more data records 110 associated with the category. In some instances, category-based querying can include generating the API query 106 based on the category (e.g., based on a natural language description of the category generated by the machine learning agent 104, etc.) and querying the API data structure 108 based on the API query 106. In some instances, the second action can include generating an API query 106 associated with the request 102 and selecting one or more data records 110 based on a distance (e.g., cosine distance, Euclidean distance, etc.) between the embedding of the API query 106 and the embedding of the one or more data records 110. However, this is not required. For example, the first query can include any method for selecting a category. As an example, the machine-learned agent 104 can be provided with an input context (e.g., instruction content) that includes an indication of multiple API categories, and the machine-learned agent 104 can generate a category selection based on the input context. Additionally, the second action can include any method for narrowing down the multiple data records 110, such as providing data indicating the data record 110 as input context to the machine-learned agent 104 (e.g., along with instruction content that instructs the machine-learned agent 104 to select the data record 110 from the multiple data records 110). In some instances, the multi-step query process can include three or more actions, such as selecting a category; then selecting a subcategory; and then selecting a single data record 110 belonging to the subcategory. In some instances, the multi-step query process can involve multiple data records 110 grouped in ways other than predefined categories. For example, in some instances, n data records 110 can be retrieved based on the top n most recent embeddings of the embeddings of the API query 106. In some instances, the machine-learned agent 104 can be provided with an input context including the most recent n data records 110 and instructed to select the best match. Other implementations are possible.

[0077] Data records 110 may include all or part of a data entry (e.g., a database row, an object instance, a file, etc.) of an API data structure 108. In some instances, each data record 110 may be associated with a machine-learned embedding (e.g., include the machine-learned embedding, be indexed by the machine-learned embedding, etc.) for comparison with API queries 106. In some instances, each data record 110 may include input context data configured to be provided as input to the machine-learned agent 104. In some instances, the machine-learned embedding associated with a data record 110 may include a machine-learned embedding of a natural language sequence (e.g., a query, a phrase, a sentence, etc.) describing the API function associated with the data record 110. In some instances, the machine-learned embedding associated with a data record 110 may include a machine-learned embedding of documentation for the API function associated with the data record 110. Such documentation may include, for example, natural language documentation; method signatures, parameter lists, or other technical documentation (e.g., in a computer programming language, etc.) describing how to call the API function; or other document data. In some instances, the input context data of the data record 110 may include data indicating an API tool currently available for use by the machine-learned agent 104 or a computing device associated with the machine-learned agent 104. In some instances, the data record 110 may include a negative data entry that does not indicate a currently available API tool for performing the requested action. Figure 2 1 and negative data record 210 further describe example implementation details of example negative data entries. In some instances, data record 110 may include other data, such as metadata (e.g., structured metadata, natural language metadata, metadata embedded via machine learning, etc.). For example, data record 110 may include data indicating one or more categories or subcategories associated with the corresponding API, function, or instruction content of data record 110.

[0078] The data indicating the available API tools for performing the requested action may include various types of data, such as computer-readable API documentation data (e.g., syntax data, method signature data, etc.), natural language API documentation data, prompt data for prompting the machine-learned model (e.g., instructions, few-shot prompts, thought chain prompts, etc.), or other appropriate data. For example, the data indicating the available API tools may include input context to be provided to the machine-learned agent 104 so that the machine-learned agent 104 generates one or more API instructions 112 (e.g., API calls, method calls, function calls, etc.) to cause the available API tools to perform one or more actions.

[0079] The API instructions 112 may be, for example, computer-readable instructions that, when executed by the API tool, cause the API tool to perform one or more operations, such as one or more actions associated with the request 102 (e.g., the requested action, intermediate actions required to facilitate the requested action, etc.).

[0080] The API tools 114 may include, for example, any tools that can be accessed via an API (e.g., hardware tools, software tools, firmware tools, etc.). For example, in some instances, the API tools 114 may include software (e.g., applications, operating systems, etc.) installed on a device (e.g., a mobile device, a smartphone, etc.) running the machine-learning agent 104; software available via a network (e.g., the Internet); hardware devices (e.g., Internet-connected hardware devices, Bluetooth-connected hardware devices, etc.); etc. In some instances, the hardware API tools may include hardware tools connected (e.g., via a network, via a wireless or wired connection, etc.) to the device running the machine-learning agent 104. In some instances, the API tools may include navigation APIs (e.g., map-related APIs, global positioning system-related APIs, etc.); communication APIs (e.g., APIs associated with making phone calls, emails, or text messages (such as SMS or MMS); APIs associated with communication applications (such as messaging applications, social media communication applications, etc.); scheduling APIs (e.g., calendars, alarm clocks, automated task scheduling, etc.); media player APIs (e.g., video players (such as YouTube), audio players, etc.); shopping APIs; payment APIs (such as Google Pay); mobile banking APIs; travel-related APIs (e.g., flight booking, hotel booking, etc.); or APIs for any application installed on the mobile device. In some instances, the API tools 114 may include hardware devices, such as Bluetooth-connected locks, door openers, garage door openers, etc.; Internet-connected doorbells or security camera devices; smart home devices, such as smart TVs, smart appliances, lighting devices, thermostats, etc.; or any other API-accessible hardware tools.

[0081] Action 116 can include, for example, any action that can be performed by API tool 114. For example, action 116 can include an operation configured to complete a task or a portion of a task associated with request 102. In some instances, action 116 can include one operation or multiple operations. In some instances, action 116 can include multiple operations performed by a processor device (e.g., a processor device running a software application; a processor device connected to or embedded in a hardware tool; etc.), such as a processor device associated with a computing device on which machine-learned agent 104 is stored.

[0082] In some implementations, Figure 1 The system may perform multiple API retrieval cycles associated with a request 102. For example, the machine-learned agent 104 may perform multiple corresponding API queries 106; retrieve multiple corresponding data records 110; generate multiple corresponding API instructions 112 based on input context data of the data records 110; cause multiple corresponding API tools 114 to perform multiple corresponding actions 116; and so on. In some examples, the machine-learned agent 104 may perform multiple API queries 106 based on multiple operations required to perform the requested task associated with the request 102. For example, the machine-learned agent 104 may analyze the request 102; identify a first component operation associated with the request 102 based on the request; perform the API query 106, retrieve the data records 110, and generate the API instructions 112 to cause the API tool 114 to perform the action 116 corresponding to the first component operation. The machine-learned agent 104 may perform a similar series of actions for a second component operation; a third component operation; and so on (e.g., before generating a first success / failure message 118). In some instances, the machine-learned agent 104 can identify multiple component operations by generating one machine-learned sequence (e.g., before generating any API queries 106, etc.) or multiple machine-learned sequences (e.g., identifying component operations individually during each API search cycle). For example, in some instances, a second component operation can be determined based in part on a result associated with a first component operation (e.g., success, failure, output value, data record 110, etc.). In some instances, the machine-learned agent 104 can attempt to perform a first component operation and fail (e.g., because no API tool 114 is available to perform the first component operation, etc.), and one or more second component operations can be identified based on the failure (e.g., alternative operations, etc.). Figure 2 Provides additional details of example substitution operations.

[0083] Success / failure message 118 may include, for example, a message (e.g., to a user, to another computing device, etc.) indicating that request 102 has been successfully fulfilled or explaining that request 102 could not be fulfilled. In some instances, success / failure message 118 may include additional information, such as information providing details about action 116 performed; information explaining why request 102 could not be fulfilled; or other appropriate information. Figure 2 And failure / follow-up message 218 further provides example implementation details of a failure message indicating that request 102 cannot be fulfilled.

[0084] In some instances, a success / failure message can be generated based at least in part on data received from the API tool 114. For example, in some instances, the API tool can return success / failure data 119 indicating whether the action 116 has been successfully performed. For example, in some instances, the API tool 114 can be configured to return an error message or error code if the API instruction 112 cannot be successfully executed. In such instances, the computing system can provide a failure message 118 to the user based on the error message or error code. In some instances, generating the failure message 118 based on the error message from the API tool can include providing the error message along with additional input context (such as instruction context, few-shot prompt context, thought chain prompt context, etc.) to the machine-learned agent 104. In some instances, the failure message 118 can be generated without using the machine-learned agent 104. For example, in some instances, the error message from the API tool 114 can be provided directly to the user. In some instances, the failure message 118 can be generated based on a data structure that associates the error message or error code with a user message associated with the error message or error code. As an illustrative example, an HTTP 404 error code may be associated with data indicating a human-readable message that the resource could not be found. Similarly, the API tool 114 may be configured to return a success message or success code if the API instruction 112 is successfully executed. In such an instance, the computing system may generate a success message 118 based on the success data 119 in a manner similar to (e.g., identical to) that described above for the failure message 118.

[0085] In some instances, success / failure message 118 can be based at least in part on data from a source other than API tool 114. For example, failure message 118 can be generated based on: a data record 110 indicating that no API tool is available to fulfill request 102; a determination by machine learning agent 104 that API instruction 112 could not be generated; a determination by an automated syntax checker that API instruction 112 generated by machine learning agent 104 is invalid; or other suitable data. Figure 2 And failure / follow-up message 218 further provides example implementation details of a failure message indicating that request 102 cannot be fulfilled.

[0086] Figure 21 is a block diagram of an example system for performing API retrieval using negative data records according to example aspects of the present disclosure. A machine-learned agent 104 may receive a request 102 and may provide one or more API queries 106 to an API data structure 108. Based on the API queries 106, the API data structure 108 may return one or more negative data records 210 indicating that no API is available to satisfy the request 102. In response, the machine-learned agent 104 may perform one or more retry or fallback protocols (e.g., based on data received in the negative data records 210). The machine-learned agent 104 may generate one or more failure messages 218 indicating that the request 102 cannot be satisfied. In some instances, the negative data records 210 may be based at least in part on availability data 220 received from the API tool 114. For example, in response to receiving availability data 220 indicating that the API tool 114 has become available, the API data structure 108 may be updated to replace the negative data records 210 with data records 110 indicating that the API is available. Similarly, in response to receiving availability data 220 indicating that API tool 114 has become unavailable, API data structure 108 may be updated to replace data record 110 with negative data record 210 indicating that the API is unavailable.

[0087] Negative data record 210 may be, for example, a data record 110 indicating that an API for performing a particular action (e.g., a requested action, an intermediate action required to perform the requested action, etc.) is unavailable or cannot be confidently selected based on request 102. Negative data record 210 may include, for example, one or more machine-learned embeddings; instruction content; and other appropriate data. For example, in some instances, the machine-learned embedding of negative data record 210 may be a machine-learned embedding of a description (e.g., a word, phrase, sentence, etc.) of an action for which no action API is available (e.g., "feed pet") or an action for which an available API cannot be confidently selected (e.g., an ambiguous action, such as "telling" someone something). In some instances, negative data record 210 may include a machine-learned embedding configured for comparison (e.g., based on a distance metric) with the machine-learned embedding of API query 106. For example, in some instances, API query 106 may include a machine-learned embedding computed using a similar (e.g., identical) machine-learned model as the machine-learned model used to compute the machine-learned embedding of negative data record 210. In this way, for example, the machine-learned embeddings of API query 106 can share a semantic space with the machine-learned embeddings of negative data record 210, and the distance between the machine-learned embeddings in the shared semantic space (e.g., vector distance, etc.) can indicate the similarity of the semantic meanings between the embeddings. In some instances, negative data record 210 can have any of the properties described above with respect to data record 110, and vice versa. In some instances, negative data record 210 can be processed in any manner described above with respect to data record 110 (e.g., retrieved via API query 106, provided as input to machine-learned agent 104, etc.), and vice versa.

[0088] In some instances, the instruction content of the negative data record 210 may include one or more instructions to suggest one or more alternative actions to be performed instead of the requested action. In some instances, the alternative action may include scheduling the requested action to be performed at a future time when the API tool 114 becomes available. In some instances, the alternative action may include an alternative action that can be performed by a currently available API tool 114. In some instances, the alternative action may include an action to make the API tool 114 available, such as installing the API tool 114 (e.g., on a mobile device that includes the machine-learned agent 104) or providing access authorization data (e.g., login data, API key data, user permission data, etc.).

[0089] In some instances, the alternative action may include scheduling the requested action for a future time when the API tool 114 becomes available. For example, the instruction data of the negative data entry 210 may include data indicating a time when the API tool 114 is expected to become available, or data indicating conditions under which the API tool 114 is expected to become available (e.g., the device enters communication range; the device is powered on or otherwise activated; access authorization is obtained; etc.). As an illustrative example, a negative data record 210 associated with a web-based API tool 114 may include data indicating that the API tool 114 is expected to become available when the device (e.g., a mobile device, etc.) regains access to the internet (e.g., by leaving airplane mode; by coming within range of a Wi-Fi signal or a cellular internet signal; etc.). In such instances, the instruction content may include instructions for generating a user message that asks the user whether they want to schedule the requested action for when the device regains access to the internet (or when the API tool 114 otherwise becomes available). In response to receiving a user request to perform a suggested action, the machine-learning agent 104 may generate one or more instructions (e.g., API instructions 112, etc.) to cause the suggested action to be performed (e.g., API tool 114, etc.).

[0090] Generating instructions for causing the suggested action to be performed may include, for example, providing all or part of the instruction content of the negative data entry 210 to the machine-learned agent 104; generating an intermediate output using the machine-learned agent 104 based on the provided instruction content, the intermediate output including one or more instructions to perform the suggested action; and executing the one or more instructions. In some instances, the instruction content of the negative data entry 210 may include two or more input contexts that are configured to be provided separately to the machine-learned agent 104. For example, in some instances, the negative data entry 210 may include first instruction content configured to cause the machine-learned agent 104 to suggest an alternative action and second instruction content configured to cause the machine-learned agent 104 to generate one or more instructions (e.g., API instructions 112) for performing the suggested action.

[0091] In some examples, multiple alternative actions may be suggested (e.g., multiple alternative actions embedded close to the API query 106 according to a distance threshold, etc.). For example, the machine-learned agent 104 may generate a user message suggesting multiple alternative actions and prompting the user to approve or disapprove each action; select one or more alternative actions to perform; or otherwise instruct the machine-learned agent 104 or computing device on how to proceed.

[0092] In some instances, the instructions for scheduling an action that is currently unavailable may include instructions for scheduling an action at a specific time; instructions for starting an event listener to trigger an alternative action at a time when the function becomes available (e.g., when a specific condition is met); or other instructions for scheduling an action. The second instruction content may include, for example, data indicating an API tool 114 (e.g., Calendar API tool 114, Alarm Manager API tool 114, JobScheduler API tool 114, etc.) for scheduling an action at a specific date and time, or data indicating an API tool 114 (e.g., JobScheduler API tool 114, event listener or event handler API tool 114, etc.) for starting an event listener. The second instruction content or third instruction content of the negative data record 210 may also include, for example, data indicating an API tool 114 for performing the requested action. For example, the third instruction content associated with the unavailable API tool 114 for performing the requested action may be provided to the machine-learned agent 104, which may generate the first API instruction 112 to be scheduled based on the third instruction content. Additionally, a second instruction content associated with a scheduling API tool 114 (e.g., an event listener API tool 114, a time-based scheduling API tool 114, etc.) can be provided as input to the machine learning agent 104 (e.g., in combination with an API instruction 112 generated based on a third input content, etc.), and the machine learning agent can generate a scheduling API instruction 112 to schedule the first API instruction 112 to be scheduled.

[0093] In some instances, the alternative action may include an alternative action that can be performed by a currently available API tool 114. As an illustrative example, if a video messaging application is not installed on the mobile device associated with the machine-learned agent 104, an API query 206 based on a request 102 to send a video message to a particular person may return a negative data record 210. In such an instance, the alternative action may include contacting the person using a text messaging API tool 114 or other API tool 114 for sending text-based communications (e.g., email, social media messenger, etc.). In some instances, the alternative action may be an action similar to the requested action. For example, the alternative action may include an action associated with a machine-learned embedding that is similar to the machine-learned embedding associated with the requested action (e.g., API query 106). Similar machine-learned embeddings may include, for example, machine-learned embeddings that are close to each other according to a distance metric (e.g., cosine distance, Euclidean distance, Manhattan distance, etc.). For example, in some instances, the one or more suggested alternative actions may include one or more actions associated with a machine-learned embedding (e.g., a keyword embedding, an API description or function description embedding, etc.) that is less than a distance threshold from the machine-learned embedding of the API query 106 or the negative data record 210. In some instances, the alternative actions may include actions that belong to the same category (e.g., within a hierarchical or categorical representation of the API data structure 108) as the requested action for which no API tool 114 is currently available. For example, in some instances, the API data structure 108 may include a first data structure for determining an API tool category or action category (e.g., communication; video; music; etc.) associated with the API query 106 and a second data structure for determining an API tool 114 within the determined category. Additional levels of hierarchy are possible (e.g., subcategories, sub-subcategories, etc.).

[0094] In some instances, the replacement action may include an action to make the API tool 114 available, such as installing the API tool 114 (e.g., on a mobile device including the machine-learned agent 104), providing access authorization data (e.g., login data, API key data, etc.), providing user permission to use the API tool 114, or otherwise making the API tool 114 available (e.g., by turning off airplane mode, turning on Bluetooth, turning on near-field communication, etc.). For example, in some instances, the instruction content of the negative data record 210 may include one or more instructions for generating a subsequent message 218 requesting the user to enter access authorization data (e.g., a username and password, etc.), provide permission to access the API tool 114, approve the installation of the API tool 114, or otherwise approve an action to make the API tool 114 available (e.g., action 116). In some instances, in response to the API tool 114 becoming available, the machine-learned agent 104 may automatically perform the requested action 116 using the now-available API tool 114 (e.g., without requiring the user to submit an additional request 102, etc.). For example, in some instances, the instruction content of the negative data record 210 may include first instruction content for generating a user message suggesting an alternative action; second instruction content for generating API instructions 112 to perform the suggested alternative action in response to the user request to make the API tool 114 available; and third instruction content for performing the requested action 116 (e.g., associated with the request 102) in response to the API tool 114 becoming available.

[0095] In some instances, the instruction content of the negative data record 210 may include one or more instructions for suggesting user actions (e.g., including one or more actions that cannot be performed using the API tool 114). For example, the suggested user actions may include downloading or installing the API tool 114 by the user via a graphical user interface; obtaining access authorization data for the API tool 114 by the user (e.g., when the user does not remember or has not yet obtained the access authorization data); or other actions that cannot be performed, scheduled, triggered, or otherwise caused to be performed by the machine learning agent 104. In some instances, the instruction content for suggesting user actions may include detailed user instructions, such as step-by-step instructions; hyperlinks or other navigation data for performing the user action; or other content that can assist the user in performing the suggested user action.

[0096] In some instances, the instruction content of negative data record 210 may include one or more instructions for requesting additional information (e.g., from a user). For example, negative data record 210 may include instruction content for generating a subsequent message 218 (e.g., as described above) requesting the user to provide access authorization data.

[0097] As another example, the negative data record 210 may include instruction content to generate a follow-up message 218 asking the user to clarify a particular request. For example, in some instances, the request 102 may include a requested action that can be performed by multiple API tools 114 (e.g., an action to "tell" someone something, etc.) or a requested action that is otherwise unclear or ambiguous. As another example, an ambiguous or vague term (e.g., "schedule," etc.) may appear to refer to two or more different actions (e.g., setting a calendar reminder, making an appointment, sending a scheduled email, etc.). The negative data record 210 may include embeddings that are specifically configured to match ambiguous or vague terms. In this way, for example, the system can directly detect ambiguous or vague terms and initiate a response protocol (e.g., a follow-up action), rather than relying on a similarity measure between the ambiguous or vague term and a potentially available API record. For example, negative data record 210 may include instructional content to generate: a subsequent message 218 requesting the user to select one of a plurality of available API tools 114; a subsequent message 218 asking the user questions about request 102 or requesting the user to rephrase request 102; or other subsequent message 218 for clarification. In some instances, subsequent message 218 requesting the user to select between a plurality of available API tools 114 may be, include, or consist of suggestions for alternative actions (e.g., as described above). In some instances, request 102 may be unclear for other reasons, such as typographical errors; lack of audio clarity in an audio-based request 102 (e.g., mumbled, slurred, or otherwise unclear speech); data transmission errors; or any other source of uncertainty. In such instances, the negative data record 210 may include instruction content to generate: a subsequent message 218 requesting the user to repeat or rephrase the request 102; a failure message 218 indicating that the request could not be understood; a subsequent message 218 suggesting one or more actions or API tools 114 that may satisfy the request 102 (e.g., an API tool 114 selected with a medium or low confidence level, etc.); or other appropriate failure / subsequent message 218. In response to receiving a user response to the subsequent message 218, the machine-learned agent 104 may perform one or more additional actions based on the user response. For example, the machine-learned agent 104 may generate an API instruction 112 based at least in part on the user response. In some instances, the API instruction 112 may be based on the second instruction content of the negative data record 210 associated with the subsequent message 218.

[0098] In some instances, the instruction content of the negative data record 210 may include one or more instructions for generating a failure message 218 (e.g., to be provided to a user). The instruction content for generating the failure message 218 may include, for example, one or more message templates; one or more system prompts instructing the machine-learning agent 104 to generate the failure message 218; one or more few-shot or thought chaining prompts; or other appropriate input context for generating the failure message. Generating the failure message 218 may include, for example, providing the instruction content of the negative data record 210 to the machine-learning agent 104 as input, and the machine-learning agent 104 generating the failure message 218 based on the instruction content. In some instances, generating the failure message 218 may include providing all or part of the request 102 to the machine-learning agent 104, and the failure message 218 may be based in part on the request 102.

[0099] In some instances, the instruction content of the negative data record 210 may include instructions to perform one or more other actions, such as querying the second API data structure 108; providing a request to the second machine-learned agent 104; or other appropriate actions. For example, in some instances, the first API data structure 108 may include data indicating API tools 114 currently available for use by the machine-learned agent 104, and the second API data structure 108 may include data indicating API tools 114 that can be made available for use by the machine-learned agent 104 (e.g., by downloading, installing, purchasing, logging in, or otherwise obtaining access to the API tools 114). In such instances, the instruction content of the negative data record 210 may instead include instructions to query the second API data structure 108 to identify the API tools 114 that can be made available for use to perform the requested action; and instructions to suggest (e.g., to the user) one or more actions for making the API tools 114 available. As another example, the first machine-learned agent 104 can be a less powerful agent (e.g., having fewer parameters, a smaller context window) than the second machine-learned agent 104. For example, the first machine-learned agent 104 can include a small machine-learned model configured to perform low-latency inference locally on a mobile device. The second machine-learned agent 104 can include, for example, a larger machine-learned model (e.g., hosted as a cloud-based service, etc.) to perform actions that cannot be accurately performed by the first machine-learned agent 104. In some instances, the smaller first machine-learned agent 104 can query the same or different first API data structure 108 as the API data structure 108 queried by the second machine-learned agent. As an example, a less reliable smaller model can be configured to query a first API data structure 108 having a larger number of negative data records, and a more reliable larger model can be configured to query a second API data structure 108 having a smaller number of negative data records 210.

[0100] Failure / follow-up message 218 may include, for example, computer-readable data, human-readable data, or both. For example, in some instances, failure / follow-up message 218 may include natural language data to be provided to the user (e.g., in text format, audio format, multimodal format, etc.). Failure message 218 may include data indicating that request 102 could not be fulfilled, along with any other data that may be helpful or useful. Follow-up message 218 may include, for example, a request for additional information (e.g., clarification information, access authorization information, etc.); a suggested alternative action; or any other message configured to request or suggest a follow-up action from the user.

[0101] The availability data 220 may include any data indicating the availability of one or more API tools. The availability data 220 may include, for example, computer-readable data, human-readable data, or both. For example, in some instances, the API tool 114 may send availability data 220 that includes computer-readable data (e.g., an error code, availability data sent according to a communication protocol, etc.) indicating that the API tool 114 is unavailable, along with a human-readable explanation of the reason for the unavailability (e.g., "Bluetooth device cannot be accessed because Bluetooth connection is turned off," etc.). In some instances, the availability data 220 may be received directly from the API tool 114 (e.g., via a push notification, a webhook, an API call, etc.), or may be inferred from other data (e.g., request timeout data; device setting data, such as Bluetooth settings, airplane mode, etc.; data received from an uptime / downtime tracking service, etc.).

[0102] In some instances, the availability data 220 may include data indicating one or more conditions under which a currently unavailable API tool 114 is expected to become available. For example, the availability data 220 may include data indicating a time when the API tool 114 is expected to become available; an action that must be performed to make the API tool 114 available (e.g., turning on Bluetooth, downloading or installing the API tool 114, entering access authorization data, etc.); or other conditions associated with the availability of the API tool 114.

[0103] In some instances, the computing device may update the API data structure 108 based on the availability data 220 (e.g., periodically, immediately upon receiving the availability data 220, etc.). For example, in response to receiving the availability data 220 indicating that the API tool 114 recently became available, the computing device may replace the negative data record 210 indicating that the API tool 114 is unavailable with the data record 110 including instructions for accessing the API tool 114. As another example, in response to receiving the availability data 220 indicating that the API tool 114 recently became unavailable, the computing device may replace the data record 110 indicating that the API tool 114 is unavailable with the negative data record 210 indicating that the API tool 114 is unavailable. In some instances, the instruction content of the negative data record 210 may include instructions based on the conditions under which the API tool 114 is expected to become available. For example, the instruction content of the negative data record 210 may include instructions to suggest an action associated with the conditions under which the API tool 114 is expected to become available. The suggested actions may include, for example, scheduling the requested action for a time when the API tool 114 is expected to be available (e.g., when an event listener determines that a condition is met, etc.); downloading or installing the API tool 114; turning on Bluetooth; etc. In some examples, the computing device may periodically query one or more availability data sources and update the API data structure based on the queries.

[0104] In some instances, the availability data 220 may include one or more events detected by the computing device (e.g., airplane mode or Bluetooth being turned on or off; the device entering or leaving Wi-Fi or cellular internet range; an error code received from an API tool 114; etc.). For example, the computing device may include one or more event listeners configured to automatically update one or more data records 110 upon detecting an event (e.g., a predetermined event) that affects the availability of one or more corresponding API tools 114 associated with the one or more data records. The event may include, for example, an event that causes an unavailable API tool 114 to become available; an event that causes an available API tool 114 to become unavailable; or any other event (e.g., a push notification, etc.) that indicates a change in the availability of an API tool 114. In response to detecting an event indicating that one or more API tools 114 have become unavailable, the event listener may replace the one or more corresponding data records 110 indicating that the API tool 114 is unavailable with one or more negative data records 210 indicating that the API tool 114 is unavailable. Similarly, in response to detecting an event indicating that one or more API tools 114 have become available, the event listener may replace one or more corresponding negative data records 210 indicating that the API tools 114 are unavailable with one or more data records 110 indicating that the API tools 114 are unavailable. As an illustrative example, the event listener may be configured to detect an event indicating that the computing device has been connected to the Internet or disconnected from the Internet. The event listener may include data (e.g., a list, etc.) indicating multiple API tools 114 that require Internet access to use or be able to access the data. Based on the detected event and the data indicating multiple API tools 114, the event listener may update the multiple data records 110 with data indicating that the API tools 114 have become available or unavailable. For example, in response to airplane mode being turned on, the event listener may update the multiple data records 110 with instruction content configured to cause the machine learning agent 104 to generate a failure / follow-up message 218, which indicates that the request 102 cannot be satisfied without Internet access and recommends that the user turn off airplane mode.

[0105] In some instances, availability data 220 indicating a change in the availability of the API tool 114 may be received during an API retrieval cycle, and the machine-learned agent 104 may take one or more actions in response to the change in availability. For example, the machine-learned agent 104 may retrieve a data record 110 indicating an API tool 114 that appears to be available; generate an API instruction 112 for the API tool 114; and the API tool 114 may return availability data 220 (e.g., an error message, an error code, etc.) in response to the API instruction 112 indicating that the API tool 114 is unavailable or cannot be successfully used. In response to receiving the availability data 220, the computing device may update the data record 110 to include data indicating that the API tool 114 is unavailable, and the machine-learned agent 104 may perform one or more actions based on the data indicating that the tool is unavailable. For example, the computing device may replace the data record 110 with a negative data record 210, and the machine-learned agent 104 may perform one or more actions based on the negative data record 210.

[0106] In some instances, the usability data 220 may include moderation data (e.g., AI security audit data, copyright compliance audit data, moderation data based on user preferences or community preferences, etc.). For example, in an instance where the action 116 is determined to be an unsatisfactory response to the request 102 (e.g., based on user feedback, AI researcher feedback, copyright notification data, etc.), one or more negative data records 210 may be added based on the request 102. For example, the newly added negative data record 210 may include a machine-learned embedding similar to (e.g., the same as) the machine-learned embedding of the API query 106 generated during the selection of the API tool 114 to perform the unsatisfactory action 116. In some instances, the newly added negative data record 210 may include instruction content (e.g., manually generated instruction content, automatically generated instruction content, etc.) for determining whether the request 102 can be satisfactorily fulfilled, or other negative data record 210 instruction content.

[0107] In some instances, the availability data 220 or other data used to update the API data structure 108 may include data from a user, another computing device, or other source. For example, in some instances, the API data structure 108 may be updated based at least in part on user feedback data. For example, in some instances, a first API query based on a first request 102 may return a negative data record 210 that may cause the machine-learned agent 104 to suggest an alternative action or generate a subsequent message 218 seeking clarification of a request. In response to receiving user input approving the alternative action or clarification of the request 102, the machine-learned agent 104 or computing device may generate a subsequent message 218 asking the user whether the computing device would like the user's input to be remembered. In response to the user's request to remember the input, the computing device may replace the negative data record 210 with a positive data record 110 indicating an API tool 114 for performing an alternative action or for satisfying the clarified request 102. As another example, the machine-learned agent 104 may request user feedback on the quality of its performance, such as by asking the user whether a particular request 102 was fulfilled satisfactorily. In response to a user response indicating that a particular request 102 was not satisfactorily fulfilled, the data record 110 indicating the API tool 114 used to respond to the request 102 may be replaced with a negative data record 210 based on the user feedback. For example, in some instances, the user feedback may include an explanation of the reason for the user's dissatisfaction, an explanation of the user's preferences, or other user-specific clarification information. In such instances, the negative data record 210 may include instructional content based on the user-specific clarification information. For example, in some instances, the user-specific clarification information may be stored directly in the negative data record 210 so as to be provided as instructional content to the machine-learned agent 204. In some instances, the computing system may generate instructional content based on the user-specific clarification information (e.g., using a machine-learned sequence processing model, etc.).

[0108] Figure 31 is a block diagram of an example system for utilizing custom context for API retrieval. A machine-learned agent 104 may receive a request 102 and may provide one or more API queries 106 to an API data structure 108. Based on the API queries 106, the machine-learned agent 104 may receive a data record 110 indicating currently available API tools 114. Additionally, the machine-learned agent 104 may generate one or more custom context queries 322 based on the request 102 to retrieve one or more data records 326 from a custom context data structure 324 that includes custom context data. In some instances, the machine-learned agent may send a confirmation message 328 seeking confirmation that the data record 326 includes an appropriate custom context for satisfying the request, or a failure message 328 indicating that an appropriate custom context was not found. Based on the data record 326 and the data record 110, the machine-learned agent 104 may generate one or more API instructions 112 to cause one or more API tools 114 to perform one or more actions 116. In some examples, API tool 114 can generate success / failure data 119 (e.g., a message, etc.) indicating whether action 116 was successfully performed. Success / failure data 119 can be provided to machine-learned agent 104, which can generate one or more success / failure messages 118 based on success / failure data 119, data record 326, data record 110, or other appropriate data.

[0109] Custom context query 322 may include, for example, any data configured to access custom context data structure 324 to retrieve relevant data records 326. Custom context query 322 may include various types of data, such as machine-learned embedded data (e.g., vector, matrix, or tensor data; numeric or binary embedded data; etc.), natural language data, computer-readable instructions for accessing custom context data structure 324, and the like. For example, in some instances, custom context query 322 may include API instructions 112 for accessing custom context data structure 324 using API tool 114. For example, in some instances, machine-learned agent 104 may recognize that custom context is required to fulfill request 102 and may generate API query 106 for retrieving custom context (e.g., by generating a natural language query such as "get custom context," "retrieve contact information," "find Terry's phone number," etc.). As another example, in some instances, the data record 110 indicating the API tool 114 may include context for determining whether additional custom context is needed (e.g., instruction content, a few sample prompts, etc.). In some instances, the data record 110 indicating the API tool 114 may include context for generating the API instruction 112 or the API query 106 for retrieving custom context when needed. As an illustrative example, a data record 110 indicating an API tool for making phone calls may include one or more instructions for accessing a contact data structure that stores phone number information for a plurality of contacts.

[0110] In some instances, custom context query 322 may include machine-learned embeddings that are configured to be compared (e.g., based on a distance metric) to machine-learned embeddings contained in custom context data structure 324. For example, in some instances, custom context data structure 324 may include a semantic knowledge data structure, where each data entry of the semantic knowledge data structure includes one or more machine-learned embeddings associated with the data entry. In such instances, custom context query 322 may include machine-learned embeddings computed using a machine-learned model that is similar to (e.g., the same) as the machine-learned model used to compute the machine-learned embeddings of custom context data structure 324. In this way, for example, the machine-learned embeddings of custom context query 322 may share a semantic space with the machine-learned embeddings of custom context data structure 324, and the distance (e.g., vector distance, etc.) between the machine-learned embeddings in the shared semantic space may indicate the similarity of semantic meaning between the embeddings.

[0111] The custom context query 322 including the machine-learned embedding can be generated, for example, by one or more machine-learned sequence processing models (e.g., the machine-learned agent 104, etc.) based on the request 102. For example, in some instances, a first machine-learned sequence processing model can generate a natural language custom context query based on the request 102, and a second machine-learned sequence processing model (e.g., a sentence encoder such as sentence-T5, etc.) can generate a machine-learned embedding based on the natural language custom context query. However, this is not required. For example, in some instances, an input context can be provided to the first sequence processing model, which is configured to generate a natural language custom context query based on the request 102, and the machine-learned embedding can be extracted directly from an intermediate layer or hidden layer (e.g., the last layer before the output layer, etc.) of the sequence processing model. Other implementations are possible.

[0112] Generating a natural language custom context query using a first sequence processing model (e.g., via a machine learning agent 104 or a component thereof) can include, for example, providing an input context including all or part of a request 102 and additional context to the first sequence processing model. In some instances, the additional context can include one or more instructions (e.g., natural language instructions), such as instructions to determine whether a custom context is needed or instructions to generate a natural language custom context query. In some instances, the additional context can include a few-shot prompt. For example, the additional context can include a small number (e.g., 3, 5, 10, etc.) of example request-query pairs that illustrate examples of queries associated with example requests. In some instances, the additional context can include thought chaining prompts, such as a step-by-step inference chain based on one or more examples of one or more example requests 102.

[0113] Custom context data structure 324 can be, for example, a data structure (e.g., a database such as a vector database, a file, a file system, a collection of data objects, a hash table, a dictionary, a data structure including settings or configuration information, etc.) that holds a plurality of data records 326. In some examples, custom context data structure 324 can be, include, be comprised of, or be otherwise accessible via an API tool 114. For example, API tool 114 can include one or more API tools 114 for reading, writing, updating, deleting, or otherwise accessing custom context data structure 324.

[0114] In some instances, the data record 326 may include custom context data. Custom context data may include various context data that the machine-learning agent 104 or the device running the machine-learning agent has access to (e.g., according to user-defined permission data, etc.). For example, the custom context data may include settings data; contact data (e.g., phone contacts, email contacts, etc.); user preference data; historical data associated with the device or application; knowledge associated with a particular user or device, such as knowledge stored during previous user interactions with the machine-learning agent 104 or the device running the machine-learning agent; or other custom context. In some example implementations, the custom context information may include: user-specific navigation data for use with a navigation API (e.g., data associating "office" with a user's office location; data associating "my dentist" with a specific dentist's office; data associating "Ray's" with Famous Original Ray's Pizza; etc.); user-specific access authorization data for use with one or more APIs (e.g., login data, such as a username and password; security certificate data; API keys; etc.); user-specific data associating a device with a user-specific name for the device (e.g., associating "living room light" with LIGHT_CONTROL_004, etc.); user-specific preference data, settings data, or parameter values ​​for use with one or more APIs (e.g., font preference data for use with an email API, etc.); or other custom context data.

[0115] In some instances, data records 326 may include knowledge stored in a tuple format (e.g., pairs, triples, quads, etc.), such as knowledge relating a first value (e.g., a name, etc.) to a second value (e.g., a phone number, an email address, etc.). In some instances, data records 326 may include knowledge stored in a triple format, such as a first value-relationship-second value triple that describes a relationship between the first value and the second value. As an illustrative example, a data record 326 describing that "my dentist" corresponds to "Jane Smith, DDS" may be stored as a triple, such as: (user, dentist, "Jane Smith, DDS"). As another example, a contact phone number may be stored as a triple, such as ("Dennis", phone number, 555-321-0505). In some instances, custom context data structure 324 may include a knowledge graph. For example, the custom context data structure 324 may include a knowledge graph stored as first value-relationship-second value triples, where each first value or second value of the data record 326 may represent a node of the knowledge graph, and each relationship may represent an edge of the knowledge graph.

[0116] In some instances, retrieving data from a knowledge graph can include a multi-hop search. For example, retrieving data from a knowledge graph can include retrieving a first data record 326 based on a first custom context query 322; generating a second custom context query 322 based on the first data record 326; and retrieving a second data record 326 based on the second custom context query 322. As an illustrative example, fulfilling a request to "navigate to my dentist's office" can include first retrieving a first data record 326 including a triple such as (user, dentist, Jane Smith, DDS), and then retrieving a second data record 326 including a second triple such as (Jane Smith, DDS, office location, 123 Main Street). In some instances, the multi-hop search can include providing the request 102 to the machine-learned agent 104 along with additional context, such as instruction content, few-shot prompts, thought chaining prompts, and the like. For example, the instruction content may include instructions to identify any custom context required to fulfill request 102; decompose the custom context into component parts; and generate a first custom context query 322 based on the first component parts. As another example, the thought chaining prompt may include one or more example multi-hop traversals of a knowledge graph based on the custom context data structure 324 of one or more example requests 102 (e.g., example custom context queries 322, example data records 326, etc.).

[0117] In some instances, data records 326 may include negative data records 326 indicating that custom context data is unavailable or cannot be confidently selected based on the request. In some instances, negative data records 326 may have any of the attributes described above with respect to negative data records 210. For example, negative data records 326 may include instruction content requesting a subsequent action (e.g., clarification, etc.); instruction content for generating failure message 218; instruction content for suggesting an alternative action (e.g., suggesting a grocery store when a data record 326 for "grocery store" cannot be retrieved); and instruction content for suggesting a grocery store when a data record 326 for "grocery store" cannot be retrieved.

[0118] In some instances, the machine-learned agent 104 may modify the custom context data structure 324 or otherwise cause the custom context data structure 324 to be modified. For example, in some instances, the API data structure 108 may include one or more data records 110 that indicate one or more API tools 114 for creating, editing, or deleting a data record 326 for the custom context data structure 324. As an example, a request 102 to "save Dennis's phone number as 555-321-0505" may cause the machine-learned agent 104 to write a new data record 326; edit an existing data record 326; or otherwise cause the creation or editing of a data record 326 (e.g., via the API tool 114). As another example, the machine-learned agent 104 may cause the creation, editing, or deletion of a data record 326 based on data learned during a user interaction. For example, if Dennis's phone number is not already stored in the custom context data structure 324, or if there is data for more than one phone number for "Dennis," the request 102 to "call Dennis" may fail. In such an instance, the machine-learned agent 104 may generate a subsequent message 218 requesting Dennis's phone number or requesting other clarification. Upon receiving the clarification, the machine-learned agent 104 may generate a subsequent message 218 asking the user whether to save Dennis's phone number in the custom context data structure 324. In response to the user's request to save Dennis's phone number, the machine-learned agent 104 may generate API instructions 112 to cause a data record 326 including Dennis's phone number (e.g., a data record clarifying which "Dennis" should be selected by default, a new data record storing a new phone number, etc.) to be added to the custom context data structure 324.

[0119] In some examples, confirmation / failure message 328 can include failure / follow-up message 218 and can be generated in a similar (e.g., identical) manner as failure / follow-up message 218. For example, the request for clarification of custom context information can be made in a similar (e.g., identical) manner as described above with respect to follow-up message 218.

[0120] In some instances, a confirmation message 328 can be generated in response to retrieving a positive data record 326 that includes custom context data. The confirmation message 328 can include, for example, a message that includes all or part of the custom context data for the data record 326; a request for confirmation that the correct custom context data has been retrieved; and any other appropriate message data. As an illustrative example, the machine-learned agent 104 can process a request 102 for "navigate to my dentist"; retrieve a data record 326 that associates the user's dentist with "Jane Smith" based on the request 102; and generate a confirmation message 328 that states "Would you like me to navigate to Jane Smith's office at 123 Main Street?" In response to receiving user confirmation input (e.g., touchscreen input, audible verbal input, etc.), the machine-learned agent 104 can generate API instructions 112 based on the data record 326 and the request 102.

[0121] In some instances, the API tool may return success / failure data 119 indicating whether the action 116 has been successfully performed. For example, in some instances, the API tool 114 may be configured to return an error message or error code if the API instruction 112 cannot be successfully executed. In such instances, the computing system may provide a failure message 118 to the user based on the error message or error code. In some instances, generating the failure message 118 based on the error message from the API tool may include providing the error message along with additional input context (such as instruction context, few-shot prompt context, thought chain prompt context, etc.) to the machine-learned agent 104. In some instances, the failure message 118 may be generated without using the machine-learned agent 104. For example, in some instances, the error message from the API tool 114 may be provided directly to the user. In some instances, the failure message 118 may be generated based on a data structure that associates the error message or error code with a user message associated with the error message or error code. As an illustrative example, an HTTP 404 error code may be associated with data indicating a human-readable message that the resource cannot be found. Similarly, the API tool 114 can be configured to return a success message or success code if the API instruction 112 is successfully executed. In such an instance, the computing system can generate a success message 118 based on the success data 119 in a manner similar to (e.g., the same as) that described above for the failure message 118.

[0122] Example Method

[0123] Figure 4 A flow chart depicts an example method for operating based on negative data records according to an example embodiment of the present disclosure. Figure 4 The steps are depicted in a specific order for purposes of illustration and discussion, but the method of the present disclosure is not limited to the particular order or arrangement shown. The various steps of the example method 400 can be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0124] At 402, example method 400 may include receiving, by a computing system including one or more computing devices, a first request to perform at least one task (e.g., request 102), the first request including a natural language description of the at least one task. In some examples, example method 400 at 402 may include using one or more systems or performing Figure 2 Describes one or more activities.

[0125] At 404, the example method 400 may include generating, by the computing system, a query embedding (e.g., the API query 106), the query embedding including an embedded description of actions to be performed to perform the task. In some instances, the example method 400 may include generating, at 404, a query embedding (e.g., the API query 106) using one or more systems or processes related to the query. Figure 2 Describes one or more activities.

[0126] At 406, example method 400 can include querying, by a computing system and using a query, a data structure comprising an API embedding (e.g., API data structure 108), wherein the API embedding comprises an embedded description of functionality associated with a corresponding application programming interface (API). In some instances, the data structure can include a false positive API embedding comprising an embedded description of functionality that is not available via the corresponding API, wherein the false positive API embedding is associated with instruction content indicating that the functionality is not available. In some instances, the false positive API embedding and the corresponding instruction content can be, include, or consist of a negative data record 210. In some instances, example method 400 can include, at 406, using one or more systems or performing Figure 2 Describes one or more activities.

[0127] At 408, the example method 400 may include obtaining, by the computing system, instruction content based on the similarity between the query and the false positive API embedding. In some instances, the example method 400 may include using one or more systems or performing Figure 2 Describes one or more activities.

[0128] At 410, the example method 400 may include inputting, by a computing system, the instruction content into a machine-learned sequence processing model (e.g., the machine-learned agent 104) to generate an intermediate output. In some instances, the example method 400 may include, at 410, using one or more systems or performing Figure 2 Describes one or more activities.

[0129] At 412, the example method 400 may include performing, by the computing system, one or more operations identified in the intermediate output. In some instances, the example method 400 may include performing, at 412, using one or more systems or performing Figure 2 Describes one or more activities.

[0130] Figure 5 Depicted is a flow chart of an example method for using an API tool to perform a requested action according to an example embodiment of the present disclosure. Figure 5The steps are depicted in a specific order for purposes of illustration and discussion, but the method of the present disclosure is not limited to the particular order or arrangement shown. The various steps of the example method 500 can be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0131] At 502, the example method 500 may include receiving, by a computing system including one or more computing devices, a first request to perform at least one task (e.g., request 102), the first request including a natural language description of the at least one task. In some examples, the example method 500 at 502 may include using one or more systems or performing Figure 1 Describes one or more activities.

[0132] At 504, the example method 500 may include generating, by the computing system, a query embedding (e.g., the API query 106), the query embedding including an embedded description of actions to be performed to perform the task. In some instances, the example method 500 may include generating, at 504, a query embedding (e.g., the API query 106) using one or more systems or processes related to the query. Figure 1 Describes one or more activities.

[0133] At 506, example method 500 may include querying, by a computing system, a data structure (e.g., API data structure 108) including API embeddings using a query, wherein the API embeddings include embedded descriptions of functionality associated with corresponding application programming interfaces (APIs), wherein at least one API embedding is associated with instruction content for accessing at least one API. In some instances, the API embeddings and corresponding instruction content may be, include, or consist of data records 110. In some instances, example method 500 may include querying, using one or more systems or performing a query regarding Figure 1 Describes one or more activities.

[0134] At 508, the example method 500 may include obtaining, by the computing system, instruction content based on the similarity between the query and the at least one API embedding. In some instances, the example method 500 may include, at 508, using one or more systems or performing Figure 1 Describes one or more activities.

[0135] At 510, the example method 500 may include inputting, by a computing system, the instruction content into a machine-learned sequence processing model to generate an intermediate output. In some instances, the intermediate output may be, include, or consist of the API instructions 112. In some instances, the example method 500 may include, at 510, using one or more systems or performing Figure 1 Describes one or more activities.

[0136] At 512, the example method 500 may include performing, by the computing system, one or more operations identified in the intermediate output (e.g., act 116). In some instances, the example method 500 at 512 may include using one or more systems or performing Figure 1 Describes one or more activities.

[0137] Figure 6 Depicted is a flow chart of an example method for taking actions based on user-specific knowledge according to an example embodiment of the present disclosure. Figure 6 The steps are depicted in a specific order for purposes of illustration and discussion, but the method of the present disclosure is not limited to the particular order or arrangement shown. The various steps of the example method 600 can be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0138] At 602, the example method 600 may include receiving, by a computing system including one or more computing devices, a first request to perform at least one task (e.g., request 102), the first request including a natural language description of the at least one task. In some examples, the example method 600 at 602 may include using one or more systems or performing Figure 3 Describes one or more activities.

[0139] At 604, the example method 600 may include querying, by the computing system, at least in part based on the request (e.g., using the custom context query 322), at least one data structure (e.g., the custom context data structure 324) that includes user-specific knowledge associated with the user. In some instances, the example method 600 at 604 may include using one or more systems or processes related to the user. Figure 3 Describes one or more activities.

[0140] At 606, the example method 600 may include retrieving, by the computing system, at least one data record (e.g., data record 326) including user-specific knowledge related to at least one task based at least in part on the query. In some instances, the example method 600 at 606 may include using one or more systems or performing Figure 3 Describes one or more activities.

[0141] At 608, the example method 600 may include generating, by the computing system, at least one instruction (e.g., API instruction 112) based at least in part on the at least one data record using one or more machine-learned sequence processing models, the at least one instruction being configured to cause at least one device to perform an action (e.g., action 116) associated with at least one task when provided to an API (e.g., API tool 114). In some instances, the example method 600 may include, at 608, using one or more systems or processes related to Figure 3 Describes one or more activities.

[0142] At 610, the example method 600 may include providing instructions to the API by the computing system. In some instances, the example method 600 may include using one or more systems or performing Figure 3 Describes one or more activities.

[0143] Figure 7 A flow chart of a method 700 for training one or more machine-learned models according to aspects of the present disclosure is depicted. For example, an example machine-learned model can include a machine-learned agent 104.

[0144] One or more portions of the example method 700 can be implemented by a computing system comprising one or more computing devices (such as, for example, the computing systems described with reference to other figures). Each respective portion of the example method 700 can be performed by any one (or any combination) of the one or more computing devices. In addition, one or more portions of the example method 700 can be implemented on hardware components of the devices described herein, for example, to train one or more systems or models. Figure 7 The elements are depicted in a particular order for purposes of illustration and discussion. One of ordinary skill in the art, using the disclosure provided herein, will understand that the elements of any of the methods discussed herein may be adapted, rearranged, expanded, omitted, combined, or modified in various ways without departing from the scope of the present disclosure. Figure 7 The elements / terms described with reference to other systems and figures are described for illustrative purposes and are not intended to be limiting. One or more portions of the example method 700 may additionally or alternatively be performed by other systems.

[0145] At 702, example method 700 may include obtaining training instances. The set of training data may include multiple training instances divided between multiple data sets (e.g., a training data set, a validation data set, or a test data set). The training instances may be labeled or unlabeled. Although referred to as "training" instances in example method 700, it should be understood that runtime reasoning may form training instances when a model is trained using an evaluation of the model's performance on that runtime instance (e.g., online training / learning). Example data types for training instances and various tasks associated therewith are described throughout this disclosure.

[0146] At 704, example method 700 may include processing the training instance using one or more machine-learned models to generate an output. The output may be obtained directly from the one or more machine-learned models, or may be a downstream result of a chain of processing operations that includes the output of the one or more machine-learned models.

[0147] At 706, example method 700 may include receiving an evaluation signal associated with the output. The evaluation signal may be obtained using a loss function. Various determinations of loss may be used, such as mean squared error, likelihood loss, cross entropy loss, hinge loss, contrastive loss, or various other loss functions. The evaluation signal may be calculated using known baseline true labels (e.g., supervised learning), predicted or estimated labels (e.g., semi-supervised learning or self-supervised learning), or without labels (e.g., unsupervised learning). The evaluation signal may be a reward (e.g., for reinforcement learning). The reward may be calculated using a machine-learned reward model configured to generate a reward based on the received output. The reward may be calculated using feedback data describing human feedback on the output.

[0148] At 708, the example method 700 can include using the evaluation signal to update the machine-learned model. For example, in some embodiments, various training or learning techniques (such as, for example, backpropagation) can be used to learn the values ​​of the parameters of the machine-learned model. For example, the evaluation signal can be backpropagated from the output (or another source of the evaluation signal) through the machine-learned model to update one or more parameters of the model (e.g., based on the gradient of the evaluation signal with respect to the parameter value). For example, a system including one or more machine-learned models can be trained in an end-to-end manner. Gradient descent techniques can be used to iteratively update the parameters over multiple training iterations. In some implementations, performing error backpropagation can include performing truncated backpropagation through time. The example method 700 can include implementing a variety of generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization ability of the model being trained.

[0149] In some implementations, example method 700 can be implemented to train a machine-learned model from an initialized state to a fully trained state (e.g., when the model exhibits a desired performance profile, such as based on accuracy, precision, recall, etc.).

[0150] In some implementations, the example method 700 may be implemented for specific stages of the training process. For example, in some implementations, the example method 700 may be implemented for pre-training a machine-learned model. Pre-training may include, for example, large-scale training on potentially noisy data to achieve a broad base of performance levels across a variety of tasks / data types. In some implementations, the example method 700 may be implemented for fine-tuning a machine-learned model. Fine-tuning may include, for example, smaller-scale training on higher quality (e.g., labeled, curated, etc.) data. Fine-tuning may affect all or a portion of the parameters of the machine-learned model. For example, various parts of the machine-learned model may be "frozen" for certain training phases. For example, parameters associated with the embedding space may be "frozen" during fine-tuning (e.g., to retain information learned from a wider domain than that present in the fine-tuning dataset). Example fine-tuning methods include reinforcement learning. Reinforcement learning may be based on user feedback on model performance during use.

[0151] Example machine learning model

[0152] Figure 8 is a block diagram of an example processing flow for processing input 2 using a machine learning model 1 to generate output 3.

[0153] The machine learning model 1 can be or include one or more machine learning models or model components. Example machine learning models can include neural networks (e.g., deep neural networks). Example machine learning models can include nonlinear models or linear models. Example machine learning models can use other architectures instead of or in addition to neural networks. Example machine learning models can include decision tree-based models, support vector machines, hidden Markov models, Bayesian networks, linear regression models, k-means clustering models, etc.

[0154] Example neural networks may include feedforward neural networks, recurrent neural networks (RNNs) (including recurrent neural networks based on long short-term memory (LSTM)), convolutional neural networks (CNNs), diffusion models, generative adversarial networks, or other forms of neural networks. Example neural networks may be deep neural networks. Some example machine learning models may utilize attention mechanisms, such as self-attention. For example, some example machine learning models may include multi-head self-attention models.

[0155] The machine-learned model 1 may include a single or multiple instances of the same model configured to operate on data from the input 2. The machine-learned model 1 may include an ensemble of different models that can collaboratively interact to process data from the input 2. For example, the machine-learned model 1 may adopt a mixed expert structure. For example, see Zhou et al., Mixture-of-Experts with Expert Choice Routing, ARX IV :2202.09368v2 (October 14, 2022).

[0156] Input 2 may generally include or otherwise represent various types of data. Input 2 may include one type of data or many different types of data. Output 3 may be the same type of data or a different type of data than input 2. Output 3 may include one type of data or many different types of data.

[0157] Example data types for input 2 or output 3 include natural language text data, software code data (e.g., source code, object code, machine code, or any other form of computer-readable instructions or programming language), machine code data (e.g., binary code, assembly code, or other form of machine-readable instructions that can be directly executed by a central processing unit of a computer), assembly code data (e.g., a low-level programming language that uses symbolic representations of machine code instructions to program a processing unit), genetic data or other chemical or biochemical data, image data, audio data, audiovisual data, tactile data, biometric data, medical data, financial data, statistical data, geographic data, astronomical data, historical data, sensor data generally (e.g., digital or analog values, such as voltage or other absolute or relative level measurements from real or artificial inputs (such as from audio sensors, light sensors, displacement sensors, etc.)), etc. The data can be raw or processed and can be in any format or mode.

[0158] In the multimodal input 2 or output 3, example combinations of data types include image data and audio data, image data and natural language data, natural language data and software code data, image data and biometric data, sensor data and medical data, etc. It should be understood that any combination of data types in the input 2 or output 3 can exist.

[0159] Example input 2 may include one or more data types, such as the example data types noted above. Example output 3 may include one or more data types, such as the example data types noted above. The data type of input 2 may be the same as or different from the data type of output 3. It should be understood that the example data types noted above are provided for illustrative purposes only. The data types contemplated within the scope of the present disclosure are not limited to those examples noted above.

[0160] Example machine learning sequence processing model

[0161] Figure 9 is a block diagram of an example implementation of an example machine-learned model configured to process a sequence of information. For example, an example implementation of a machine-learned model 1 may include a machine-learned sequence processing model 4. The example system may pass input 2 to sequence processing model 4. Sequence processing model 4 may include one or more machine-learned components. Sequence processing model 4 may process data from input 2 to obtain an input sequence 5. Input sequence 5 may include one or more input elements 5-1, 5-2, ..., 5-M, etc. obtained from input 2. Sequence processing model 4 may use a prediction layer 6 to process input sequence 5 to generate an output sequence 7. Output sequence 7 may include one or more output elements 7-1, 7-2, ..., 7-N, etc. generated based on input sequence 5. The system may generate output 3 based on output sequence 7.

[0162] The sequence processing model 4 may include one or more machine-learned model components configured to ingest, generate, or otherwise infer sequences of information. For example, some example sequence processing models in the text domain are referred to as "large language models" or LLMs. See, for example, Palm 2 Technical Report, GOOGLE, https: / / ai.google / static / documents / palm2techr eport.pdf(nd). Other example sequence processing models may operate in other domains, such as, for example, the image domain, see, for example, Dosovitskiy et al., An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, AR X IV :2010.11929v2 (June 3, 2021); audio domain, see, for example, MusicLM: Generating Music From Text by Agostinelli et al., AR XIV :2301.11325v1 (January 26, 2023); Biochemical domain, for example, see Jumper et al. Highly accurate protein structure prediction with AlphaFold (Highly accurate protein structure prediction with AlphaFold), 596 Nature 583 (August 26, 2021). The sequence processing model 4 can process one or more types of data simultaneously. The sequence processing model 4 can include a relatively large model (e.g., more parameters, heavy computation, etc.), a relatively small model (e.g., fewer parameters, light computation, etc.), or both.

[0163] In general, the sequence processing model 4 can use the data from the input 2 to obtain the input sequence 5. For example, the input sequence 5 can include a representation of the data from the input 2 in a format understood by the sequence processing model 4. One or more machine-learned components of the sequence processing model 4 can ingest the data from the input 2, parse the data into segments compatible with the processing architecture of the sequence processing model 4 (e.g., via "lemmatization"), and project the segments into an input space associated with the prediction layer 6 (e.g., via "embedding").

[0164] The sequence processing model 4 can ingest data from the input 2 and parse the data into a sequence of elements to obtain an input sequence 5. For example, a portion of the input data from the input 2 can be decomposed into fragments that collectively represent the content of the portion of the input data. The fragments can provide elements of the sequence.

[0165] In some cases, elements 5-1, 5-2, ..., 5-M may represent building blocks for capturing or expressing meaningful information in a particular data domain. For example, an element may describe an "atomic unit" across one or more domains. For example, for a text input source, an element may correspond to a group of one or more word or subword components (such as a set of one or more characters).

[0166] For example, elements 5-1, 5-2, ..., 5-M can represent tokens obtained using a tokenizer. For example, a tokenizer can process a given portion of an input source and output a series of tokens representing the portion of the input source (e.g., corresponding to input elements 5-1, 5-2, ..., 5-M). Various methods can be used to perform tokenization. For example, a text input source can be tokenized using byte pair encoding (BPE) techniques. For example, see Kudo et al., SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (System Demonstration) (P ROCEEDINGS O F THE2018CONFERENCE ON EMPIRICAL METHODS IN NATURAL LANG UAGE P ROCESSING (System Demonstrations), pp. 66–71 (October 31–November 4, 2018), https: / / aclanthology.org / D18-2012.pdf. An image-based input source can be tokenized by extracting patches from the image and serializing them.

[0167] In general, any data type can be serialized and processed into an input sequence 5. It should be understood that Figure 9 The elements 5-1, 5-2, ..., 5-M depicted in may be word-grams or may be embedded representations thereof.

[0168] The prediction layer 6 can predict one or more output elements 7-1, 7-2, ..., 7-N based on the input elements. The prediction layer 6 can include one or more machine-learned model architectures, such as one or more learning parameter layers, which manipulate and transform the input to extract high-level meaning from the input elements 5-1, 5-2, ..., 5-M and the relationships between the input elements. In this way, for example, the example prediction layer 6 can predict new output elements based on the context provided by the input sequence 5.

[0169] The prediction layer 6 can evaluate associations between parts of the input sequence 5 and specific output elements. These associations can inform predictions about the likelihood that a specific output follows from the input context. For example, consider the text snippet "The carpenter's toolbox was small and heavy. It was full of ___." The example prediction layer 6 can recognize that "it" refers back to "toolbox" by determining the relationship between the corresponding embeddings. The example prediction layer 6 can also relate "it" to attributes of the toolbox, such as "small" and "heavy". Based on these associations, for example, the prediction layer 6 can assign a higher probability to the word "nails" than to the word "sawdust".

[0170] The Transformer is an example architecture that can be used in the prediction layer 4. See for example Vaswani et al., Attention Is All You Need, ARX IV :1706.03762v7 (August 2, 2023). The Transformer is an example of a machine learning model architecture that uses an attention mechanism to compute associations between items within a context window. The context window can include a sequence containing an input sequence 5 and potentially one or more output elements 7-1, 7-2, ..., 7-N. The Transformer block can include one or more attention layers and one or more post-attention layers (e.g., feed-forward layers such as multilayer perceptrons).

[0171] The prediction layer 6 may include other machine learning model architectures in addition to or in place of transformer-based architectures. For example, recurrent neural networks (RNNs) and long short-term memory (LSTM) models, as well as convolutional neural networks (CNNs), may also be used. In general, the prediction layer 6 may utilize various artificial neural networks that can understand or generate information sequences.

[0172] The output sequence 7 may include or otherwise represent the same or different data types as the input sequence 5. For example, the input sequence 5 may represent text data, and the output sequence 7 may represent text data. The input sequence 5 may represent image, audio, or audio-visual data, and the output sequence 7 may represent text data (e.g., describing the image, audio, or audio-visual data). It should be understood that the prediction layer 6 and any other gap model components of the sequence processing model 4 may be configured to receive multiple data types in the input sequence 5 and output multiple data types in the output sequence 7.

[0173] Output sequence 7 may have various relationships with input sequence 5. Output sequence 7 may be a continuation of input sequence 5. Output sequence 7 may be a supplement to input sequence 5. Output sequence 7 may translate, transform, expand, or otherwise modify input sequence 5. Output sequence 7 may answer, evaluate, confirm, or otherwise respond to input sequence 5. Output sequence 7 may implement instructions provided via input sequence 5 (or describe instructions for implementing such instructions).

[0174] The output sequence 7 can be generated autoregressively. For example, for some applications, the output of one or more prediction layers 6 can be passed through one or more output layers (e.g., a softmax layer) to obtain a probability distribution of an output vocabulary (e.g., a text or symbol vocabulary) conditioned on the set of input elements in the context window. In this way, for example, the output sequence 7 can be generated autoregressively by sampling a possible next output element, adding that element to the context window, and regenerating the probability distribution based on the updated context window, sampling a possible next output element, and so on.

[0175] The output sequence 7 can also be generated non-autoregressively. For example, multiple output elements of the output sequence 7 can be predicted together without explicit order conditions with respect to each other. For example, see Saharia et al. Non-Autoregressive Machine Translation with Latent Alignments, ARX IV :2004.07437v3 (November 16, 2020).

[0176] The output sequence 7 may include one or more parts or elements. In an example content generation configuration, the output sequence 7 may include multiple elements corresponding to multiple parts of the generated output sequence (e.g., text sentences, values ​​of discrete waveforms, computer code, etc.). In an example classification configuration, the output sequence 7 may include a single element associated with a classification output. For example, the output "vocabulary" may include the set of classes into which the input sequence will be classified. For example, the visual transformer block may pass the latent state information to a multilayer perceptron, which outputs possible class values ​​associated with the input image.

[0177] Figure 10is a block diagram of an example technique for populating an example input sequence 8. The input sequence 8 may include various functional elements that form part of the model infrastructure, such as element 8-0 obtained from a task indicator 9, which signals to any model processing the input sequence 8 that a specific task is being performed (e.g., to help adapt the model's performance to that specific task). The input sequence 8 may include various data elements from different data modalities. For example, the input modality 10-1 may include one data modality. The data-to-sequence model 11-1 may process the data from the input modality 10-1 to project the data into a format compatible with the input sequence 8 (e.g., one or more vectors whose dimensions are determined based on the dimensions of the input sequence 8) to obtain elements 8-1, 8-2, and 8-3. Another input modality 10-2 may include a different data modality. The data-to-sequence model 11-2 may project the data from the input modality 10-2 into a format compatible with the input sequence 8 to obtain elements 8-4, 8-5, and 8-6. Another input modality 10-3 may include yet another different data modality. The data-to-sequence model 11-3 may project the data from the input modality 10-3 into a format compatible with the input sequence 8 to obtain elements 8-7, 8-8, 8-9.

[0178] Input sequence 8 may be the same as or different from input sequence 5. Input sequence 8 may be a multimodal input sequence that includes elements representing data from different modalities using a common dimensional representation. For example, the embedding space may have P dimensions. Input sequence 8 may be configured to include multiple elements having P dimensions. In this way, for example, example implementations may facilitate information extraction and inference across different data modalities by projecting data into elements in the same embedding space to perform comparisons, combinations, or other computations between them.

[0179] For example, elements 8-0, ..., 8-9 can indicate specific locations within the multidimensional embedding space. Some elements can be mapped to a discrete set of locations in the embedding space. For example, elements corresponding to discrete members in a predetermined word-gram vocabulary can be mapped to discrete locations in the embedding space associated with those words. Other elements can be distributed continuously across the embedding space. For example, some data types can be decomposed into continuously defined parts (e.g., image patches), which can be described using continuously distributed locations in the embedding space.

[0180] In some implementations, the expressive power of an embedding space may not be limited to the meaning associated with any particular set of word-grams or other building blocks. For example, a continuous embedding space can encode a sequence of high-order information. Individual pieces of information (e.g., word-grams) can be mapped to specific points in the space: for example, the word-gram of the word "dog" can be projected to an embedded value that points to a specific location in the embedding space associated with canine-related information. Similarly, an image patch of an image of a dog on grass can also be projected into the embedding space. In some implementations, the projection of the image of the dog can be similar to the projection of the word "dog" while also having similarity to the projection of the word "grass", but potentially different from both. In some implementations, the projection of the image patch may not be completely aligned with any single projection of a single word. In some implementations, the projection of the image patch may be aligned with a combination of the projections of the words "dog" and "grass". In this way, for example, a high-order embedding space can encode information that can be independent of the data modality in which the information is expressed.

[0181] The task indicator 9 may include a model or model component configured to identify the task being performed and inject an input value represented by element 8-0 into the input sequence 8, the element signaling which task is being performed. For example, the input value may be provided as a data type associated with the input modality and projected along with the input modality (e.g., the input value may be a textual task label embedded along with other textual data in the input; the input value may be a pixel-based representation of the task embedded along with other image data in the input; etc.). The input value may be provided as a data type that is different from or at least independent of the other inputs. For example, the input value represented by element 8-0 may be learned within a continuous embedding space.

[0182] Input modalities 10 - 1 , 10 - 2 , and 10 - 3 may be associated with various different data types (eg, as described above with respect to input 2 and output 3 ).

[0183] The data-to-sequence models 11-1, 11-2, and 11-3 can be the same or different from each other. The data-to-sequence models 11-1, 11-2, and 11-3 can adapt to each corresponding input modality 10-1, 10-2, and 10-3. For example, the text data-to-sequence model can segment a portion of the input text and project the segmentation into elements in the input sequence 8 (e.g., elements 8-1, 8-2, 8-3, etc.). The image data-to-sequence model can segment the input image and project the segmentation into elements in the input sequence 8 (e.g., elements 8-4, 8-5, 8-6, etc.). The arbitrary data type data-to-sequence model can segment the input of the arbitrary data type and project the segmentation into elements in the input sequence 8 (e.g., elements 8-7, 8-8, 8-9, etc.).

[0184] The data-to-sequence models 11-1, 11-2, and 11-3 may form part of the machine-learned sequence processing model 4. The data-to-sequence models 11-1, 11-2, and 11-3 may be trained jointly with the machine-learned sequence processing model 4 or independently of the machine-learned sequence processing model 4. The data-to-sequence models 11-1, 11-2, and 11-3 may be trained end-to-end with the machine-learned sequence processing model 4.

[0185] Example machine learning model development platform

[0186] Figure 11 1 is a block diagram of an example model development platform 12 that can facilitate the creation, adaptation, and refinement of example machine-learned models (e.g., machine-learned model 1, sequence processing model 4, etc.). Model development platform 12 can provide a number of different tool kits that a developer system can use to develop new or adapted machine-learned models.

[0187] The model development platform 12 may provide one or more model libraries 13 containing building blocks for new models. The model library 13 may include one or more pre-trained base models 13-1, which may provide a backbone of processing capabilities across a variety of tasks. The model library 13 may include one or more pre-trained expert models 13-2, which may focus on performance in a specific domain of expertise. The model library 13 may include various model primitives 13-3, which may provide low-level architectures or components (optionally pre-trained) that may be assembled in various arrangements as needed.

[0188] The model development platform 12 may receive a selection of various model components 14. The model development platform 12 may pass the selected model components 14 to a workbench 15, which combines the selected model components 14 into a development model 16.

[0189] The workbench 15 may facilitate further refinement and adaptation of the development model 16 by utilizing a number of different tool suites integrated with the model development platform 12. For example, the workbench 15 may facilitate aligning the development model 16 with desired performance profiles on various tasks using a model alignment tool suite 17.

[0190] The model alignment toolkit 17 can provide a variety of tools for causing the development model 16 to generate outputs that are aligned with desired behavioral characteristics. Alignment can include increasing the accuracy, precision, recall, etc. of the model output. Alignment can include enforcing output styles, patterns, or other preferred characteristics of the model output. Alignment can be general or domain-specific. For example, the pre-trained base model 13-1 can start from an initial performance level across multiple domains. Alignment of the pre-trained base model 13-1 can include improving performance in a specific information or task domain (e.g., even at the expense of performance in another information or task domain).

[0191] The model alignment toolkit 17 can integrate one or more datasets 17-1 for aligning the development model 16. The curated datasets 17-1 can include labeled or unlabeled training data. The datasets 17-1 can be obtained from public domain datasets. The datasets 17-1 can also be obtained from private datasets associated with one or more developer systems for aligning custom machine learning models tailored for private use cases.

[0192] Pre-training pipeline 17-2 may include a machine learning model training workflow configured to update development model 16 on a large-scale, potentially noisy dataset. For example, pre-training may utilize unsupervised learning techniques (e.g., denoising) to process a large number of training examples to update model parameters from an initialized state and achieve a desired baseline performance. Pre-training pipeline 17-2 may utilize an unlabeled dataset from dataset 17-1 for pre-training. Workbench 15 may implement pre-training pipeline 17-2 to pre-train development model 16.

[0193] Fine-tuning pipeline 17-3 may include a machine-learned model training workflow configured to utilize higher-quality data to refine model parameters of development model 16. Fine-tuning pipeline 17-3 may update development model 16 by performing supervised training using labeled datasets from dataset 17-1. Fine-tuning pipeline 17-3 may also update development model 16 by performing reinforcement learning using reward signals derived from user feedback signals. Workbench 15 may implement fine-tuning pipeline 17-3 to fine-tune development model 16.

[0194] The hint library 17-4 may include a set of inputs configured to induce behavior aligned with a desired performance criterion. The hint library 17-4 may include few-shot hints (e.g., inputs that provide examples of desired model outputs so that the head can be appended to the desired runtime query), thought chain hints (e.g., inputs that provide step-by-step reasoning within an example to encourage the model to make comprehensive inferences), and the like.

[0195] Example prompts may be retrieved from the available repository of prompts library 17 - 4 . The example prompts may be facilitated by one or more developer systems using workbench 15 .

[0196] In some implementations, a pre-trained or fine-tuned model can achieve satisfactory performance without examples in the input. For example, a zero-shot prompt can include an input that lacks examples. The zero-shot prompt can be within the domain of the training dataset or outside the training domain.

[0197] The prompt library 17-4 may include one or more prompt engineering tools. The prompt engineering tools may provide a workflow for retrieving or learning optimized prompt values. The prompt engineering tools may facilitate direct learning of prompt values ​​(e.g., input element values) based on one or more training iterations. The workbench 15 may implement the prompt engineering tools in the development model 16.

[0198] The prompt library 17-4 can include a pipeline for prompt generation. For example, the development model 16 itself or another machine-learned model can be used to generate inputs. In this way, for example, a first model can process information about a task and output inputs for a second model to process in order to perform the steps of the task. The second model can be the same as or different from the first model. The workbench 15 can implement the prompt generation pipeline in the development model 16.

[0199] The prompt library 17-4 may include a pipeline for context injection. For example, the performance of the development model 16 on a particular task may be improved if provided with additional context for performing the task. The prompt library 17-4 may include software components configured to identify the desired context, retrieve the context from an external source (e.g., a database, a sensor, etc.), and add the context to the input prompt. The workbench 15 may implement the context injection pipeline in the development model 16.

[0200] Although various training examples described herein with respect to the model development platform 12 relate to "pre-training" and "fine-tuning," it should be understood that the model alignment toolkit 17 can generally support a variety of training techniques adapted to train a variety of machine-learned models. The example training techniques can correspond to the example training method 700 described above.

[0201] The model development platform 12 may include a model plug-in toolkit 18. The model plug-in toolkit 18 may include a variety of tools configured to enhance the functionality of machine-learned models by integrating them with other systems, devices, and software components. For example, machine-learned models can use tools to increase performance quality where appropriate. For example, deterministic tasks can be offloaded to specialized tools, instead of performing the tasks probabilistically where the risk of error increases. For example, rather than autoregressively predicting the solution to a system of equations, a machine-learned model can identify the tools to call to obtain the solution and pass the system of equations to the appropriate tool. The tool may be a traditional system of equations solver that operates deterministically to solve the system of equations. The output of the tool may be returned in response to the original query. In this way, tool usage can allow some example models to focus on the strengths of machine-learned models—for example, understanding the intent in unstructured requests for tasks—while enhancing the model's performance by offloading certain tasks to more focused tools that mechanically apply deterministic algorithms to well-defined problems.

[0202] The model plug-in tool suite 18 may include a validation tool 18-1. The validation tool 18-1 may include a tool that can parse and validate the output of a machine-learned model. The validation tool 18-1 may include engineered heuristics that establish certain thresholds to be applied to the model output. For example, the validation tool 18-1 may ground the output of the machine-learned model against a structured data source (e.g., to mitigate "hallucinations").

[0203] The model plug-in toolkit 18 may include a toolkit 18-2 for implementing one or more tools, which may include scripts or other executable code that can be executed in conjunction with the development model 16. The toolkit 18-2 may include one or more inputs configured to enable the machine learning model to implement the tool (e.g., a few-shot prompts that induce the model to output tool calls with the correct syntax, etc.). For example, the toolkit 18-2 may include fine-tuning training data for training the model to use the tool.

[0204] The model plug-in tool suite 18 may include an interface for calling an external application programming interface (API) 18-3. For example, in addition to or instead of directly implementing tool calls or tool code using the development model 16, the development model 16 may be aligned with output instructions that initiate API calls to send or obtain data via an external system.

[0205] The model plug-in toolkit 18 can be integrated with the hint library 17-4 to build a catalog of available tools for use with the development model 16. For example, the model can receive a catalog of available tools as input, and the model can generate an output that selects a tool from the available tools and initiates a tool call for using the tool.

[0206] The model development platform 12 may include a suite of computational optimization tools 19 for optimizing the computational performance of the development model 16. For example, tools for model compression 19-1 may allow the development model 16 to be reduced in size while maintaining a desired performance level. For example, model compression 19-1 may include quantization workflows, weight pruning, and sparsification techniques. Tools for hardware acceleration 19-2 may facilitate the configuration of model storage and execution formats to optimally operate on different hardware resources. For example, hardware acceleration 19-2 may include tools for optimally sharding the model for distributed processing across multiple processing units to increase bandwidth, reduce unified memory requirements, and the like. Tools for distillation 19-3 may be used to train lighter-weight models based on the knowledge encoded in the development model 16. For example, the development model 16 may be a high-performance, large, machine-learned model optimized using the model development platform 12. To obtain a lightweight model for operation in a resource-constrained environment, the smaller model may be a "student model" that learns to emulate the development model 16, which serves as a "teacher model." In this way, for example, the investment in learning the parameters and configuration of the development model 16 can be efficiently transferred to a smaller model for more efficient inference.

[0207] The workbench 15 may implement one, more, or none of the tool suites implemented in the model development platform 12. The workbench 15 may output an output model 20 based on the development model 16. The output model 20 may be a deployed version of the development model 16. The output model 20 may be a development or training checkpoint of the development model 16. The output model 20 may be a distilled, compressed, or otherwise optimized version of the development model 16.

[0208] Figure 12 is a block diagram of an example training process for training a machine-learned developed model 16. One or more portions of the example training process can be implemented by a computing system comprising one or more computing devices (such as, for example, the computing systems described with reference to other figures). Each respective portion of the example training process can be performed by any one (or any combination) of the one or more computing devices. In addition, one or more portions of the example training process can be implemented on hardware components of the devices described herein, for example, to train one or more systems or models. Figure 12The elements are depicted in a particular order for purposes of illustration and discussion. One of ordinary skill in the art, using the disclosure provided herein, will understand that the elements of any of the methods discussed herein may be adapted, rearranged, expanded, omitted, combined, or modified in various ways without departing from the scope of the present disclosure. Figure 12 The elements / terms described with reference to other systems and figures are described for illustrative purposes and are not intended to be limiting. One or more parts of the example training process may additionally or alternatively be performed by other systems.

[0209] Initially, the development model 16 may be maintained in an initial state as the initialization model 21. The development model 16 may be initialized with weight values. The initial weight values ​​may be random or based on an initialization pattern. The initial weight values ​​may be based on previous pre-training for the same or a different model.

[0210] Initialization model 21 may be pre-trained in pre-training phase 22. Pre-training phase 22 may be implemented using one or more pre-training pipelines 17-2 on data from dataset 17-1. For example, if initialization model 21 has already been pre-trained (e.g., development model 16 includes, is, or is based on a pre-trained base model or expert model), pre-training may be omitted.

[0211] Pretrained model 23 may then be a new version of development model 16, which may be maintained as development model 16 or a new development model. If development model 16 has already been pretrained, pretrained model 23 may be the initial state. Pretrained model 23 may undergo fine-tuning in a fine-tuning phase 24. Fine-tuning phase 24 may be implemented using one or more fine-tuning pipelines 17-3 on data from dataset 17-1. For example, if the pretrained model has satisfactory performance, if the model has already been fine-tuned, or if other tuning methods are preferred, fine-tuning may be omitted.

[0212] The fine-tuned model 25 may then be a new version of the development model 16, which may be maintained as the development model 16 or a new development model. If the development model 16 has already been fine-tuned, the fine-tuned model 25 may be the initial state. The fine-tuned model 25 may undergo refinement 26 with user feedback. For example, refinement 26 with user feedback may include reinforcement learning, optionally based on human feedback from human users of the fine-tuned model 25. Since reinforcement learning may be a form of fine-tuning, it should be understood that the fine-tuning phase 24 may include a phase for refinement 26 with user feedback. Refinement 26 with user feedback may produce a refined model 27. The refined model 27 may be output to a downstream system 28 for deployment or further development.

[0213] In some implementations, computational optimization operations may be applied before, during, or after each stage. For example, the initialized model 21 may undergo computational optimization 29-1 (e.g., using the computational optimization toolkit 19) prior to the pre-training stage 22. The pre-trained model 23 may undergo computational optimization 29-2 (e.g., using the computational optimization toolkit 19) prior to the fine-tuning stage 24. The fine-tuned model 25 may undergo computational optimization 29-3 (e.g., using the computational optimization toolkit 19) prior to refinement 26 utilizing user feedback. The refined model 27 may undergo computational optimization 29-4 (e.g., using the computational optimization toolkit 19) prior to output to the downstream system 28. The computational optimizations 29-1, ..., 29-4 may all be the same, all different, or include at least some different optimization techniques.

[0214] Example machine learning model inference system

[0215] Figure 13 1 is a block diagram of an inference system for operating one or more machine-learned models 1 for inference (e.g., for training, for deployment, etc.). A model host 31 can receive a machine-learned model 1. The model host 31 can host one or more model instances 31-1, which can be one or more instances of one or more models. The model host 31 can host the model instance 31-1 using available computing resources 31-2 associated with the model host 31.

[0216] Model master 31 can perform inference on behalf of one or more clients 32. Client 32 can transmit input request 33 to model master 31. Using input request 33, model master 31 can obtain input 2 to input into machine-learned model 1. Machine-learned model 1 can process input 2 to generate output 3. Using output 3, model master 31 can return output payload 34 in response to input request 33 from client 32. Output payload 34 can include or be based on output 3.

[0217] The model host 31 can utilize various other resources and tools to enhance the reasoning task. For example, the model host 31 can communicate with a tool interface 35 to facilitate the use of tools by the model instance 31-1. The tool interface 35 may include a local or remote API. The tool interface 35 may include integrated scripts or other software functions. The model host 31 can use an online learning interface 36 to facilitate continuous improvement of the machine-learned model 1. For example, the online learning interface 36 can be used in a reinforcement learning loop to retrieve user feedback on the reasoning served by the model host 31. The model host 31 can access a runtime data source 37 for enhancing the input 2 with additional contextual information. For example, the runtime data source 37 may include a knowledge graph 37-1 that facilitates structured information retrieval for information associated with the input request 33 (e.g., a search engine service). The runtime data source 37 may include a public or private, external or local database 37-2 that can store information associated with the input request 33 for enhancing the input 2. The runtime data source 37 may include account data 37 - 3 , which may be retrieved in association with a user account corresponding to the client 32 to customize the behavior of the model host 31 accordingly.

[0218] The model host 31 may be implemented by one or more computing devices or systems. The client 2 may be implemented by one or more computing devices or systems, which may include a computing device or system shared with the model host 31 .

[0219] For example, the model host 31 may be operated on a server system that provides machine learning services to client devices operating the client 32 (e.g., over a local area network or wide area network). The client device may be an end-user device used by an individual. The client device may be a server system that operates the client 32 to provide various functions as services to downstream end-user devices.

[0220] In some implementations, the model host 31 and the client 32 can operate on the same device or system. The model host 31 can be a machine learning service that runs on the device to provide machine learning functionality to one or more applications running on the client device, which can include an application that implements the client 32. The model host 31 and the client 32 can be part of the same application. For example, the model host 31 can be a subroutine or method implemented by one part of the application, and the client 32 can be another subroutine or method that uses the model host 31 to perform inference functionality within the application. It should be understood that the model host 31 and the client 32 can have a variety of different configurations.

[0221] Model instance 31-1 may include one or more machine-learned models that can be used for reasoning. Model instance 31-1 may include weights or other model components stored in persistent storage, temporarily cached, or loaded into high-speed memory. Model instance 31-1 may include multiple instances of the same model (e.g., for executing more requests in parallel on the same model). Model instance 31-1 may include instances of different models. Model instance 31-1 may include cached intermediate states of active or inactive models, which are used to accelerate reasoning of those models. For example, an inference session with a particular model may generate a significant amount of computational results that can be reused for future inference runs (e.g., using a KV cache for a transformer-based model). These computational results may be stored in association with the inference session so that the session can be executed more efficiently when resumed.

[0222] Computing resources 31-2 may include one or more processors (central processing units, graphics processing units, tensor processing units, machine learning accelerators, etc.) connected to one or more memory devices. Computing resources 31-2 may include a dynamic pool of available resources shared with other processes. Computing resources 31-2 may include a memory device that is large enough to fit the entire model instance in a single memory instance. Computing resources 31-2 may also shard the model instance across multiple memory devices (e.g., using data parallelism or tensor parallelism, etc.). Doing so can increase parallelism or execute large models using multiple memory devices that individually may not be able to fit the entire model in memory.

[0223] Input request 33 may include data for input 2. Model host 31 may process input request 33 to obtain input 2. Input 2 may be obtained directly from input request 33 or may be retrieved using input request 33. Input request 33 may be submitted to model host 31 via an API.

[0224] The model host 31 can perform reasoning on multiple batches of input requests 33 in parallel. For example, the model instance 31-1 can be configured with an input structure having a batch dimension. Individual inputs 2 can be distributed across the batch dimension (e.g., rows of an array). Individual inputs 2 can include completely different contexts. Individual inputs 2 can be multiple reasoning steps for the same task. Individual inputs 2 can be interleaved in the input structure so that any given reasoning cycle can operate on different parts of the corresponding inputs 2. In this way, for example, the model host 31 can perform reasoning on batches in parallel so that the output 3 can also include a batch dimension and return the reasoning results of the batched inputs 2 in parallel. In this way, for example, multiple batches of input requests 33 can be processed in parallel to achieve a higher throughput of the output payload 34.

[0225] Output payload 34 may include or be based on output 3 from machine-learned model 1. Model host 31 may process output 3 to obtain output payload 34. This may include chaining multiple rounds of inference (e.g., iteratively, recursively, across the same model or different models) to obtain the final output of the task to return in output payload 34. Output payload 34 may be transmitted to client 32 via an API.

[0226] The online learning interface 36 can facilitate reinforcement learning via the machine learning model 1. The online learning interface 36 can facilitate reinforcement learning with human feedback (RLHF). The online learning interface 36 can facilitate federated learning via the machine learning model 1.

[0227] The model host 31 can execute the machine-learned model 1 to perform inference for various tasks using various types of data. For example, various different inputs 2 and outputs 3 can be used for various different tasks. In some implementations, the input 2 can be or otherwise represent image data. The machine-learned model 1 can process the image data to generate an output. As an example, the machine-learned model 1 can process the image data to generate an image recognition output (e.g., recognition of the image data, potential embedding of the image data, an encoded representation of the image data, a hash of the image data, etc.). As another example, the machine-learned model 1 can process the image data to generate an image segmentation output. As another example, the machine-learned model 1 can process the image data to generate an image classification output. As another example, the machine-learned model 1 can process the image data to generate an image data modification output (e.g., a change to the image data, etc.). As another example, the machine-learned model 1 can process the image data to generate an encoded image data output (e.g., an encoded and / or compressed representation of the image data, etc.). As another example, the machine-learned model 1 can process the image data to generate an upgraded image data output. As another example, the machine-learned model 1 can process the image data to generate a prediction output.

[0228] In some implementations, the task is a computer vision task. In some cases, input 2 includes pixel data from one or more images, and the task is an image processing task. For example, the image processing task may be image classification, where the output is a set of scores, each score corresponding to a different object class and representing the likelihood that one or more images depict an object belonging to that object class. The image processing task may be object detection, where the image processing output identifies one or more regions in one or more images and, for each region, identifies the likelihood that the region depicts an object of interest. As another example, the image processing task may be image segmentation, where the image processing output defines, for each pixel in one or more images, the corresponding likelihood of each class in a predetermined set of classes. For example, the set of classes may be foreground and background. As another example, the set of classes may be object classes. As another example, the image processing task may be depth estimation, where the image processing output defines a corresponding depth value for each pixel in one or more images. As another example, the image processing task may be motion estimation, where the network input includes multiple images, and the image processing output defines, for each pixel of one of the input images, the motion of the scene depicted at that pixel between the images in the network input.

[0229] In some implementations, input 2 may be or otherwise represent natural language data. The machine learning model 1 may process the natural language data to generate an output. As an example, the machine learning model 1 may process the natural language data to generate a language encoding output. As another example, the machine learning model 1 may process the natural language data to generate a potential text embedding output. As another example, the machine learning model 1 may process the natural language data to generate a translation output. As another example, the machine learning model 1 may process the natural language data to generate a classification output. As another example, the machine learning model 1 may process the natural language data to generate a text segmentation output. As another example, the machine learning model 1 may process the natural language data to generate a semantic intent output. As another example, the machine learning model 1 may process the natural language data to generate an upgraded text or natural language output (e.g., text or natural language data of higher quality than the input text or natural language, etc.). As another example, the machine learning model 1 may process the natural language data to generate a prediction output (e.g., one or more predicted next parts of the natural language content).

[0230] In some implementations, input 2 may be or otherwise represent speech data (e.g., data describing spoken natural language, such as audio data, text data, etc.). The machine learning model 1 may process the speech data to generate an output. As an example, the machine learning model 1 may process the speech data to generate a speech recognition output. As another example, the machine learning model 1 may process the speech data to generate a speech translation output. As another example, the machine learning model 1 may process the speech data to generate a potential embedding output. As another example, the machine learning model 1 may process the speech data to generate an encoded speech output (e.g., an encoded and / or compressed representation of the speech data, etc.). As another example, the machine learning model 1 may process the speech data to generate an upgraded speech output (e.g., speech data of higher quality than the input speech data, etc.). As another example, the machine learning model 1 may process the speech data to generate a text representation output (e.g., a text representation of the input speech data, etc.). As another example, the machine learning model 1 may process the speech data to generate a prediction output.

[0231] In some implementations, input 2 may be or otherwise represent potentially coded data (e.g., a latent space representation of an input, etc.). The machine learning model 1 may process the potentially coded data to generate an output. As an example, the machine learning model 1 may process the potentially coded data to generate a recognition output. As another example, the machine learning model 1 may process the potentially coded data to generate a reconstruction output. As another example, the machine learning model 1 may process the potentially coded data to generate a search output. As another example, the machine learning model 1 may process the potentially coded data to generate a re-clustering output. As another example, the machine learning model 1 may process the potentially coded data to generate a prediction output.

[0232] In some implementations, input 2 may be or otherwise represent statistical data. Statistical data may be, represent, or otherwise include data calculated and / or computed from some other data source. The statistical data may be processed by the machine learning model 1 to generate an output. As an example, the statistical data may be processed by the machine learning model 1 to generate an identification output. As another example, the statistical data may be processed by the machine learning model 1 to generate a prediction output. As another example, the statistical data may be processed by the machine learning model 1 to generate a classification output. As another example, the statistical data may be processed by the machine learning model 1 to generate a segmentation output. As another example, the statistical data may be processed by the machine learning model 1 to generate a visualization output. As another example, the statistical data may be processed by the machine learning model 1 to generate a diagnostic output.

[0233] In some implementations, input 2 may be or otherwise represent sensor data. The machine learning model 1 may process the sensor data to generate an output. As an example, the machine learning model 1 may process the sensor data to generate an identification output. As another example, the machine learning model 1 may process the sensor data to generate a prediction output. As another example, the machine learning model 1 may process the sensor data to generate a classification output. As another example, the machine learning model 1 may process the sensor data to generate a segmentation output. As another example, the machine learning model 1 may process the sensor data to generate a visualization output. As another example, the machine learning model 1 may process the sensor data to generate a diagnostic output. As another example, the machine learning model 1 may process the sensor data to generate a detection output.

[0234] In some implementations, the machine learning model 1 can be configured to perform a task that includes encoding input data to achieve reliable and / or efficient transmission or storage (and / or corresponding decoding). For example, the task may be an audio compression task. The input may include audio data, and the output may include compressed audio data. In another example, the input includes visual data (e.g., one or more images or videos), the output includes compressed visual data, and the task is a visual data compression task. In another example, the task may include generating an embedding for the input data (e.g., input audio or visual data). In some cases, the input includes audio data representing spoken utterances, and the task is a speech recognition task. The output may include a text output mapped to the spoken utterances. In some cases, the task includes encrypting or decrypting the input data. In some cases, the task includes microprocessor performance tasks such as branch prediction or memory address translation.

[0235] In some implementations, the task is a generative task, and the machine learning model 1 can be configured to output content generated based on the input 2. For example, the input 2 can be or otherwise represent data of one or more modalities that encodes the context for generating additional content.

[0236] In some implementations, the task may be a text completion task. The machine learning model 1 may be configured to process an input 2 representing text data and generate an output 3 representing additional text data that completes a text sequence including the input 2. For example, the machine learning model 1 may be configured to generate an output 3 to complete a sentence, paragraph, or portion of text following a portion of the text represented by the input 2.

[0237] In some implementations, the task may be an instruction-following task. The machine-learned model 1 may be configured to process an input 2 representing an instruction for performing a function and generate an output 3 that furthers the goal of satisfying the instructed function (e.g., at least one step of a multi-step process for performing the function). The output 3 may represent data of the same or different modality as the input 2. For example, the input 2 may represent text data (e.g., natural language instructions for a task to be performed), and the machine-learned model 1 may process the input 2 to generate an output 3 that represents text data in response to the instruction (e.g., a natural language response, a programming language response, a machine language response, etc.). The input 2 may represent image data (e.g., image-based instructions for a task to be performed, optionally accompanied by text instructions), and the machine-learned model 1 may process the input 2 to generate an output 3 that represents text data in response to the instruction (e.g., a natural language response, a programming language response, a machine language response, etc.). One or more outputs 3 may be generated iteratively or recursively to sequentially process and complete steps toward completing the requested function. For example, the initial output may be executed by an external system or processed by the machine-learned model 1 to complete the initial steps of performing the function. Multiple steps may be performed in which a final output is obtained in response to the initial instructions.

[0238] In some implementations, the task may be a question-answering task. The machine learning model 1 may be configured to process an input 2 representing a question to be answered and generate an output 3 that advances the goal of returning an answer to the question (e.g., at least one step of a multi-step process for performing the function). The output 3 may represent data of the same or different modality as the input 2. For example, the input 2 may represent text data (e.g., natural language instructions for a task to be performed), and the machine learning model 1 may process the input 2 to generate an output 3 that represents text data in response to the question (e.g., a natural language response, a programming language response, a machine language response, etc.). The input 2 may represent image data (e.g., image-based instructions for a task to be performed, optionally accompanied by text instructions), and the machine learning model 1 may process the input 2 to generate an output 3 that represents text data in response to the question (e.g., a natural language response, a programming language response, a machine language response, etc.). One or more outputs 3 may be generated iteratively or recursively to sequentially process and complete steps toward answering the question. For example, the initial output may be executed by an external system or processed by the machine learning model 1 to complete the initial steps of obtaining an answer to the question (e.g., querying a database, performing calculations, executing scripts, etc.). Multiple steps may be performed in which a final output responsive to the question is obtained.

[0239] In some implementations, the task may be an image generation task. The machine learning model 1 may be configured to process an input 2 that represents a context about a desired portion of the image content. The context may include text data, image data, audio data, and the like. The machine learning model 1 may be configured to generate an output 3 that represents image data depicting an image associated with the context. For example, the machine learning model 1 may be configured to generate pixel data for an image. The value of a channel associated with a pixel in the pixel data may be selected based on the context (e.g., based on a probability determined based on the context).

[0240] In some implementations, the task may be an audio generation task. The machine learning model 1 may be configured to process an input 2 that represents a context about a desired portion of the audio content. The context may include text data, image data, audio data, and the like. The machine learning model 1 may be configured to generate an output 3 that represents audio data associated with the context. For example, the machine learning model 1 may be configured to generate waveform data in the form of an image (e.g., a spectrogram). The values ​​of the channels associated with the pixels of the image may be selected based on the context. The machine learning model 1 may be configured to generate waveform data in the form of a sequence of discrete samples of a continuous waveform. The values ​​of the sequence may be selected based on the context (e.g., based on a probability determined based on the context).

[0241] In some implementations, the task may be a data generation task. The machine learning model 1 may be configured to process an input 2 that represents a context about a desired portion of data (e.g., data from various data domains, such as sensor data, image data, multimodal data, statistics, etc.). For example, the desired data may be synthetic data used to train other machine learning models. The context may include any data type. The machine learning model 1 may be configured to generate an output 3 that represents data that is aligned with the desired data. For example, the machine learning model 1 may be configured to generate data values ​​for populating a data set. The value of a data object may be selected based on the context (e.g., based on a probability determined based on the context).

[0242] Example computing systems and devices

[0243] Figure 144 is a block diagram of an example networked computing system that can perform aspects of an example implementation of the present disclosure. The system can include multiple computing devices and systems communicatively coupled via a network 49. An example computing device 50 is described to provide an example of a computing device that can perform any aspect of the present disclosure (e.g., implement the model host 31, the client 32, or both). An example server computing system 60 is described as an example of a server computing system that can perform any aspect of the present disclosure (e.g., implement the model host 31, the client 32, or both). The computing device 50 and the server computing system 60 can interact collaboratively (e.g., via the network 49) to perform any aspect of the present disclosure (e.g., implement the model host 31, the client 32, or both). A model development platform system 70 is an example system that can host or provide a model development platform 12 for developing machine-learned models. A third-party system 80 is an example system that any of the computing device 50, the server computing system 60, or the model development platform system 70 can interact with when performing various aspects of the present disclosure (e.g., using third-party tools, accessing third-party databases or other resources, etc.).

[0244] The network 49 may be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and may include any number of wired or wireless links. In general, communications over the network 49 may be conducted via any type of wired or wireless connection using a wide variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), or protection schemes (e.g., VPN, secure HTTP, SSL). The network 49 may also be implemented via a system bus. For example, Figure 14 One or more devices or systems may be co-located with, housed by, or otherwise integrated into one or more other devices or systems.

[0245] Computing device 50 may be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop), a mobile computing device (e.g., a smartphone or tablet), a game console or controller, a wearable computing device, an embedded computing device, a server computing device, a virtual machine operating on a host device, or any other type of computing device. Computing device 50 may be a client computing device. Computing device 50 may be an end-user computing device. Computing device 50 may be a computing device that provides services to end users (end users may use another computing device to interact with computing device 50).

[0246] The computing device 50 may include one or more processors 51 and a memory 52. ​​The processor 51 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or a plurality of processors operatively connected. The memory 52 may include one or more non-transitory computer-readable storage media, such as HBM, RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The memory 52 may store data 53 and instructions 54 that can be executed by the processor 51 to cause the computing device 50 to operate. The operations may implement any one or more features described herein. The operations may implement the example methods and techniques described herein.

[0247] The computing device 50 may also include one or more input components for receiving user input. For example, the user input component may be a touch-sensitive component (e.g., a touch-sensitive display or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component may be used to implement a virtual keyboard. Other example user input components include a microphone, a camera, a lidar, a physical keyboard or other buttons, or other means by which a user can provide user input.

[0248] The computing device 50 may store or include one or more machine-learned models 55. The machine-learned models 55 may include one or more machine-learned models 1, such as a sequence processing model 4. The machine-learned models 55 may include one or more model instances 31-1. The machine-learned models 55 may be received from a server computing system 60, a model development platform system 70, a third-party system 80 (e.g., an application distribution platform), or developed locally on the computing device 50. The machine-learned models 55 may be loaded into the memory 52 and used or otherwise implemented by the processor 51. The computing device 50 may implement multiple parallel instances of the machine-learned models 55.

[0249] The server computing system 60 may include one or more processors 61 and memory 62. The processor 61 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or a plurality of processors operatively connected. The memory 62 may include one or more non-transitory computer-readable storage media, such as HBM, RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The memory 62 may store data 63 and instructions 64, which may be executed by the processor 61 to cause the server computing system 60 to operate. The operations may implement any one or more features described herein. The operations may implement the example methods and techniques described herein.

[0250] In some implementations, the server computing system 60 includes or is otherwise implemented by one or more server computing devices. In instances where the server computing system 60 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0251] The server computing system 60 may store or otherwise include one or more machine-learned models 65. The machine-learned models 65 may be the same as or different from the machine-learned models 55. The machine-learned models 65 may include one or more machine-learned models 1, such as the sequence processing model 4. The machine-learned models 65 may include one or more model instances 31-1. The machine-learned models 65 may be received from the computing device 50, the model development platform system 70, a third-party system 80, or developed locally on the server computing system 60. The machine-learned models 65 may be loaded into the memory 62 and used or otherwise implemented by the processor 61. The server computing system 60 may implement multiple parallel instances of the machine-learned models 65.

[0252] In an example configuration, the machine-learned model 65 may be included in or otherwise stored and implemented by the server computing system 60 to establish a client-server relationship with the computing device 50 for providing model inference. For example, the server computing system 60 may implement the model host 31 on behalf of the client 32 on the computing device 50. For example, the machine-learned model 65 may be implemented by the server computing system 60 as part of a web service (e.g., a remote machine-learned model hosting service, such as an online interface for performing machine-learned model operations on the server computing system 60 over a network). For example, the server computing system 60 may communicate with the computing device 50 via a local intranet or Internet connection. For example, the computing device 50 may be a workstation or endpoint in communication with the server computing system 60, wherein an implementation of the machine-learned model 65 is managed by the server computing system 60 to perform inference remotely (e.g., for runtime or training operations), with output being returned (e.g., projected, streamed, etc.) to the computing device 50. The machine-learned model 65 may work collaboratively or interoperably with the machine-learned model 55 on the computing device 50 to perform various tasks.

[0253] The model development platform system 70 may include one or more processors 71 and a memory 72. The processor 71 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or multiple processors operatively connected. The memory 72 may include one or more non-transitory computer-readable storage media, such as HBM, RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The memory 72 may store data 73 and instructions 74, which may be executed by the processor 71 to cause the model development platform system 70 to operate. The operations may implement any one or more features described herein. The operations may implement the example methods and techniques described herein. The example operations include the functionality described herein with respect to the model development platform 12. This functionality and other functionality may be implemented by a developer tool 75.

[0254] The third-party system 80 may include one or more processors 81 and a memory 82. The processor 81 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or multiple processors operatively connected. The memory 82 may include one or more non-transitory computer-readable storage media, such as HBM, RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The memory 82 may store data 83 and instructions 84, which may be executed by the processor 81 to enable the third-party system 80 to operate. The operations may implement any one or more features described herein. The operations may implement the example methods and techniques described herein. The example operations include the functionality described herein regarding tools and other external resources (e.g., third-party resources 85) called when training machine learning models 1, 4, 16, 20, 55, 65, etc. or performing reasoning using the machine learning models.

[0255] Figure 14An example arrangement of computing systems that can be used to implement the present disclosure is shown. Other computing system configurations may also be used. For example, in some implementations, one or both of computing system 50 or server computing system 60 can implement all or a portion of the operations of model development platform system 70. For example, computing system 50 or server computing system 60 can implement developer tools 75 (or extensions thereof) to develop, update / train, or refine machine-learned models 1, 4, 16, 20, 55, 65, etc. using one or more of the techniques described herein with respect to model alignment toolkit 17. In this way, for example, computing system 50 or server computing system 60 can develop, update / train, or refine machine-learned models based on a local dataset (e.g., for model personalization / customization, as permitted by user data preference selections).

[0256] Figure 15 is a block diagram of an example computing device 98 according to an example embodiment of the present disclosure. Computing device 98 can be a user computing device or a server computing device (e.g., computing device 50, server computing system 60, etc.). Computing device 98 can implement model host 31. For example, computing device 98 can include multiple applications (e.g., applications 1 to N). Each application can contain its own machine learning library and machine-learned model. For example, each application can include a machine-learned model. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc. Figure 15 As shown, each application can communicate with multiple other components of the computing device (such as, for example, one or more sensors, a context manager, a device state component, or additional components). In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to that application.

[0257] Figure 16 is a block diagram of an example computing device 99 according to an example embodiment of the present disclosure. Computing device 99 may be the same as or different from computing device 98. Computing device 99 may be a user computing device or a server computing device (e.g., computing device 50, server computing system 60, etc.). Computing device 98 may implement model host 31. For example, computing device 99 may include multiple applications (e.g., applications 1 through N). Each application may communicate with a central intelligence layer. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. In some implementations, each application may use an API (e.g., a public API across all applications) to communicate with the central intelligence layer (and the models stored therein).

[0258] The central intelligence layer can include multiple machine learning models. Figure 16 As shown, a corresponding machine-learned model can be provided for each application and managed by a central intelligence layer. In other implementations, two or more applications can share a single machine-learned model. For example, in some implementations, the central intelligence layer can provide a single model for all applications. In some implementations, the central intelligence layer is included in the operating system of the computing device 99 or is otherwise implemented by the operating system.

[0259] The central intelligence layer can communicate with the central device data layer. The central device data layer can be a centralized data repository for the computing device 99. Figure 16 As shown, the central device data layer can communicate with multiple other components of the computing device (such as, for example, one or more sensors, context managers, device state components, or additional components). In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0260] Additional Disclosure

[0261] The technology discussed herein relates to servers, databases, software applications, and other computer-based systems, as well as the actions taken and the information sent to and from such systems. The inherent flexibility of computer-based systems allows for a variety of possible configurations, combinations, and partitioning of tasks and functions between and among components. For example, the processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0262] Although the present invention has been described in detail with respect to various specific example embodiments of the present invention, each example is provided by way of explanation rather than limitation of the present invention. Those skilled in the art can easily produce changes, modifications, and equivalents to such embodiments after understanding the foregoing. Therefore, the present invention does not exclude such modifications, changes, or additions to the present invention that would be readily understood by those of ordinary skill in the art. For example, a feature shown or described as part of one embodiment can be used together with another embodiment to produce yet another embodiment. Therefore, it is intended that the present invention encompass such changes, modifications, and equivalents.

[0263] The aspects of the present disclosure have been described with respect to their illustrative embodiments. Any and all features in the appended claims can be combined or rearranged in any possible way, including combinations of claims that are not explicitly listed together in combination, because the example claim dependencies listed herein should not be understood to limit the scope of the possible combinations of features disclosed herein. Therefore, the scope of the present disclosure is illustrative rather than restrictive, and the present disclosure does not exclude such modifications, changes or additions to this theme that will be easily understood by those of ordinary skill in the art. In addition, this paper uses a list of example elements connected by conjunctions such as "and", "or", "but" to describe terms. It should be understood that such conjunctions are provided only for the purpose of explanation. For example, the clauses and other project sequences connected by specific conjunctions such as "or" can refer to "and / or", "at least one of", "any combination" of the example elements listed therein, etc. Terms such as "based on" should be understood to mean "at least partially based on".

[0264] The term "capable" should be understood to refer to the possibility of a feature in various implementations, rather than to specify a capability that must be present in every implementation. For example, the phrase "X is capable of Y" should be understood to indicate that in various implementations, X may be configured to do Y, rather than to indicate that X must always be capable of Y in every instance. It should be understood that in various implementations, X may not be able to do Y and still be within the scope of the present disclosure.

[0265] The term "may" should be understood to refer to the possibility of a feature in various implementations, rather than to specify a capability that must be present in every implementation. For example, the phrase "X may do Y" should be understood to indicate that in various implementations, X may be configured to do Y, rather than to indicate that X must always be able to do Y in every instance. It should be understood that in various implementations, X may not be able to do Y and still be within the scope of the present disclosure.

Claims

1. A computer-implemented method for use with an API tool for a machine-learned sequence processing model, comprising: Receiving, by a computing system comprising one or more computing devices, a first request to perform at least one task, the first request comprising a natural language description of the at least one task; generating, by the computing system, a query embedding comprising an embedded description of actions to be performed to perform the task; querying, by the computing system and using a query, a data structure comprising an API embedding, wherein the API embedding comprises an embedded description of functionality associated with a corresponding application programming interface (API), wherein the data structure includes a false positive API embedding comprising an embedded description of a function that is not available via a corresponding API, wherein the false positive API embedding is associated with instruction content indicating that the function is not available; Obtaining, by the computing system, the instruction content based on a similarity between the query and the false positive API embedding; Inputting the instruction content into the machine-learned sequence processing model by the computing system to generate an intermediate output; as well as One or more operations identified in the intermediate output are performed by the computing system.

2. The computer-implemented method of claim 1 , comprising: generating, by the computing system and using the machine-learned sequence processing model, a natural language description of the actions to be performed to perform the task; as well as The natural language description of the actions to be performed to perform the task is embedded by the computing system to obtain the embedded description of the actions to be performed to perform the task.

3. The computer-implemented method of claim 1 , further comprising: receiving, by the computing system via the one or more APIs, availability data indicating that a first API is currently unavailable; as well as The data structure is updated by the computing system based on the availability data. 4 . The computer-implemented method of claim 3 , wherein updating the data structure comprises replacing an API embedding associated with the first API with a false positive API embedding associated with the first API.

5. The computer-implemented method of claim 1 , wherein: The instruction content includes one or more instructions for generating a user message; The intermediate output includes a user message indicating that the first request cannot currently be satisfied; and The operations include providing the user message to a user associated with the first request.

6. The computer-implemented method of claim 5, wherein: The instruction content includes one or more instructions for suggesting alternative actions to be performed; and The user message includes a suggestion to perform the alternative action.

7. The computer-implemented method of claim 6, wherein: The instruction content includes at least one of the following: Data indicating the time when the functionality will become available; and one or more instructions for determining the time when the functionality will become available; and The alternative action includes scheduling execution of the function at the time when the function becomes available.

8. The computer-implemented method of claim 6, wherein: The instruction content includes data indicating an API for performing the function, wherein the API is associated with software that is not currently installed on the computing system; and The replacement action includes installing the software on the computing system.

9. The computer-implemented method of claim 8, further comprising: receiving, by the computing system, a user request to install the software on the computing system; installing, by the computing system, the software on the computing system in response to the user request; as well as The data structure is updated, by the computing system, to include an API embedding indicating that the API associated with the software is currently available.

10. The computer-implemented method of claim 6, wherein: The alternative action comprises an alternative functionality available via a corresponding API; and the computer-implemented method further comprises: receiving, by the computing system, a request to perform the alternative action; generating, by the computing system, instructions for causing the API to perform the substitute function based on the API embedding associated with the substitute function; and The instructions are provided by the computing system to the API.

11. The computer-implemented method of claim 1 , wherein the instruction content includes data indicating an API for which the computing device lacks authentication data, and the computer-implemented method further comprises: Requesting, by the computing system, authentication data for accessing the API; receiving, by the computing system, the authentication data for accessing the API; updating, by the computing system in response to receiving the authentication data, the data structure to include an API embedding indicating that the API is currently available; generating, by the machine-learned sequence processing model based at least in part on the authentication data, instructions for causing the API to perform the actions to be performed to perform the task; as well as The instructions are provided by the computing system to the API.

12. The computer-implemented method of claim 1 , wherein the data structure includes device-specific API information indicating APIs accessible to the one or more computing devices, and wherein the device-specific API information includes one or more of: data indicative of software installed on the one or more computing devices; and Data indicating an API for which the one or more computing devices have access authorization data.

13. The computer-implemented method of claim 1 , wherein: The computing system includes a mobile phone; The machine-learned sequence processing model runs locally on the mobile phone; and The data structure is stored locally on the mobile phone.

14. The computer-implemented method of claim 1 , further comprising: receiving, by the computing system, a second request including a natural language description of at least one second task; generating, by the computing system, a query embedding comprising an embedded description of a second action to be performed to perform the second task; querying, by the computing system, the data structure based on the query embedding; Obtaining, by the computing system, an API embedding, the API embedding comprising one or more instructions for causing the API to perform the second action; retrieving at least one data record including user-specific knowledge associated with the second action from a second data structure including user-specific knowledge associated with a user associated with the second request; inputting, by the computing system, the user-specific knowledge associated with the second action and the one or more instructions into the machine-learned sequence processing model to generate an intermediate output; as well as One or more operations identified in the intermediate output are performed by the computing system.

15. The computer-implemented method of claim 1 , wherein: The API embedding includes data indicating one or more functional categories associated with the functionality associated with a corresponding application programming interface (API); The false positive API embedding includes data indicating one or more false positive feature categories; and The instruction content is obtained based at least in part on the false positive functional category.

16. The computer-implemented method of claim 15, wherein obtaining the instruction content comprises: selecting, by the computing system, a false positive functional class using the machine-learned sequence processing model; as well as The instruction content is obtained by the computing system based on the selected false positive category and a similarity between the query and the false positive API embedding.

17. The computer-implemented method of claim 15, wherein obtaining the instruction content comprises: selecting, by the computing system, the false positive class based on a similarity between the query and the false positive class; as well as The instruction content is obtained by the computing system based on the false positive category.

18. A computer-implemented method as described in claim 1, wherein the first request is provided in a format that includes audio data, and generating the query embedding includes providing at least a portion of the audio data to the machine-learned sequence processing model, wherein the machine-learned sequence processing model is a multimodal machine-learned sequence processing model.

19. A computing system comprising one or more processors and one or more non-transitory computer-readable media storing instructions executable by the one or more processors to cause the computing system to perform operations comprising: receiving a first request to perform at least one task, the first request comprising a natural language description of the at least one task; generating a query embedding comprising an embedded description of actions to be performed to perform the task; querying a data structure including an API embedding using a query, wherein the API embedding includes an embedded description of functionality associated with a corresponding application programming interface (API), wherein the data structure includes a false positive API embedding comprising an embedded description of a function that is not available via a corresponding API, wherein the false positive API embedding is associated with instruction content indicating that the function is not available; obtaining the instruction content based on a similarity between the query and the false positive API embedding; inputting the instruction content into the machine-learned sequence processing model to generate an intermediate output; as well as One or more operations identified in the intermediate output are performed.

20. One or more non-transitory computer-readable media storing instructions executable by a computing system to perform operations comprising: receiving a first request to perform at least one task, the first request comprising a natural language description of the at least one task; generating a query embedding comprising an embedded description of actions to be performed to perform the task; querying a data structure including an API embedding using a query, wherein the API embedding includes an embedded description of functionality associated with a corresponding application programming interface (API), wherein the data structure includes a false positive API embedding comprising an embedded description of a function that is not available via a corresponding API, wherein the false positive API embedding is associated with instruction content indicating that the function is not available; obtaining the instruction content based on a similarity between the query and the false positive API embedding; inputting the instruction content into the machine-learned sequence processing model to generate an intermediate output; as well as One or more operations identified in the intermediate output are performed.

Images