Efficient use of tools with language models

Abstract

We propose a computer system that enables machine learning-based language models to select the precise structural tools they should utilize when solving challenging tasks. In detail, current language models rely heavily on internal knowledge to solve all downstream tasks, leading to hallucination and unfounded answers. In contrast, we propose a novel tool-using framework for language models that allows them to intelligently route queries to the most relevant and predefined skills.

Description

[Technical Field]

[0001] Related applications This application claims priority and benefit to U.S. Provisional Patent Application No. 63 / 456,300, filed on 31 March 2023, which is incorporated herein by reference in its entirety.

[0002] This disclosure generally relates to the use of machine learning for language modeling. More specifically, this disclosure relates to a computing system that generates prompts that enable a machine-trained language model to efficiently use structural tools to generate responses to queries. [Background technology]

[0003] Natural language processing (NLP) has seen rapid progress in recent years, primarily due to improvements in learning-based algorithms and other forms of machine or "neural" learning. One specific task within the field of NLP is contextual text generation. In contextual text generation tasks, an agent (e.g., a machine learning model) is tasked with generating output text from a given context. In some exemplary approaches to contextual text generation tasks, a text-to-text model reads input contextual text and then directly generates output text. An example of a contextual text generation task is a question-answering task, where the input context contains a question and the desired output text is the answer to that question.

[0004] The current state-of-the-art models for contextual text generation tend to be converter-based neural models, some of which are commonly referred to as “Large-Scale Language Models” (or LLMs). However, these so-called LLMs have several drawbacks. Specifically, while a pre-trained LLM model may exhibit significant intelligence, its knowledge is constrained by the information contained in (and learned from) its training dataset and / or the information introduced within the contextual text input. Thus, its factual knowledge is significantly limited and generally does not change over time. Therefore, when required to generate output containing factual information, the model will typically generate inaccurate facts, such as hallucination, or provide outdated information. Relying on incorrect factual information can lead to inefficiencies, such as incorrect actions (e.g., computerized actions) being taken that need to be corrected or otherwise rectified, and can result in redundant and unnecessary use of resources (e.g., computing resources).

[0005] The inability of current language models to access external knowledge is just one example of the larger challenges associated with such models. Specifically, existing language models have little ability to select, access, or utilize various structural tools such as computational tools (e.g., computers), information retrieval tools (e.g., stored data, search engines, knowledge graphs), and / or programming tools (e.g., compilers, programming language interpreters). Access to these tools would significantly improve the ability of language models to handle more diverse and complex queries. [Overview of the Initiative]

[0006] Aspects and advantages of the embodiments of this disclosure are partially described in the following description, can be learned from the description, or can be learned through the practice of the embodiments.

[0007] One exemplary aspect of the present disclosure relates to a method implemented on a computer for performing contextual text generation. The method includes obtaining data describing a query by a computing system comprising one or more computing devices. The method includes generating query embeddings for the query by the computing system, the query embeddings being represented in a latent embedding space. The method includes performing a similarity search on the query embeddings in the latent embedding space by the computing system to identify one or more previously defined embeddings associated with one or more previously defined query-response pairs. The method includes generating a prompt by the computing system based on the query and one or more previously defined query-response pairs. The method includes providing the prompt as input for processing by a machine learning language model by the computing system. The method includes receiving a model response to the query output by the machine learning language model based on processing of the prompt by the computing system.

[0008] Other aspects of this disclosure cover a variety of systems, apparatus, non-temporary computer-readable media, user interfaces, and electronic devices.

[0009] These and other features, aspects and advantages of the various embodiments of this disclosure will be better understood by referring to the following description and the appended claims. The appended drawings incorporated herein and forming part of this specification illustrate exemplary embodiments of this disclosure and, together with the description, serve to illustrate the relevant principles.

[0010] A detailed description of embodiments intended for those skilled in the art is provided herein with reference to the accompanying drawings. [Brief explanation of the drawing]

[0011] [Figure 1]The following is a block diagram of an exemplary data flow for generating a system response to a query according to an exemplary embodiment of the present disclosure. [Figure 2] A graphic diagram of an exemplary process for generating a system response to a query, according to an exemplary embodiment of the present disclosure, is shown. [Figure 3] This flowchart illustrates an exemplary method for training a machine learning model according to exemplary embodiments of the present disclosure. [Figure 4] This is a block diagram of an exemplary processing flow for processing inputs(multiple) and generating outputs(multiple) using machine learning models(multiple), according to exemplary embodiments of the aspects of this disclosure. [Figure 5] This is a block diagram of an exemplary sequence processing model according to an exemplary embodiment of the aspects of the present disclosure. [Figure 6] This is a block diagram of exemplary technology for feeding an exemplary input sequence for processing by a sequence processing model, according to exemplary embodiments of the aspects of the present disclosure. [Figure 7] This is a block diagram of an exemplary model development platform according to an exemplary embodiment of the aspects of the present disclosure. [Figure 8] This is a block diagram of an exemplary training workflow for training a machine learning model according to an exemplary embodiment of the aspects of this disclosure. [Figure 9] This is a block diagram of an inference system for running one or more machine learning models to perform inference, according to an exemplary embodiment of an aspect of the present disclosure. [Figure 10] This is a block diagram of an exemplary network computing system according to an exemplary embodiment of the aspects of the present disclosure. [Figure 11] This is a block diagram of an exemplary computing device according to an exemplary embodiment of the aspects of the present disclosure. [Figure 12] This is a block diagram of an exemplary computing device according to an exemplary embodiment of the aspects of the present disclosure.

Best Mode for Carrying Out the Invention

[0012] Reference numerals repeated throughout the several drawings are intended to identify like features in various embodiments.

[0013] Overview Exemplary aspects of the present disclosure are directed to a computer system that enables a machine-learned language model to select the precise structural tools to utilize when solving difficult tasks. Specifically, current language models rely heavily on internal knowledge to solve all downstream tasks, which causes hallucinations and unfounded answers. In contrast, the present disclosure proposes a new tool usage framework for language models, whereby the language model can smartly route queries to the most relevant pre-defined skills.

[0014] As an example, a computing system can generate an embedded cache associated with a set of previously defined query-response pairs. For example, some or all of the previously defined query-response pairs can demonstrate the use of tools in a service that generates answers to queries. Next, when a new query is received, the computing system can generate a query embedding of the query and perform a similarity search in the embedding space to identify and retrieve some (e.g., top k) of the most similar previously defined query-response pairs.

[0015] These obtained previously defined query-response pairs can be combined with new queries to generate prompts for a machine-learned language model. The language model can process the prompt to generate a model response. For example, the model response can include one or more tokens that invoke the use of one or more structured tools such as computing tools (e.g., computers), information acquisition tools (e.g., stored data, search engines, knowledge graphs), and / or programming tools (e.g., compilers, programming language interpreters, etc.). The computing system can process the model response (e.g., by executing the structured tools invoked by the model response) to generate a system response that answers the query.

[0016] More specifically, the ability to reliably and efficiently solve difficult problems is greatly improved by accessing and correctly using different structured tools. Therefore, enabling the language model to use tools is attractive in that: 1) the execution by the tool is more deterministic than purely probabilistic, so the authenticity of the completion of the language model is usually more reliable; 2) domain-specific knowledge is delegated to external tools, so the size of the language model can potentially be reduced; and 3) the language model using tools is, by definition, less dependent on the specific input provided, so it is less affected by out-of-distribution performance degradation.

[0017] Therefore, the present disclosure proposes a new general framework that enables a language model (e.g., an existing pre-trained language model) to efficiently use external tools. In some embodiments, the proposed framework may first include creating a set of pre-defined query-response pairs. In some embodiments, these query-response pairs may be referred to as "prototypes" and can correspond to, or be derived from, questions and answers accumulated during validation error analysis.

[0018] Next, in order for the language model to perform routing functions for tool usage, a computing system implementing the proposed framework can use an embedding model (e.g., a pre-trained text embedding model) to embed each query into a set of prototypes. Then, when a new query is received, it can similarly embed the query using the embedding model.

[0019] The computing system can generate prompts used to trigger the completion of the language model by concatenating the newly received query with one or more of the top k most similar re-ranked prototypes. The language model can process the prompts and generate a model response. After the language model is complete, the computing system can execute the generated model response, with the support of a “toolbox” containing the necessary tools (which may include, for example, data, libraries, and functions) to successfully execute the model response, where appropriate.

[0020] Exemplary experiments demonstrate that the proposed framework significantly improves the language model's ability to correctly respond to more diverse and complex query sequences. More specifically, the exemplary experiments show that the proposed framework outperforms current state-of-the-art methods for several different tasks requiring complex processing.

[0021] The system and method of this disclosure offer several technical effects and advantages. As an example of these technical effects and advantages, the proposed approach enables the language model to leverage structural tools to access additional information, such as additional factual information. Thus, the language model can invoke and use such structural tools to access additional information that may be current, factual, domain-specific, client-specific, or user-specific. This enhances the knowledge available to the language model when creating text output and further improves the system's flexibility by enabling the introduction of various information sources for different use cases.

[0022] The proposed structural tools, in addition to improving the quality of the model's output, also lead to savings in computational resources such as processor usage, memory usage, and network bandwidth. Specifically, the knowledge available to previous language models was limited to the information contained in (and learned from) their training datasets and / or introduced within the contextual text input. Therefore, the knowledge of factual information was significantly limited and generally did not change over time. Thus, when required to generate output containing factual information, the model would typically produce inaccurate facts, like hallucination, or provide outdated information. Therefore, in order to keep language models up-to-date with changing real-world facts, to port the language model to a new domain or set of user information, or to deploy the model to a new situation where new information was at issue in other ways, the entire language model needs to be retrained. Retraining a language model requires the use of computational resources such as processor usage, memory usage, and network bandwidth.

[0023] However, the use of the structuring tools proposed in this disclosure eliminates the need to keep language models up-to-date with changing real-world facts, to port language models to new domains or sets of user information, or to retrain language models to deploy them to new situations where new information is at issue. Instead, the model can simply be given access (e.g., via the structuring tools) to additional information that may be up-to-date, factual, domain-specific, client-specific, or user-specific. Thus, without the need to retrain the model, it can be easily ported to different domains, applications, users, etc., and / or provide responses that leverage up-to-date factual information, thereby saving computational resources significantly. By encoding the context in the form of intermediate analysis that can interface with (potentially external) information sources, the process can contribute to solving technical constraints in the delivery of information and / or functionality.

[0024] Similarly, other exemplary technical benefits stem from the ability of models to leverage external sources to acquire information rather than having to memorize all the information necessary to respond to various inputs (e.g., in the form of learned relationships). In detail, past approaches required storing and using large models (e.g., on user devices with constrained memory and / or battery availability) that were large enough in size (e.g., number of parameters) to learn and store relationships between various inputs and outputs. In contrast, some exemplary embodiments of this disclosure can enable “thin” (smaller) models to persist on user devices or other mobile clients or browsers. Thin models can leverage various structural tools (e.g., cloud services) to save battery, computation, storage, updates, etc. Thus, smaller models with access to structural tools can achieve performance similar to or better than large embedded models, thereby saving computing resources such as memory usage, network bandwidth, and energy consumption.

[0025] As another example, the proposed technique allows machine learning models to better select which tools to use from several available tools. More specifically, by providing useful examples demonstrating the correct tool usage, the model can automatically select the appropriate tool to use from several different available tools. If the model is highly reliable, it may even invoke a single tool instead of several different tools. This reduces the number of interactions between the model and various tools, thus saving computational resources.

[0026] As another example, the proposed model demonstrates improved interpretability. For instance, the model response generated by the model can be examined or inspected (e.g., before or after execution) to interpret or understand how the final system response was generated in response to the contextual input. Improved interpretability allows for more efficient use of computational resources, such as processor usage and memory usage. For example, a lack of interpretability in the output of a language model can lead to a lack of confidence in or reliance on the model output, potentially resulting in unnecessary overhead or other work (e.g., computerized behavior) to "double-check" the credibility or usefulness of the model output. Improving interpretability can improve the reliability of a computerized system. More specifically, the reliability of the system response may be verified and / or evaluated to establish the system's usefulness for a particular task.

[0027] Exemplary embodiments of this disclosure will be described in further detail here with reference to the drawings.

[0028] Examples of model and tool usage Aspects of this disclosure relate to a process in which in-context demonstration samples are used to prompt a language model to mimic a workflow presented in a few-shot sample, thereby enabling the language model to solve a task it has never seen before. In detail, several exemplary embodiments include three main stages: (a) a stage of predefining and embedding a prototype for solving a task; (b) a stage of searching the k most relevant questions in the prototype during inference for each new query question and re-ranking them to inject them into prompts; and (c) a stage of executing generated content with the support of a pre-loaded toolbox to obtain the final result.

[0029] An example of accumulating prototypes from errors One aspect of this disclosure relates to the accumulation and use of demonstrative query-response pairs, which may be referred to as “prototypes.” In some embodiments, for each of several evaluation tasks, the corresponding dataset may be split into a test set and a validation set in a specific ratio (e.g., 4:1) to discover prototypes that may lead to the best performance. Additional prototypes can then be accumulated by maintaining a “prototype pool” containing prototypes that answered incorrectly in all validation runs.

[0030] One exemplary procedure for accumulating prototypes from errors can be described as Algorithm 1.

[0031] Algorithm 1: Accumulating prototypes from errors Input:valid data={x i ,y i} i=1···,N ,valid_acc=0,Evaluate LM,review_manual={x j , y j} j=1···, m Initialize flag=false. repeat For i = 1 to N do Evaluate the LM with respect to valid_data using review_manual LM(x i ) ≠ y i If so Write the solution for x (i.e., y i ) i . Add the new {x i , y i} to review_manual End if End for Calculate valid_acc with respect to valid_data If valid_acc does not improve over k times flag = true End if Until flag is true

[0032] In some embodiments, the prototype pool can be initialized with some randomly selected questions and answers. A stopping criterion can be provided for the accumulation. This is whether the accuracy for the validation set does not increase after k validations using the updated prototype pool. In one example, k = 3 is this "patience threshold".

[0033] Example of a routed tool usage prompt When some prototypes are provided from the prototype pool (e.g., as conditional inputs included in the prompt), a pre-trained language model (LM) can solve some of the questions by mimicking the workflow presented to the prototypes. In an exemplary experiment, it has been shown that the success of the model can be significantly increased by actively selecting and inputting a prototype having a workflow similar to the workflow required to answer the query question. Thus, one aspect of the present disclosure is directed to a routing system for smartly selecting an appropriate prototype for a given query question.

[0034] In detail, before starting the evaluation of a particular task, some exemplary systems can first pre-embed all questions in stored prototypes using an embedding system and save those embeddings in a cache file. Then, during the evaluation, all new query questions are first embedded using the same embedding system. Next, the computing system can use the embeddings of the prototype questions to calculate a distance metric (e.g., pairwise cosine similarity). This distance metric (e.g., cosine similarity) can be called a relevance score. Finally, the computing system includes several (e.g., the top k) of the most similar prototypes in a prompt used to trigger LM generation.

[0035] Example execution for verification Upon receiving a prompt prepared by the routing module, the LM can complete the generation of the query question by considering the workflow presented in the prototype included in the prompt.

[0036] In some exemplary embodiments, tool usage can be easily implemented by executing code, so most of the answers provided in the prototype are code. For this purpose, some exemplary embodiments may include the configuration and use of a “toolbox” for each task. For example, the toolbox may correspond to a portion of global memory into which data, code snippets, and / or random parameters can be loaded to assist in the execution of LM-generated code.

[0037] As an example, the following items can be preloaded into the toolbox: a dataset file (e.g., a .csv file), a solver class that implements some desired functionality (e.g., the classic Dijkstra's algorithm), and any packages necessary for successful execution (e.g., pandas, numpy, etc.). These preloaded codes can be run once before starting the task evaluation. For each question, the generated code (e.g., the model response generated by LM, called LM_gen) can be executed by the computing system using exec(LM_gen,globals()) to obtain validation results.

[0038] In some embodiments, the computing system uses eval('answer') to extract the value stored in the answer variable to obtain the final answer (for example, the LM is expected to simulate the answer in a prototype in the final step, which also stores the answer in the variable answer).

[0039] Example of a query response system Figure 1 shows an exemplary query response system 110 that implements the above concept. More specifically, Figure 1 shows an exemplary data flow block diagram for generating a system response 136 to a query 112 according to an exemplary embodiment of the present disclosure.

[0040] As shown in Figure 1, the computing system can obtain data describing query 112. For example, query 112 can be received from a user. For example, the query may be a question in natural language.

[0041] The computing system can use the embedding model 114 to generate a query embedding 116 for query 112. For example, the query embedding 116 can be represented in a latent embedding space.

[0042] The computing system can perform a similarity search 120 on query embeddings 116 in the potential embedding space to identify one or more previously defined embeddings 124 associated with one or more previously defined query-response pairs 118. For example, performing a similarity search 120 may include ranking one or more previously defined query-response pairs 118. For example, the previously defined query-response pairs 118 may be ranked based on a distance measure (e.g., pairwise cosine similarity) evaluated between the embedding 122 and the query embedding 116 for each pair 118. In some embodiments, performing a similarity search 120 may include identifying the top k set of previously defined embeddings 122 from the embedding space, where k is a hyperparameter. In other embodiments, all embeddings 122 with a distance measure smaller than some threshold may be returned.

[0043] More specifically, a computing system can use an embedding model 114 to generate an embedding 122 for a previously defined query-response pair 118. For example, this process can be performed offline before the query 112 is received, and the embedding 122 can be stored in a database for use when the query 112 is received. For example, in some embodiments, the embedding 122 can be generated from only the query portion of a previously defined query-response pair 118. According to one aspect of this disclosure, some or all of the previously defined query-response pairs 118 can demonstrate the use of one or more structural tools 138.

[0044] In some embodiments, a specific subset of previously defined query-response pairs 18 may be associated with different structure tools 138. For example, a first pair may be an example of using a first tool, and a second pair may be an example of using a second tool. The user associated with query 112 may be permitted to access, or otherwise use, some, but not all, of the structure tools 138. In some embodiments, the similarity search 120 may be limited to searching only for previously defined embeddings 124 generated from query-response pairs 118 associated with structure tools 138 that the user is permitted to access or use. Additionally or alternatively, the user may be allowed to select one or more of the structure tools 138 (e.g., via a user interface). In some embodiments, the similarity search 120 may be limited to searching only for previously defined embeddings 124 generated from query-response pairs 118 associated with one or more of the structure tools 138 selected by the user.

[0045] The computing system can generate a prompt 128 based on the query 112 and one or more previously defined query-response pairs 124 identified by the similarity search 120 (for example, by performing prompt construction 126). As an example, the computing system can concatenate the query 112 with one or more previously defined query-response pairs 124 identified by the similarity search 120.

[0046] The computing system can provide a prompt 128 as input for processing by the machine learning language model 130. For example, the language model 130 may be a pre-trained large-scale language model such as a BERT model, a LaMDA model, or a PaLM model. Based on the processing of the prompt 128, the computing system can receive a model response 132 for the query 112 output by the machine learning language model 130.

[0047] The computing system can perform a response execution 134 to process the model response 132 and generate a system response 136. The system response 136 can be provided as output to the user. For example, the system response 136 may be a natural language answer to a natural language question.

[0048] According to aspects of this disclosure, in some embodiments, the model response 132 may include one or more tokens, which, when executed by a computing system (for example, as shown as response execution 134), cause a structuring tool 138 to retrieve or transform information. Thus, in some embodiments, processing the model response 132 by performing response execution 134 may include executing one or more tokens to cause the structuring tool 138 to retrieve or transform information. The system response 136 may be at least partially based on the information retrieved or transformed by the structuring tool 138 in response to the execution of one or more tokens.

[0049] In some embodiments, the toolbox 140 may be provided for use by a structural tool(s) 138. For example, the toolbox 140 may include any knowledge, data, or information that enables the structural tool to retrieve or transform information requested or indicated by a model response 132.

[0050] To provide an example for illustrative purposes, in some embodiments, the model response 132 may include instructions expressed in a computer language (e.g., an executable computer program), and the structure tool 138 may be a programming language interpreter configured to execute the instructions expressed in the computer language. In some such embodiments, the toolbox 140 may include preloaded information such as one or more libraries or datasets associated with the computer language. One or more libraries or datasets may be provided in a shared computer environment with the programming language interpreter. As an example, the computer language may be the Python computer language.

[0051] Other examples include, or may include or utilize, a database lookup for accessing additional information from a database, or an API call for requesting and receiving additional information via an application programming interface (API), or a query service for querying results from a search engine, knowledge graph, or digital assistant, or a calculator tool or other computing tool.

[0052] In some embodiments, the machine learning language model 130 can be optionally fine-tuned against previously defined query-response pairs 118 before using the model 130, as shown in Figure 1.

[0053] Example Explanation Figure 2 shows a graphic diagram of a specific exemplary process for generating a system response to a query according to an exemplary embodiment of the present disclosure. As shown in part (a), the computing system can first accumulate a set of question-answer pairs from validation errors (referred to as the “prototype pool”). As shown in part (b), the computing system can embed questions into the prototype pool, and each time a query is made to the LM with a new question, the computing system searches the review manual for the k most similar questions, re-ranks the corresponding prototypes, and concatenates them as prompt injections. As shown in part (c), prompt-triggered LM generation can be executed in an environment (e.g., by a Python interpreter) that preloads data, libraries, code snippets, etc. that can support a particular task. Finally, the answers can be collected from variables at runtime.

[0054] More specifically, Figure 2 shows a graphic diagram of an exemplary process for generating a system response to a query, according to an exemplary embodiment of the present disclosure. The process shown in Figure 2 illustrates the interaction between a predefined query-response pair known as a prototype and a new user query. This leads to the generation of a system response by using a machine learning-prepared language model and a toolbox of resources.

[0055] In the first stage (a), the process includes selecting relevant prototypes (e.g., Prototype 1, Prototype 4, and Prototype 8) based on a relevance score indicating their similarity to the new user query. Each prototype includes a question and an answer, the answer may be in the form of non-executable natural language and / or an executable program. For example, Prototype 4, with a relevance score of 0.6, provides an answer in executable code to determine the number of flights operated by Delta, while Prototype 1, with a higher relevance score of 0.9, demonstrates the use of the Dijkstra solver class to find the cheapest flight under specific constraints.

[0056] The second stage (b) involves combining a new user query with a re-ranked prototype to form a prompt to be injected into the language model. The exemplary user query represents a desire to find the lowest price for a flight from SFO to JFK, including a preference for Delta flights and up to two layovers. The re-ranked prototype is selected to help the language model understand the context and requirements of the query by providing examples of similar problems and their solutions.

[0057] In the final stage (c), the language model processes a combined prompt of the new query and the selected prototype to construct a model response. The response leverages both constructive and planning skills demonstrated in the prototype, such as filtering a dataset of Delta flights and using a planning algorithm to find the cheapest flight options within given constraints. The model response, including executable code, is then run using a toolbox for flight booking. The toolbox includes basic resources such as the flight dataset, internal code snippets like the Dijkstra solver class, and external libraries (e.g., pandas). The execution results in a system response of the lowest flight price, confirming the success of applying the methods and tools to resolve the user's query.

[0058] This interactive execution process serves as an exemplary embodiment of the proposed technology, demonstrating that a computing system can efficiently use structural tools to leverage machine learning-prepared language models and predefined query-response prototypes to generate accurate and up-to-date responses to complex queries.

[0059] Example of a method Figure 3 shows a flowchart of a method 300 for training one or more machine learning models according to an aspect of the present disclosure. For example, an exemplary machine learning model may include {references to the claimed model(s)}.

[0060] One or more parts of Exemplary Method 300 may be implemented by a computing system including one or more computing devices, such as the computing system described with reference to other figures. Each part of Exemplary Method 300 may be implemented by any (or any combination) of one or more computing devices. Furthermore, one or more parts of Exemplary Method 300 may be implemented by hardware components of the device(s) described herein, for example, to train one or more systems or models. Figure 3 shows the elements to be implemented in a particular order for illustrative and explanatory purposes. Those skilled in the art will understand, by using the disclosure provided herein, that any element of the methods described herein may be adapted, rearranged, extended, omitted, combined, or modified in various ways without departing from the scope of this disclosure. Figure 3 is illustrated for illustrative purposes only, with reference to elements / terms described in relation to other systems and drawings, and is not intended to limit it. One or more parts of Exemplary Method 300 may be implemented by other systems, additionally or alternatively.

[0061] In 302, exemplary method 300 may include obtaining training instances. A training dataset may include multiple training instances, which are divided among multiple datasets (e.g., a training dataset, a validation dataset, or a test dataset). Training instances may be labeled or unlabeled. Although these are referred to as “training” instances in exemplary method 300, it should be understood that runtime inference can form training instances if the model is trained using evaluation of the model’s performance against its runtime instances (e.g., online training / learning). Exemplary data types for training instances and the various tasks associated with them are described throughout this disclosure.

[0062] In 304, exemplary method 300 may include processing a training instance using one or more machine-trained models to produce an output. The output may be obtained directly from one or more machine-trained models, or it may be the result of a downstream chain of processing operations that includes the output of one or more machine-trained models.

[0063] In 306, exemplary method 300 may include receiving an evaluation signal associated with an output. The evaluation signal can be obtained using a loss function. Various methods for determining the loss can be used, such as mean squared error, likelihood loss, cross-entropy loss, hinge loss, contrastive loss, or various other loss functions. The evaluation signal can be computed using known ground truth labels (e.g., supervised learning), predicted or estimated labels (e.g., semi-supervised or self-supervised learning), or unlabeled (e.g., unsupervised learning). The evaluation signal may be a reward (e.g., for reinforcement learning). The reward can be computed using a machine-learned reward model configured to generate a reward based on the received output(s). The reward can be computed using feedback data describing human feedback on the output(s).

[0064] In 308, exemplary method 300 may include updating a trained model using an evaluation signal. For example, the parameter values ​​of a trained model(s) may be learned in some embodiments using various training or learning techniques, such as backpropagation. For example, an evaluation signal may be backpropagated from the output (or other source of the evaluation signal) through the trained model(s) to update one or more parameters of the model(s) (for example, based on the gradient of the evaluation signal's parameter(s)). For example, a system(s) comprising one or more trained models may be trained in an end-to-end manner. The parameters may be iteratively updated over several training iterations using gradient descent techniques. In some embodiments, performing error backpropagation may include performing time-truncate backpropagation. Exemplary method 300 may include performing several generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization ability of the trained model.

[0065] In some embodiments, exemplary method 300 can be performed to train a machine learning model from an initialized state to a fully trained state (for example, when the model exhibits a desired performance profile based on accuracy, precision, recall, etc.).

[0066] In some embodiments, exemplary method 300 can be performed for a specific stage of a training procedure. For example, in some embodiments, exemplary method 300 can be performed to pre-train a machine learning model. Pre-training may include, for example, large-scale training on potentially noisy data to achieve a broad base of performance levels across various task / data types. In some embodiments, exemplary method 300 can be performed to fine-tune a machine learning model. Fine-tuning may include, for example, smaller-scale training on higher-quality data (e.g., labeled, curated, etc.). Fine-tuning may affect all or some of the parameters of the machine learning model. For example, different parts of the machine learning model may be “frozen” during a particular training stage. For example, parameters associated with the embedding space may be “frozen” during fine-tuning (e.g., to retain information learned from a broader domain than present in the fine-tuning dataset). Exemplary fine-tuning approaches include reinforcement learning. Reinforcement learning may be based on user feedback on the model's performance in use.

[0067] Examples of machine learning models Figure 4 is a block diagram of an exemplary processing flow for processing input 2(multiple) using machine learning model 1(multiple) to generate output 3(multiple).

[0068] The machine-trained model 1(v) may be one or more machine-trained models or model components, or may include them. An exemplary machine-trained model may include a neural network (e.g., a deep neural network). An exemplary machine-trained model may include a nonlinear model or a linear model. An exemplary machine-trained model may use other architectures instead of, or in addition to, a neural network. An exemplary machine-trained model may include decision tree-based models, support vector machines, hidden Markov models, Bayesian networks, linear regression models, k-means clustering models, etc.

[0069] Exemplary neural networks may include feedforward neural networks, recurrent neural networks (RNNs) including long-short-term memory (LSTM) based recurrent neural networks, convolutional neural networks (CNNs), distributed models, generative adversarial networks, or other forms of neural networks. Exemplary neural networks may also be deep neural networks. Some exemplary machine-learned models may utilize attention mechanisms such as self-attention. For example, some exemplary machine-learned models may include multi-head self-attention models.

[0070] A machine learning model 1(or more) may contain one or more instances of the same model configured to work on data from input 2(or more). A machine learning model 1(or more) may contain an ensemble of different models that can collaboratively interact to process data from input 2(or more). For example, a machine learning model 1(or more) may employ a mixture-of-experts (MoE) structure. See, for example, Zhou et al., Mixture-of-Experts with Expert Choice Routing, arXiv:2202.09368v2 (October 14, 2022).

[0071] Input 2(or more) can generally contain various types of data, or represent various types of data in other ways. Input 2(or more) can contain one type or many different types of data. Output 3(or more) may be the same type(or more) of data as Input 2(or more) or different types of data. Output 3(or more) can contain one type or many different types of data.

[0072] Examples of data types for Input 2(or more) or Output 3(or more) include natural language text data, software code data (e.g., source code, object code, machine code, or any other form of computer-readable instruction or programming language), machine code data (e.g., binary code, assembly code, or any other form of machine-readable instruction that can be directly executed by a computer's central processing unit), assembly code data (e.g., a low-level programming language that uses a symbolic representation of machine code instructions to program a processing unit), genetic data or other chemical or biochemical data, image data, audio data, audiovisual data, tactile data, biometric data, medical data, financial data, statistical data, geographic data, astronomical data, historical data, and sensor data in general (e.g., digital or analog values, such as voltage or other absolute or relative value measurements obtained from actual or artificial inputs such as audio sensors, light sensors, or displacement sensors). The data may be raw or processed, and may be in any format or schema.

[0073] In multimodal input 2 or output 3, exemplary combinations of data types include image data and audio data, image data and natural language data, natural language data and software code data, image data and biometric data, sensor data and medical data, etc. Please understand that any combination of data types is possible in input 2 or output 3.

[0074] An exemplary input 2 may include one or more data types, such as the exemplary data types described above. An exemplary output 3 may include one or more data types, such as the exemplary data types described above. The data types of input 2 may be the same as or different from the data types of output 3. It should be understood that the exemplary data types described above are provided for illustrative purposes only. The data types contemplated within the scope of this disclosure are not limited to the examples above.

[0075] Examples of machine learning-based sequencing models Figure 5 is a block diagram of an exemplary embodiment of an exemplary machine learning model configured to process a sequence of information. For example, an exemplary embodiment of machine learning model 1(or more) may include machine learning sequence processing model 4(or more). The exemplary system can pass input 2(or more) to sequence processing model 4(or more). Sequence processing model 4(or more) may include one or more machine learning components. Sequence processing model 4(or more) can process data from input 2(or more) to obtain input sequence 5. Input sequence 5 may include one or more input elements 5-1, 5-2, ..., 5-M, etc., obtained from input 2(or more). Sequence processing model 4 can process input sequence 5 using prediction layer 6(or more) to generate output sequence 7. Output sequence 7 may include one or more output elements 7-1, 7-2, ..., 7-N, etc., generated based on input sequence 5. The system can generate output 3(or more) based on output sequence 7.

[0076] A sequence processing model 4(or more) may include one or more machine learning model components configured to take in, generate, or otherwise infer sequences of information. For example, some exemplary sequence processing models in the text domain are referred to as “Large-Scale Language Models,” or LLMs. See, for example, the PaLM 2 Technical Report, Google, https: / / ai.google / static / documents / palm2techreport.pdf(nd). Other exemplary sequencing models can operate in other domains, such as the image domain (e.g., Dosovitskiy et al., An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, arXiv:2010.11929v2 (June 3, 2021)), the speech domain (e.g., Agostinelli et al., MusicLM: Generating Music From Text, arXiv:2301.11325v1 (January 26, 2023)), and the biochemistry domain (e.g., Jumper et al., Highly accurate protein structure prediction with AlphaFold, 596 Nature 583 (August 26, 2021)). Sequencing models 4(or more) can process one or more types of data simultaneously. The sequence processing model 4 (or more) may include relatively large models (e.g., many parameters, high computational cost), relatively small models (e.g., few parameters, low computational load), or both.

[0077] Generally, a sequence processing model 4(or more) can use data from input 2(or more) to obtain an input sequence 5. For example, input sequence 5 may contain a representation of the data from input 2(or more) in a format understood by the sequence processing model 4(or more). One or more machine learning-trained components of the sequence processing model 4(or more) can take data from input 2(or more), parse that data into pieces compatible with the processing architecture of the sequence processing model 4(or more) (e.g., via "tokenization"), and project those pieces into the input space associated with the prediction layer 6(or more) (e.g., via "embedding").

[0078] Sequence processing models 4(or more) can take data from input 2(or more), parse the data into a sequence of elements, and obtain input sequence 5. For example, a portion of the input data from input 2(or more) can be broken down into pieces, and collectively these pieces represent the content of a portion of that input data. These pieces can provide elements for a sequence.

[0079] Elements 5-1, 5-2, ..., 5-M can, in some cases, represent constituent units for capturing or representing meaningful information in a particular data domain. For example, an element can describe "atomic units" across one or more domains. For instance, in the case of a text input source, an element could correspond to a group of one or more word or subword components, such as one or more sets of characters.

[0080] For example, elements 5-1, 5-2, ..., 5-M can represent tokens obtained using a tokenizer. For example, a tokenizer can process a given portion of an input source and output a set of tokens representing that portion of the input source (e.g., corresponding to input elements 5-1, 5-2, ..., 5-M). Various tokenization techniques can be used. For example, a text input source(s) can be tokenized using byte-pair coding (BPE) techniques. See, for example, Kudo et al., SentencePiece: A simple and language-independent subword tokenizer and detokenizer for Neural Text Processing, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (System Demonstrations), pp. 66-71 (October 31 - November 4, 2018), https: / / aclanthology.org / D18-2012.pdf. Image-based input sources (multiple sources are possible) can be tokenized by extracting and serializing patches from the images.

[0081] In general, any data type can be serialized and processed into input sequence 5. It should be understood that the elements 5-1, 5-2, ..., 5-M shown in Figure 5 may be tokens or their embedded representations.

[0082] The prediction layer 6(or more) can predict one or more output elements 7-1, 7-2, ..., 7-N based on the input elements. The prediction layer 6(or more) may include one or more trained machine learning model architectures, such as one or more trained parameter layers that manipulate and transform the input(or more) to extract higher-dimensional meaning and relationships between elements from the input elements 5-1, 5-2, ..., 5-M. In this way, for example, the exemplary prediction layer 6(or more) can predict a new output(or more) taking into account the context provided by the input sequence 5.

[0083] Predictive layers 6(or more) can evaluate associations between parts of input sequence 5 and specific output elements. These associations can provide information for predicting the likelihood that a particular output will follow the input context. For example, consider the text snippet "The carpenter's toolbox was small and heavy. It was full of __." An example predictive layer 6(or more) can determine that "It" refers back to "toolbox" by determining the relationships between each embedding. An example predictive layer 6(or more) can also link "It" to attributes of the toolbox such as "small" and "heavy". Based on these associations, predictive layer 6(or more) can assign the word "nails" with a higher probability than the word "sawdust," for example.

[0084] A transformer is an exemplary architecture that may be used in a prediction layer 4(or more). See, for example, Vaswani et al., Attention Is All You Need, ARXIV:1706.03762v7 (August 2, 2023). A transformer is an example of a machine learning model architecture that uses an attention mechanism to compute associations between items in a context window. A context window may include an input sequence 5 and a sequence containing, possibly, one or more output elements 7-1, 7-2, ..., 7-N. A transformer block may include one or more attention layers and one or more post-attention layers (or more) (e.g., feedforward layers such as multilayer perceptrons).

[0085] The prediction layer 6(or more) may include, in addition to or instead of, a converter-based architecture, other machine learning model architectures. For example, not only convolutional neural networks (CNNs) but also recurrent neural networks (RNNs) and long-term short-term memory (LSTM) models can be used. In general, the prediction layer 6(or more) can leverage various types of artificial neural networks that can understand or generate sequences of information.

[0086] Output sequence 7 may contain the same or different data types as input sequence 5, or represent them in other ways. For example, input sequence 5 may represent text data, and output sequence 7 may represent text data. Input sequence 5 may represent image, audio, or audiovisual data, and output sequence 7 may represent text data (for example, describing image, audio, or audiovisual data). It will be understood that prediction layers 6(or more) and any other intervening model components of sequence processing model 4(or more) can be configured to receive various data types of input sequence 5(or more) and output various data types of output sequence 7(or more).

[0087] Output sequence 7 can have various relationships with input sequence 5. Output sequence 7 may be a continuation of input sequence 5. Output sequence 7 may be a complement to input sequence 5. Output sequence 7 can translate, transform, extend, or otherwise modify input sequence 5. Output sequence 7 can answer, evaluate, confirm, or otherwise respond to input sequence 5. Output sequence 7 can implement (or write instructions for implementing) the instructions provided via input sequence 5.

[0088] The output sequence 7 can be generated autoregressively. For example, in some applications, the output of one or more prediction layers 6 can be passed through one or more output layers (e.g., softmax layers) to obtain a probability distribution over an output vocabulary (e.g., a text or symbol vocabulary) conditioned on a set of input elements in a context window. In this way, the output sequence 7 can be generated autoregressively, for example, by sampling the element most likely to be output next, adding that element to the context window, regenerating the probability distribution based on the updated context window, sampling the element most likely to be output next, and so on.

[0089] Output sequence 7 can be generated non-autoregressively. For example, multiple output elements of output sequence 7 can be predicted together without explicit sequential conditioning of each other. See, for example, Saharia et al., Non-Autoregressive Machine Translation with Latent Alignments, ARXIV:2004.07437v3 (November 16, 2020).

[0090] Output sequence 7 may contain one or more parts or elements. In an exemplary content generation configuration, output sequence 7 may contain multiple elements (e.g., sentences of text, discretized waveform values, computer code, etc.) corresponding to multiple parts of the generated output sequence. In an exemplary classification configuration, output sequence 7 may contain a single element associated with the classification output. For example, the output "Vocabulary" may contain a set of classes to which the input sequence is classified. For example, a visual transducer block may pass latent state information to a multilayer perceptron that outputs the most likely class values ​​associated with the input image.

[0091] Figure 6 is a block diagram of an exemplary technique for feeding an exemplary input sequence 8. The input sequence 8 may include various functional elements that form part of the model infrastructure, such as element 8-0, obtained from a task indicator 9 that signals to any model(s) processing the input sequence 8 that a particular task is being performed (for example, to help fit the performance of the model(s) to that particular task). The input sequence 8 may include various data elements from different data modalities. For example, input modality 10-1 may include data from one modality. A data-to-sequence model 11-1 can process the data from input modality 10-1 and project that data into a format compatible with the input sequence 8 (for example, one or more vectors with dimensions set according to the dimensions of the input sequence 8) to obtain elements 8-1, 8-2, and 8-3. Other input modalities 10-2 may include data from different modalities. The data-to-sequence model 11-2 can project data from input modality 10-2 into a format compatible with input sequence 8, obtaining elements 8-4, 8-5, and 8-6. Other input modalities 10-3 may include other different data modalities. The data-to-sequence model 11-3 can project data from input modality 10-3 into a format compatible with input sequence 8, obtaining elements 8-7, 8-8, and 8-9.

[0092] Input sequence 8 may be the same as or different from input sequence 5. Input sequence 8 may be a multimodal input sequence containing elements that represent data from different modalities using a common dimensional representation. For example, the embedding space may have P dimensions. Input sequence 8 may be configured to contain multiple elements having P dimensions. In this way, for example, exemplary embodiments can facilitate information extraction and inference across diverse data modalities by projecting data onto elements in the same embedding space for comparison, combination, or other computations between them.

[0093] For example, elements 8-0, ..., 8-9 can represent specific locations within a multidimensional embedding space. Some elements can be mapped to a discrete set of locations within the embedding space. For instance, elements corresponding to individual members of a given vocabulary of tokens can be mapped to discrete locations within the embedding space associated with those tokens. Other elements can be continuously distributed throughout the embedding space. For example, some data types can be decomposed into continuously defined parts (e.g., image patches) that can be described using continuously distributed locations within the embedding space.

[0094] In some embodiments, the expressiveness of the embedding space may not be limited to the meaning associated with any particular set of tokens or other constituent units. For example, a continuous embedding space can encode a spectrum of higher-order information. Individual pieces of information (e.g., tokens) can be mapped to specific points within that space; for example, the token for the word "dog" can be projected onto an embedding value that points to a specific location within the embedding space associated with dog-related information. Similarly, an image patch of a dog on grass can be projected onto the embedding space. In some embodiments, the projection of the dog image may be similar to the projection of the word "dog," while also having similarity to the projection of the word "grass," and at the same time, being different from both. In some embodiments, the projection of the image patch may not exactly match any single projection of a single word. In some embodiments, the projection of the image patch may match a combination of the projections of the words "dog" and "grass." In this way, for example, a higher-order embedding space can encode information that is independent of the data modality in which the information is represented.

[0095] The task indicator 9 may include a model or model component configured to inject input values, represented by elements 8-0, into the input sequence 8, which identify the task being performed and signal which task is being performed. For example, the input values ​​can be provided as a data type associated with an input modality and projected together with that input modality (for example, the input values ​​may be text task labels embedded in the input along with other text data, or pixel-based representations of tasks embedded in the input along with other image data). The input values ​​can be provided as a data type that is different from, or at least independent of, other inputs(s). For example, the input values ​​represented by elements 8-0 can be learned in a contiguous embedding space.

[0096] The input modalities 10⁻¹, 10⁻², and 10⁻³ can be associated with various different data types (for example, as described above for input 2(or more) and output 3(or more)).

[0097] The data-to-sequence models 11-1, 11-2, and 11-3 may be the same or different from each other. The data-to-sequence models 11-1, 11-2, and 11-3 can be adapted to their respective input modalities 10-1, 10-2, and 10-3. For example, a text data-to-sequence model can subdivide a portion of the input text and project the subdivided portion onto an element(s) in the input sequence 8 (e.g., elements 8-1, 8-2, 8-3, etc.). An image data-to-sequence model can subdivide an input image and project the subdivided portion onto an element(s) in the input sequence 8 (e.g., elements 8-4, 8-5, 8-6, etc.). A data-to-sequence model for any data type can subdivide the input of that data type and project the subdivided portion onto an element(s) in the input sequence 8 (e.g., elements 8-7, 8-8, 8-9, etc.).

[0098] The data-to-sequence models 11-1, 11-2, and 11-3 can form part of a machine learning-based sequence processing model 4(or more). The data-to-sequence models 11-1, 11-2, and 11-3 can be trained jointly with or independently of the machine learning-based sequence processing model 4(or more). The data-to-sequence models 11-1, 11-2, and 11-3 can be trained end-to-end using the machine learning-based sequence processing model 4(or more).

[0099] Examples of machine learning model development platforms Figure 7 is a block diagram of an exemplary model development platform 12 that can facilitate the creation, adaptation, and refinement of exemplary machine learning models (e.g., machine learning model 1(or more), sequence processing model 4(or more)). The model development platform 12 can provide multiple different toolkits that the developer system can employ in developing new or adapted machine learning models.

[0100] The model development platform 12 can provide one or more model libraries 13 containing the constituent units of a new model. The model library 13 may include one or more pre-trained foundational models 13-1, which can provide a backbone of processing power across various tasks. The model library 13 may include one or more pre-trained expert models 13-2, which can focus on performance in a specific domain of expertise. The model library 13 may include various model primitives 13-3 that can provide low-level architectures or (optionally pre-trained) constituent elements, which can be assembled into various configurations as needed.

[0101] The model development platform 12 can receive selections of various model components 14. The model development platform 12 can pass the selected model components 14 to the workbench 15, which then incorporates the selected model components 14 into the development model 16.

[0102] The workbench 15 can facilitate further refinement and adaptation of the development model 16 by leveraging several different toolkits integrated with the model development platform 12. For example, the workbench 15 can easily align the development model 16 to a desired performance profile for various tasks by using the model alignment toolkit 17.

[0103] The model alignment toolkit 17 can provide several tools for generating outputs that match desired behavioral characteristics in the development model 16. Alignment may include improving the accuracy, precision, recall, etc., of the model output. Alignment may include ensuring that the model output adheres to an output style, schema, or other desired characteristics. Alignment may be general or domain-specific. For example, a pre-trained base model 13-1 can start with an initial level of performance across multiple domains. Alignment of the pre-trained base model 13-1 may include improving performance in information or tasks in a specific domain (e.g., at the expense of performance in other domains of information or tasks).

[0104] The model alignment toolkit 17 can integrate one or more datasets 17-1(or more) to match the development model 16. The curated datasets 17-1(or more) may include labeled or unlabeled training data. The datasets 17-1(or more) can be obtained from public domain datasets. The datasets 17-1(or more) can be obtained from private datasets associated with one or more developer systems(or more) for the alignment of bespoke machine learning models(or more) customized for private use cases.

[0105] The pre-training pipeline 17-2 may include a machine learning model training workflow configured to update the development model 16 across a large and potentially noisy dataset. For example, pre-training can utilize unsupervised learning techniques (e.g., denoising) to process a large number of training instances to update model parameters from an initialized state and achieve desired baseline performance. The pre-training pipeline 17-2 can perform pre-training by leveraging an unlabeled dataset within dataset 17-1(or more). The workbench 15 can implement the pre-training pipeline 17-2 to pre-train the development model 16.

[0106] The fine-tuning pipeline 17-3 may include a machine learning model training workflow configured to improve the model parameters of the development model 16 using higher quality data. The fine-tuning pipeline 17-3 can update the development model 16 by performing supervised training using labeled datasets in dataset 17-1(or more). The fine-tuning pipeline 17-3 can update the development model 16 by performing reinforcement learning using reward signals from user feedback signals. The workbench 15 can implement the fine-tuning pipeline 17-3 for fine-tuning the development model 16.

[0107] The prompt library 17-4 may include a set of inputs configured to elicit behavior that meets a desired performance criterion. The prompt library 17-4 may include few-shot prompts (e.g., inputs that provide examples of desired model outputs to prepend to a desired runtime query), chain-of-thought prompts (e.g., inputs that provide step-by-step reasoning in examples to facilitate thorough reasoning by the model), and so on.

[0108] Exemplary prompts can be obtained from the available repositories of the prompt library 17-4. Exemplary prompts may be provided by one or more developer systems using the workbench 15.

[0109] In some embodiments, a pre-trained or fine-tuned model can achieve sufficient performance even without any examples in the input. For example, a zero-shot prompt may contain input without examples. The zero-shot prompt may be within a domain in the training dataset or outside of the training domain(s).

[0110] The prompt library 17-4 may include one or more prompt engineering tools. The prompt engineering tools can provide workflows for obtaining or learning optimized prompt values. The prompt engineering tools can facilitate direct learning of prompt values ​​(e.g., input element values) based on one or more training iterations. The workbench 15 can implement the prompt engineering tools in the development model 16.

[0111] The prompt library 17-4 may include a pipeline for prompt generation. For example, inputs can be generated using the development model 16 itself or other machine learning models. In this way, for example, the first model can process information about a task and output an input that the second model processes to carry out the steps of the task. The second model may be the same as or different from the first model. The workbench 15 can implement the prompt generation pipeline in the development model 16.

[0112] The prompt library 17-4 may include a pipeline for context injection. For example, the performance of the development model 16 for a particular task may improve when additional context is provided for performing that task. The prompt library 17-4 may include software components configured to identify a desired context, retrieve that context from an external source (e.g., a database, a sensor, etc.), and add that context to the input prompt. The workbench 15 can implement a pipeline for context injection in the development model 16.

[0113] While the various training examples described herein with respect to the model development platform 12 refer to "pre-training" and "fine-tuning," it should be understood that the model alignment toolkit 17 can generally support a variety of training techniques adapted for training various machine learning models. Exemplary training techniques can correspond to the aforementioned exemplary training methods 300.

[0114] The model development platform 12 may include a model plugin toolkit 18. The model plugin toolkit 18 may include a variety of tools configured to extend the functionality of machine-learned models by integrating them with other systems, devices, and software components. For example, machine-learned models can use tools as needed to improve the quality of their performance. For example, deterministic tasks can be offloaded to dedicated tools instead of probabilistically performing tasks with an increased risk of error. For example, instead of autoregressively predicting the solution to a system of linear equations, a machine-learned model can recognize which tool to call to obtain the solution and pass the system of equations to the appropriate tool. The tool may be a conventional system of equations solver that can act deterministically to solve the system of equations. The output of the tool may be returned in response to the original query. In this way, by using tools, some exemplary models can focus on the strengths of machine-learned models, such as understanding the intent of unstructured requests to a task, while simultaneously extending the model's performance by offloading specific tasks to more focused tools to mechanically apply deterministic algorithms to more specific problems.

[0115] The Model Plugin Toolkit 18 may include a validation tool 18-1. The validation tool 18-1 may include a tool that can analyze and verify the output(s) of a machine learning model. The validation tool 18-1 may include heuristics designed to establish specific thresholds to apply to the model output. For example, the validation tool 18-1 may justify the output of a machine learning model to a structured data source (e.g., to mitigate hallucination).

[0116] The model plugin toolkit 18 may include a tool package 18-2 for implementing one or more tools, which may include scripts or other executable code that can be run with the development model 16. The tool package 18-2 may include one or more inputs (e.g., few-shot prompts that guide the model to output tool calls in the appropriate syntax) configured to cause a machine-trained model(s) to implement the tool. The tool package 18-2 may include, for example, fine-tuned training data for training the model to use the tool.

[0117] The model plug-in toolkit 18 may include an interface for calling an external application programming interface (API) 18-3. For example, in addition to directly implementing tool calls and tool code in the development model 16, or instead, the development model 16 may be configured to output instructions that initiate API calls to send or retrieve data via an external system.

[0118] The model plugin toolkit 18, in conjunction with the prompt library 17-4, can build a catalog of available tools for use in the development model 16. For example, the model can receive a catalog of available tools as input, and the model can select a tool from the available tools and generate an output that initiates a tool call to use that tool.

[0119] The model development platform 12 may include a computational optimization toolkit 19 for optimizing the computational performance of the development model 16. For example, tools for model compression 19-1 may allow the size of the development model 16 to be reduced while maintaining a desired level of performance. For example, model compression 19-1 may include quantization workflows, weight pruning, and sparsification techniques. Tools for hardware acceleration 19-2 may facilitate the configuration of model storage and executables to work optimally with different hardware resources. For example, hardware acceleration 19-2 may include tools for optimally partitioning the model for distributed processing across multiple processing units to increase bandwidth and reduce integrated memory requirements. Tools for distillation 19-3 may provide training for a lighter model based on the knowledge encoded in the development model 16. For example, the development model 16 may be a high-performance, large-scale machine learning model optimized using the model development platform 12. To obtain a lightweight model for execution in resource-constrained environments, the smaller model may be a "student model" which learns by mimicking the development model 16 as the "teacher model". In this way, for example, the investment made in training the parameters and configuration of the development model 16 can be efficiently transferred to a smaller model for more efficient inference.

[0120] Workbench 15 may implement one or more of the toolkits implemented in the model development platform 12, or it may not implement any of them. Workbench 15 may output an output model 20 based on the development model 16. The output model 20 may be a deployed version of the development model 16. The output model 20 may be a development or training checkpoint of the development model 16. The output model 20 may be a distilled, compressed, or otherwise optimized version of the development model 16.

[0121] Figure 8 is a block diagram of an exemplary training flow for training a machine learning-prepared development model 16. One or more parts of the exemplary training flow may be performed by a computing system including one or more computing devices, such as the computing system described with reference to other figures. Each part of the exemplary training flow may be performed by any one (or any combination) of one or more computing devices. Furthermore, one or more parts of the exemplary training flow may be performed by hardware components of the device(s) described herein, for example, to train one or more systems or models. Figure 8 shows the elements to be performed in a particular order for illustrative and explanatory purposes. Those skilled in the art will understand, by using the disclosure provided herein, that any element of the method described herein may be adapted, rearranged, extended, omitted, combined, or modified in various ways without departing from the scope of this disclosure. Figure 8 is described for illustrative purposes with reference to elements / terms described in relation to other systems and drawings, and is not intended to be limiting. One or more parts of the exemplary training flow may be performed by other systems, additionally or alternatively.

[0122] First, the development model 16 can maintain its initial state as an initialized model 21. The development model 16 can be initialized with weight values. The initial weight values ​​may be random or based on an initialization schema. The initial weight values ​​may be based on previous pre-training on the same or different models.

[0123] The initialized model 21 can be pre-trained in the pre-training stage 22. The pre-training stage 22 can be performed using one or more pre-training pipelines 17-2 on data from dataset 17-1(or more). For example, if the initialized model 21 is already pre-trained (e.g., if the development model 16 is a pre-trained base model or expert model, or is based on a pre-trained base model or expert model), pre-training can be omitted.

[0124] In this case, the pre-trained model 23 may be a new version of the development model 16, which can persist as the development model 16 or as a new development model. The pre-trained model 23 may be in its initial state if the development model 16 is already pre-trained. The pre-trained model 23 can be fine-tuned in the fine-tuning stage 24. The fine-tuning stage 24 can be performed using one or more fine-tuning pipelines 17-3 on data from dataset 17-1(or more). For example, fine-tuning can be omitted if the performance of the pre-trained model is sufficient, if the model has already been fine-tuned, or if another fine-tuning method is preferred.

[0125] In this case, the fine-tuned model 29 may be a new version of the development model 16, which can persist as development model 16 or as a new development model. If the development model 16 is already fine-tuned, the fine-tuned model 29 may be in its initial state. The fine-tuned model 29 can be improved by user feedback 26. For example, improvements based on user feedback 26 may include reinforcement learning based on human feedback from human users of the fine-tuned model 25. Since reinforcement learning can take the form of fine-tuning, it should be understood that the fine-tuning stage 24 may include a stage for improvement using user feedback 26. Improvements based on user feedback 26 may create a refined model 27. The refined model 27 may be output to a downstream system 28 for deployment or further development.

[0126] In some embodiments, computational optimization operations may be applied before, during, or after each stage. For example, an initialized model 21 may undergo computational optimization 29-1 (e.g., using the computational optimization toolkit 19) before the pre-training stage 22. A pre-trained model 23 may undergo computational optimization 29-2 (e.g., using the computational optimization toolkit 19) before the fine-tuning stage 24. A fine-tuning model 25 may undergo computational optimization 29-3 (e.g., using the computational optimization toolkit 19) before being refined by user feedback 26. A refined model 27 may undergo computational optimization 29-4 (e.g., using the computational optimization toolkit 19) before being output to a downstream system 28(or more). Computational optimizations 29-1, ..., 29-4(or more) may all be the same, all be different, or include at least some different optimization techniques.

[0127] Examples of machine learning model inference systems Figure 9 is a block diagram of an inference system for running one or more machine learning models 1(or more) to perform inference (e.g., for training and deployment). Model host 31 can receive machine learning models 1(or more). Model host 31 can host one or more model instances 31-1(or more), which may be one or more instances of one or more models. Model host 31 can host model instances 31-1(or more) using available computing resources 31-2 associated with model host 31.

[0128] The model host 31 can perform inference on behalf of one or more clients 32. Clients 32 can send input requests 33 to the model host 31. Using the input requests 33, the model host 31 can obtain inputs 2 for input to a machine-trained model 1. The machine-trained model 1 can process inputs 2 to produce outputs 3. Using outputs 3, the model host 31 can return an output payload 34 to respond to the input requests 33 from the clients 32. The output payload 34 may contain or be based on outputs 3.

[0129] The model host 31 can extend its inference tasks by utilizing various other resources and tools. For example, the model host 31 can communicate with a tool interface 35 to facilitate the use of tools by model instances 31-1(or more). The tool interface 35 may include local or remote APIs. The tool interface 35 may include integrated scripts or other software functions. The model host 31 can involve online learning interfaces 36(or more) to facilitate the continuous improvement of machine-learned models 1(or more). For example, the online learning interfaces 36(or more) can be used within a reinforcement learning loop to obtain user feedback on the inferences provided by the model host 31. The model host 31 can access runtime data sources 37(or more) to extend input 2(or more) with additional contextual information. For example, runtime data sources 37(or more) may include a knowledge graph 37-1 that facilitates the retrieval of structured information for information associated with input requests 33(or more) (e.g., search engine services). The runtime data source 37(or more) may include public or private, external or local databases 37-2(or more) that can store information associated with input requests 33(or more) to extend input 2(or more). The runtime data source 37(or more) may include account data 37-3, which may be retrieved in association with user accounts corresponding to client 32 in order to customize the behavior of the model host 31 accordingly.

[0130] Model host 31 can be implemented by one or more computing devices or systems. Client 2(s) can be implemented by one or more computing devices or systems, which may include computing devices or systems shared with model host 31.

[0131] For example, the model host 31 can operate on a server system that provides machine learning services to client devices (multiple) that run client 32 (or more) (for example, via a local or wide area network). The client devices (multiple) may be end-user devices used by individuals. The client devices (multiple) may be a server system that runs client 32 (or more) and provides various functions as services to downstream end-user devices.

[0132] In some embodiments, the model host 31 may operate on the same device or system as the client 32(or more). The model host 31 may be a machine learning service that runs on a device and provides machine learning functionality to one or more applications running on client devices, which may include application implementation clients 32(or more). The model host 31 may be part of the same application as the client 32(or more). For example, the model host 31 may be a subroutine or method implemented by one part of the application, and the client 32(or more) may be another subroutine or method that engages with the model host 31 to perform inference functionality within the application. It should be understood that the model host 31 and the client 32(or more) may have a variety of different configurations.

[0133] A model instance 31-1(or more) may contain one or more machine learning models available for inference. A model instance 31-1(or more) may contain weights or other model components that are stored in persistent storage, temporarily cached, or loaded into high-speed memory. A model instance 31-1(or more) may contain multiple instances of the same model (for example, to run more requests in parallel with the same model). A model instance 31-1(or more) may contain instances(or more) of different models(or more). A model instance 31-1(or more) may contain cached intermediate states of active or inactive models(or more) used to speed up inference for those models. For example, an inference session with a particular model may generate a significant amount of computational results that can be reused for future inference runs (for example, using a KV cache for a transformer-based model). These computational results can be associated with and stored for that inference session so that the session can run more efficiently when it is resumed.

[0134] Computational resources 31-2(or more) may include one or more processors (such as a central processing unit, graphical processing unit, tensor processing unit, or machine learning accelerator) connected to one or more memory devices. Computational resources 31-2(or more) may include a dynamic pool of available resources shared with other processes. Computational resources 31-2(or more) may include a memory device large enough to accommodate an entire model instance in a single memory instance. Computational resources 31-2(or more) may also partition a model instance(or more) across multiple memory devices (for example, using data parallelism or tensor parallelism). This may be done to increase parallelism or to run a large model using multiple memory devices that cannot fit the entire model in memory on individual memory devices.

[0135] Input request 33 may contain data for input 2(or more). Model host 31 can process input request 33 to obtain input(or more). Input(or more) can be obtained directly from input request 33 or by using input request 33. Input request 33 can be submitted to model host 31 via API.

[0136] The model host 31 can perform inference in parallel across batches of input requests 33. For example, a model instance 31-1 can consist of an input structure having a batch dimension. Separate inputs 2(or more) can be distributed across the batch dimension (e.g., rows of an array). Separate inputs 2(or more) can contain completely different contexts. Separate inputs 2(or more) can be multiple inference steps of the same task. Separate inputs 2(or more) can be arranged alternately within the input structure, so that any given inference cycle can act on different parts of each input 2(or more). In this way, for example, the model host 31 can perform inference in parallel over batches, so that the output(or more) can also include a batch dimension and return inference results for the batched inputs 2(or more) in parallel. In this way, for example, batches of input requests 33(or more) can be processed in parallel for higher throughput of the output payload(or more) 34(or more).

[0137] The output payload 34 may contain, or be based on, the outputs 3(or more) from machine learning model 1(or more). The model host 31 can process the outputs 3(or more) to obtain the output payload 34. This may involve chaining multiple inferences (e.g., iteratively, recursively, across the same model(or more) or different models(or more)) to arrive at the final output of the task returned in the output payload 34. The output payload 34 can be sent to client 32(or more) via an API.

[0138] The online learning interface 36(multiple) can facilitate reinforcement learning of the machine learning model 1(multiple). The online learning interface 36(multiple) can facilitate reinforcement learning with human feedback (RLHF). The online learning interface 36(multiple) can facilitate associative learning of the machine learning model 1(multiple).

[0139] The model host 31 can run a machine learning model 1(or more) to perform inference on various tasks using various types of data. For example, various different inputs 2(or more) and outputs 3(or more) can be used for various different tasks. In some embodiments, input 2(or more) may be image data or can be represented in other ways. The machine learning model 1(or more) can process the image data to generate an output. For example, the machine learning model 1(or more) can process the image data to generate an image recognition output (e.g., recognition of image data, latent embedding of image data, coded representation of image data, hash of image data, etc.). As another example, the machine learning model 1(or more) can process the image data to generate an image segmentation output. As yet another example, the machine learning model 1(or more) can process the image data to generate an image classification output. As yet another example, the machine learning model 1(or more) can process the image data to generate an image data modification output (e.g., modification of image data, etc.). As another example, a machine learning model 1(or more) can process image data to generate encoded image data output (e.g., encoded and / or compressed representations of the image data). As yet another example, a machine learning model 1(or more) can process image data to generate upscaled image data output. As yet another example, a machine learning model 1(or more) can process image data to generate predictive output.

[0140] In some embodiments, the task is a computer vision task. In some cases, input 2(or more) includes pixel data from one or more images, and the task is an image processing task. For example, the image processing task may be image classification, and the output is a set of scores, each corresponding to a different object class, representing the likelihood that one or more images depict an object belonging to that object class. The image processing task may be object detection, and the image processing output identifies one or more regions within one or more images, and for each region, the likelihood that the region depicts an object of interest. As another example, the image processing task may be image segmentation, and the image processing output defines, for each pixel in one or more images, the likelihood for each category within a given set of categories. For example, the set of categories may be foreground and background. As yet another example, the set of categories may be object classes. As yet another example, the image processing task may be depth estimation, and the image processing output defines, for each pixel in one or more images, each depth value. As another example, the image processing task may be motion estimation, where the network input includes multiple images, and the image processing output defines the motion of the scene depicted between pixels in the network input, for each pixel in the input images.

[0141] In some embodiments, input 2(or more) may be natural language data or can be represented in other ways. A machine learning model 1(or more) can process the natural language data to produce an output. For example, a machine learning model 1(or more) can process natural language data to produce a language coding output. Another example is a machine learning model 1(or more) can process natural language data to produce a latent text embedding output. Another example is a machine learning model 1(or more) can process natural language data to produce a translation output. Another example is a machine learning model 1(or more) can process natural language data to produce a classification output. Another example is a machine learning model 1(or more) can process natural language data to produce a text segmentation output. Another example is a machine learning model 1(or more) can process natural language data to produce a semantic intent output. As another example, a machine learning model 1(or more) can process natural language data to generate upscaled text or natural language output (e.g., text or natural language data of higher quality than the input text or natural language). As yet another example, a machine learning model 1(or more) can process natural language data to generate predictive output (e.g., one or more predicted next parts of natural language content).

[0142] In some embodiments, input 2(or more) may be speech data (e.g., data describing spoken natural language, such as audio data or text data) or can represent the speech data in other ways. A machine learning model 1(or more) can process the speech data to produce an output. For example, a machine learning model 1(or more) can process the speech data to produce a speech recognition output. Another example is a machine learning model 1(or more) can process the speech data to produce a speech translation output. Another example is a machine learning model 1(or more) can process the speech data to produce a latent embedding output. Another example is a machine learning model 1(or more) can process the speech data to produce an encoded speech output (e.g., an encoded representation and / or compressed representation of the speech data). Another example is a machine learning model 1(or more) can process the speech data to produce an upscaled speech output (e.g., speech data of higher quality than the input speech data). As another example, a machine learning model 1(or more) can process speech data to generate a textual representation output (e.g., a textual representation of the input speech data). As yet another example, a machine learning model 1(or more) can process speech data to generate a predictive output.

[0143] In some embodiments, input 2(or more) may be latent coded data (e.g., a latent spatial representation of the input) or can be represented in other ways. A machine learning model 1(or more) can process the latent coded data to produce an output. For example, a machine learning model 1(or more) can process the latent coded data to produce a recognition output. As another example, a machine learning model 1(or more) can process the latent coded data to produce a reconstruction output. As yet another example, a machine learning model 1(or more) can process the latent coded data to produce a search output. As yet another example, a machine learning model 1(or more) can process the latent coded data to produce a reclustering output. As yet another example, a machine learning model 1(or more) can process the latent coded data to produce a prediction output.

[0144] In some embodiments, input 2(or more) may be statistical data, or may represent statistical data in other ways. The statistical data may be, may represent, or may otherwise include computer-processed or computed data from some other data source. A machine learning model 1(or more) can process the statistical data to produce an output. For example, a machine learning model 1(or more) can process the statistical data to produce a recognition output. As another example, a machine learning model 1(or more) can process the statistical data to produce a prediction output. As yet another example, a machine learning model 1(or more) can process the statistical data to produce a classification output. As yet another example, a machine learning model 1(or more) can process the statistical data to produce a segmentation output. As yet another example, a machine learning model 1(or more) can process the statistical data to produce a visualization output. As yet another example, a machine learning model 1(or more) can process the statistical data to produce a diagnostic output.

[0145] In some embodiments, input 2(or more) may be sensor data or can represent sensor data in other ways. A machine learning model 1(or more) can process the sensor data to generate an output. For example, a machine learning model 1(or more) can process sensor data to generate a recognition output. As another example, a machine learning model 1(or more) can process sensor data to generate a prediction output. As yet another example, a machine learning model 1(or more) can process sensor data to generate a classification output. As yet another example, a machine learning model 1(or more) can process sensor data to generate a segmentation output. As yet another example, a machine learning model 1(or more) can process sensor data to generate a visualization output. As yet another example, a machine learning model 1(or more) can process sensor data to generate a diagnostic output. As yet another example, a machine learning model 1(or more) can process sensor data to generate a detection output.

[0146] In some embodiments, a machine-trained model(s) may be configured to perform tasks that involve encoding (and / or correspondingly decoding) input data for reliable and / or efficient transmission or storage. For example, the task may be an audio compression task. The input may include audio data, and the output may include compressed audio data. In another example, the input may include visual data (e.g., one or more images or videos), and the output may include compressed visual data, and the task is a visual data compression task. In yet another example, the task may involve generating embeddings for input data (e.g., input audio or visual data). In one case, the input may include audio data representing spoken utterances, and the task is a speech recognition task. The output may include text output mapped to the spoken utterances. In one case, the task may involve encrypting or decrypting input data. In one case, the task may involve microprocessor performance tasks, such as branch prediction or memory address translation.

[0147] In some embodiments, the task is a generative task, and a machine-trained model 1(or more) can be configured to output content generated considering an input 2(or more). For example, input 2(or more) may be data from one or more modalities that encode a context for generating additional content, or can otherwise represent data from one or more modalities.

[0148] In some embodiments, the task may be a text completion task. A machine-trained model 1(or more) can be configured to process an input 2(or more) representing text data and to produce an output 3(or more) representing additional text data that completes a text sequence containing the input 2(or more). For example, a machine-trained model 1(or more) can be configured to produce an output 3(or more) to complete a sentence, paragraph, or portion of text that follows a portion of text represented by the input 2(or more).

[0149] In some embodiments, the task may be an instruction that follows the task. A machine learning model 1(or more) can be configured to process an input 2(or more) representing an instruction to perform a certain function and to produce an output 3(or more) that achieves the goal of satisfying the function of that instruction (e.g., at least one step of a multi-step procedure to perform that function). The output 3(or more) can represent data of the same or different modality as the input 2(or more). For example, the input 2(or more) may represent text data (e.g., a natural language instruction for a task to be performed), and the machine learning model 1(or more) can process the input 2(or more) to produce an output 3(or more) representing text data in response to the instruction (e.g., a natural language response, a programming language response, a machine language response, etc.). Input 2(or more) can represent image data (e.g., image-based instructions for a task to be performed, optionally accompanied by text instructions), and a machine learning model 1(or more) can process input 2(or more) to generate outputs 3(or more) representing text data in response to the instructions (e.g., natural language responses, programming language responses, machine language responses, etc.). One or more outputs 3(or more) can be generated iteratively or recursively to sequentially process and achieve the steps necessary to accomplish the requested function. For example, an initial output can be executed by an external system or processed by a machine learning model 1(or more) to complete the initial steps for performing the function. Multiple steps may be executed to obtain a final output in response to the first instruction.

[0150] In some embodiments, the task may be a question-answering task. A machine-trained model 1(or more) can be configured to process an input 2(or more) representing a question to be answered and to produce an output 3(or more) (e.g., at least one step of a multi-step procedure for performing its function) that achieves the goal of returning an answer to that question. The output 3(or more) can represent data of the same or different modality as the input 2(or more). For example, the input 2(or more) can represent text data (e.g., natural language instructions for a task to be performed), and the machine-trained model 1(or more) can process the input 2(or more) to produce an output 3(or more) (e.g., a natural language response, a programming language response, a machine language response, etc.) that represents text data responding to the question. Input 2(or more) can represent image data (e.g., image-based instructions for a task to be performed, optionally accompanied by text instructions), and a machine learning model 1(or more) can process input 2(or more) to generate outputs 3(or more) representing text data in response to a question (e.g., natural language response, programming language response, machine language response, etc.). One or more outputs 3(or more) can be generated iteratively or recursively to sequentially process and achieve steps to achieve an answer to a question. For example, an initial output can be executed by an external system or processed by a machine learning model 1(or more) to complete initial steps to obtain an answer to a question (e.g., querying a database, performing calculations, executing a script, etc.). Multiple steps may be performed to obtain a final output that responds to the question.

[0151] In some embodiments, the task may be an image generation task. A machine-trained model 1(or more) can be configured to process an input 2(or more) representing context about a desired portion of image content. The context may include text data, image data, audio data, etc. The machine-trained model 1(or more) can be configured to produce an output 3(or more) representing image data depicting an image related to the context. For example, the machine-trained model 1(or more) can be configured to generate pixel data for an image. The channel(or more) values ​​associated with the pixels in the pixel data can be selected based on the context (for example, based on probabilities determined based on the context).

[0152] In some embodiments, the task may be an audio generation task. A machine-trained model 1(or more) can be configured to process an input 2(or more) representing context about a desired portion of audio content. The context may include text data, image data, audio data, etc. The machine-trained model 1(or more) can be configured to produce an output 3(or more) representing audio data relevant to the context. For example, the machine-trained model 1(or more) can be configured to generate waveform data in image format (e.g., spectrogram). The channel(or more) values ​​associated with pixels in the image can be selected based on the context. The machine-trained model 1(or more) can be configured to generate waveform data in the form of a sequence of discrete samples of a continuous waveform. The sequence values ​​can be selected based on the context (e.g., based on probabilities determined based on the context).

[0153] In some embodiments, the task may be a data generation task. A machine-trained model 1(or more) can be configured to process an input 2(or more) that represents a context about a desired portion of data (e.g., data from various data domains, such as sensor data, image data, multimodal data, or statistical data). The desired data may be, for example, synthetic data for training other machine-trained models. The context may include any data type(or more). The machine-trained model 1(or more) can be configured to produce an output 3(or more) that represents data that matches the desired data. For example, the machine-trained model 1(or more) can be configured to generate data values ​​for injecting into a dataset. The values ​​of the data objects(or more) can be selected based on the context (e.g., based on probabilities determined based on the context).

[0154] Exemplary computing systems and devices Figure 10 is a block diagram of an exemplary network computing system capable of performing an exemplary embodiment of the present disclosure. The system may include a plurality of computing devices and systems that are communicably coupled over a network 49. An exemplary computing device 50 is described to provide an example of a computing device capable of performing any embodiment of the present disclosure (e.g., implementing a model host 31, a client 32(or more), or both). An exemplary server computing system 60 is described as an example of a server computing system capable of performing any embodiment of the present disclosure (e.g., implementing a model host 31, a client 32(or more), or both). The computing devices 50 and the server computing system 60(or more) can interact in cooperation (e.g., over the network 49) to perform any embodiment of the present disclosure (e.g., implementing a model host 31, a client 32(or more), or both). A model development platform system 70 is an exemplary system that can host or provide a model development platform 12(or more) for developing machine learning models. The third-party systems 80(or more) are exemplary systems(or more) that any of the computing devices 50, server computing systems 60(or more), or model development platform systems 70(or more) can interact with in the execution of various aspects of this disclosure (e.g., the use of third-party tools, access to third-party databases or other resources).

[0155] Network 49 may be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or any combination thereof, and may include any number of wired or wireless links. Generally, communication over Network 49 can be carried out over any type of wired or wireless connection using a wide variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encoding or formatting (e.g., HTML, XML), or protection schemes (e.g., VPN, Secure HTTP, SSL). Network 49 may also be implemented via a system bus. For example, one or more devices or systems in Figure 10 may be located in the same place as one or more other devices or systems, may be housed in them, or may be integrated with them in other ways.

[0156] Computing device 50 may be any type of computing device, such as a personal computing device (e.g., a laptop or desktop), a mobile computing device (e.g., a smartphone or tablet), a game console or controller, a wearable computing device, an embedded computing device, a server computing device, a virtual machine running on a host device, or any other type of computing device. Computing device 50 may be a client computing device. Computing device 50 may be an end-user computing device. Computing device 50 may be a computing device that provides services to an end-user (the end-user can interact with computing device 50 using other computing devices).

[0157] The computing device 50 may include one or more processors 51 and memory 52. ​​The processor 51(s) may be any suitable processing device (e.g., a processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and may be one processor or multiple operably connected processors. The memory 52 may include one or more non-temporary computer-readable storage media such as HBM, RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 52 may store data 53 and instructions 54 that can be executed by the processor 51(s) to cause the computing device 50 to perform operations. The operation may implement any one or more features described herein. The operation may implement exemplary methods and techniques described herein.

[0158] Furthermore, the computing device 50 may include one or more input components that receive user input. For example, a user input component may be a touch-sensitive component (e.g., a touch-sensitive display screen or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or stylus). The touch-sensitive component may serve to implement a virtual keyboard. Other exemplary user input components include a microphone, a camera, LiDAR, a physical keyboard or other buttons, or other means by which the user can provide user input.

[0159] The computing device 50 may store or contain one or more machine learning models 55. The machine learning models 55 may include one or more machine learning models 1(or more), such as a sequence processing model 4. The machine learning models 55 may include one or more model instances 31(or more). The machine learning models 55(or more) can be received from a server computing system 60(or more), a model development platform system 70, a third-party system 80(or more) (e.g., an application distribution platform), or can be developed locally on the computing device 50. The machine learning models 55(or more) can be loaded into memory 52 and used by a processor 51(or more), or otherwise implemented. The computing device 50 can implement multiple parallel instances of the machine learning models 55(or more).

[0160] A server computing system 60(or more) may include one or more processors 61 and memory 62. The processors 61(or more) may be any suitable processing device (e.g., a processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and may be one processor or multiple operably connected processors. The memory 62 may include one or more non-temporary computer-readable storage media, such as HBM, RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, and combinations thereof. The memory 62 can store data 63 and instructions 64 that can be executed by the processors 61(or more) to cause the server computing system 60(or more) to perform operations. The operations may implement any one or more features described herein. The operations may implement exemplary methods and techniques described herein.

[0161] In some embodiments, the server computing system 60 includes one or more server computing devices, or is otherwise implemented by them. If the server computing system 60 includes multiple server computing devices, such server computing devices may operate in a sequential computing architecture, a parallel computing architecture, or any combination thereof.

[0162] The server computing system 60 can store, or otherwise include, one or more machine learning models 65. The machine learning models 65(or more) may be the same as or different from the machine learning models 55(or more). The machine learning models 65 may include one or more machine learning models 1(or more), such as the sequence processing model 4. The machine learning models 65 may include one or more model instances 31(or more). The machine learning models 65(or more) can be received from the computing device 50, the model development platform system 70, the third-party system 80(or more), or developed locally in the server computing system 60(or more). The machine learning models 65(or more) can be loaded into memory 62 and used by the processor 61(or more), or otherwise implemented. The server computing system 60(or more) can implement multiple parallel instances of the machine learning models 65(or more).

[0163] In an exemplary configuration, the machine learning model 65 can be contained within or otherwise stored and implemented by the server computing system 60 to establish a client-server relationship with the computing device 50 for providing model inference. For example, the server computing system 60(or more) can implement a model host 31 on the computing device 50 on behalf of the client 32(or more). For example, the machine learning model 65 may be implemented by the server computing system 60 as part of a web service (e.g., a remote machine learning model host service such as an online interface for performing machine learning model calculations over a network on the server computing system 60(or more)). For example, the server computing system 60(or more) can communicate with the computing device 50 via a local intranet or internet connection. For example, the computing device 50 may be a workstation or endpoint communicating with a server computing system 60(or more), and the implementation of the machine learning model 65 is managed by the server computing system 60(or more) to remotely perform inference (e.g., for runtime or training operations), and its output(or more) is returned to the computing device 50 (e.g., cast, streamed, etc.). The machine learning model 65 can work cooperatively or interactively with the machine learning model 55 on the computing device 50 to perform a variety of tasks.

[0164] A model development platform system 70 may include one or more processors 71 and a memory 72. The processor 71 may be any suitable processing device (e.g., a processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and may be one processor or multiple operably connected processors. The memory 72 may include one or more non-temporary computer-readable storage media, such as HBM, RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, and combinations thereof. The memory 72 can store data 73 and instructions 74 that can be executed by the processor 71 to cause the model development platform system 70 to perform operations. The operations may implement any one or more features described herein. The operations may implement exemplary methods and techniques described herein. Exemplary operations include the functions described herein with respect to the model development platform 12. These functions and other functions may be implemented by developer tools 75.

[0165] A third-party system 80 may include one or more processors 81 and memory 82. The processor 81 may be any suitable processing device (e.g., a processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and may be one processor or multiple operably connected processors. The memory 82 may include one or more non-temporary computer-readable storage media such as HBM, RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, and combinations thereof. The memory 82 can store data 83 and instructions 84 that are executed by the processor 81 to enable the third-party system 80 to perform operations. The operation may implement any one or more features described herein. The operation may implement exemplary methods and techniques described herein. Exemplary operations include the functions described herein with respect to tools and other external resources (e.g., third-party resources) that are invoked when training or inferring using machine learning models 1, 4, 16, 20, 55, 65, etc.

[0166] Figure 10 shows one exemplary configuration of a computing system that may be used to implement the present disclosure. Other computing system configurations can be used similarly. For example, in some embodiments, one or both of the computing system 50 or server computing system 60(or more) can perform all or part of the operation of the model development platform system 70. For example, the computing system 50 or server computing system 60(or more) can implement developer tools 75(or more) (or extensions thereof) to develop, update / train or improve machine learning models 1, 4, 16, 20, 55, 65, etc., using one or more techniques described herein with respect to the model alignment toolkit 17. In this way, for example, the computing system 50 or server computing system 60(or more) can develop, update / train or improve machine learning models based on local datasets (for example, for personalization / customization of the model, as permitted by user data preference selection).

[0167] Figure 11 is a block diagram of an exemplary computing device 98 implemented according to an exemplary embodiment of the present disclosure. The computing device 98 may be a user computing device or a server computing device (e.g., computing device 50, server computing system 60, etc.). The computing device 98 may implement a model host 31. For example, the computing device 98 may include multiple applications (e.g., applications 1 to N). Each application may include its own machine learning library and machine-trained model(s). For example, each application may include a machine-trained model. Exemplary applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. As shown in Figure 11, each application may communicate with several other components of the computing device, such as one or more sensors, a context manager, a device state component, or additional components. In some embodiments, each application may communicate with each device component using an API (e.g., a public API). In some embodiments, the API used by each application is specific to that application.

[0168] Figure 12 is a block diagram of an exemplary computing device 99 implemented according to an exemplary embodiment of the present disclosure. Computing device 99 may be the same as or different from computing device 98. Computing device 99 may be a user computing device or a server computing device (e.g., computing device 50, server computing system 60(or more)). Computing device 98 may implement a model host 31. For example, computing device 99 may include multiple applications (e.g., applications 1 to N). Each application may communicate with a central intelligence layer. Exemplary applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc. In some embodiments, each application may communicate with a central intelligence layer (and models(or more) stored therein) using an API (e.g., a common API across all applications).

[0169] The central intelligence layer may include multiple machine learning models. For example, as shown in Figure 12, each machine learning model may be provided for each application and managed by the central intelligence layer. In other embodiments, two or more applications may share a single machine learning model. For example, in some embodiments, the central intelligence layer may provide a single model to all applications. In some embodiments, the central intelligence layer may be contained within the operating system of the computing device 99 or otherwise implemented by the operating system of the computing device 99.

[0170] The central intelligence layer can communicate with the central device data layer. The central device data layer may be a centralized repository of data from the computing device 99. As shown in Figure 12, the central device data layer can communicate with some of the other components of the computing device, such as one or more sensors, a context manager, a device state component, or additional components. In some embodiments, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0171] Additional disclosures The technologies described herein refer to servers, databases, software applications, and other computer-based systems, as well as actions performed on such systems and information sent to and received from such systems. The inherent flexibility of computer-based systems allows for a wide variety of feasible configurations, combinations, and divisions of tasks and functions between components. For example, the processes described herein can be performed using a single device or component, or multiple devices or components working together. Databases and applications can be run on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0172] While the subject matter has been described in detail with respect to various specific exemplary embodiments, each example is provided for illustrative purposes only and does not limit the disclosure. Those skilled in the art, having attained the foregoing understanding, can readily produce modifications, variations, and equivalents to such embodiments. Therefore, the disclosure of the subject matter does not preclude the inclusion of such modifications, variations, or additions to the subject matter that would be readily apparent to those skilled in the art. For example, features illustrated or described as part of one embodiment may be used in other embodiments to produce yet another embodiment. Thus, the disclosure is intended to include such modifications, variations, and equivalents.

[0173] The aspects of this disclosure have been described in relation to their exemplary embodiments. Any and all of the following claims and features should not be construed as limiting the scope of possible combinations of features disclosed herein by the dependency of the exemplary claims enumerated herein, so that combinations can be combined or rearranged in any way, including combinations of claims not expressly enumerated. Accordingly, the scope of this disclosure is illustrative and not limiting, and the disclosure of this subject matter does not preclude such modifications, variations, or additions to the subject matter, as would be readily apparent to those skilled in the art. Furthermore, terms are described herein using lists of exemplary elements joined by conjunctions such as “and,” “or,” and “but.” It should be understood that such conjunctions are provided for illustrative purposes only. For example, clauses and other sets of items joined by certain conjunctions such as “or” may refer to “and / or,” “at least one of the exemplary elements enumerated therein,” “any combination of,” etc. Terms such as “based on” should be understood as “at least partially based on.”

[0174] The term "can" should be understood not as referring to a capability necessarily present in all embodiments, but rather as referring to the potential features in various embodiments. For example, the phrase "X can perform Y" should be understood as indicating that in various embodiments, X may be configured to perform Y, and not as indicating that X must always be able to perform Y in all examples. In various embodiments, X may remain within the scope of this disclosure even if it is unable to perform Y.

[0175] The term "may" should be understood not as referring to a capability necessarily present in all embodiments, but rather as referring to the possibility of features in various embodiments. For example, the phrase "X may perform Y" should be understood as indicating that in various embodiments, X may be configured to perform Y, and not as indicating that X must always be able to perform Y in all examples. It should be understood that in various embodiments, X may remain within the scope of this disclosure even if it is unable to perform Y.

Claims

1. A computer-implemented method for automating the use of a tool based on learned similarity, wherein the method is A computing system including one or more computing devices acquires data to describe a query, The computing system generates query embeddings for the query, wherein the query embeddings are represented in a latent embedding space. The computing system performs a similarity search on the query embeddings in the potential embedding space in order to identify one or more previously defined embeddings associated with one or more previously defined query response pairs. The computing system generates a prompt based on the query and one or more previously defined query-response pairs. The computing system provides the prompt as input for processing by a machine learning-trained language model, The computing system receives the model response to the query output by the machine learning language model based on the processing of the prompt. A method of implementation in a computer, including

2. The computer-implemented method according to claim 1, wherein the model response, when executed by the computing system, includes one or more tokens that cause a structural tool to acquire or transform information.

3. The computing system further includes processing the model response to generate a system response, Processing the model response includes executing one or more tokens to cause the structuring tool to retrieve or transform the information, The system response is at least partially based on the information obtained or transformed by the structuring tool in response to the execution of one or more tokens. A method implemented in a computer according to claim 2.

4. The computer implementation method according to claim 3, wherein the query includes a natural language question, and the system response includes a natural language answer to the natural language question.

5. A method implemented in a computer according to any one of claims 2 to 4, wherein the structural tool includes a programming language interpreter configured to execute instructions expressed in a computer language.

6. The computing system preloads one or more libraries or datasets associated with the computer language into the computer environment associated with the programming language interpreter. A method implemented in a computer according to claim 5, further comprising:

7. The computer language is the Python computer language, and the method is implemented on a computer according to claim 5 or 6.

8. The aforementioned structural tool is Database lookup to access additional information from the database, API calls to request and receive additional information via the Application Programming Interface (API), A query service that queries results from search engines, knowledge graphs, or digital assistants. A method implemented in a computer according to any one of claims 2 to 7, including the method described in any one of claims 2 to 7.

9. A method implemented in a computer according to any one of claims 1 to 8, wherein the computing system generating the prompt based on the query and the one or more previously defined query-response pairs includes concatenating the query with the one or more previously defined query-response pairs.

10. A method implemented in a computer according to any one of claims 1 to 9, wherein the computing system generates the prompt based on the query and the one or more previously defined query-response pairs, the method comprising ranking the one or more previously defined query-response pairs.

11. At least one of the previously defined query-response pairs demonstrates the use of a structure tool, Processing of the prompt by the machine learning language model includes the use of the structure tool, A method implemented in a computer according to any one of claims 1 to 10.

12. Before obtaining the data that describes the aforementioned query, The computing system generates the previously defined embedding for each of the one or more previously defined query-response pairs based at least partially on the query portion of the previously defined query-response pair. A computer-implemented method according to any one of claims 1 to 11, further comprising:

13. A computer-implemented method according to any one of claims 1 to 12, wherein the computing system performing the similarity search of the query embeddings within the potential embedding space includes identifying the top k sets of previously defined embeddings from the embedding space, where k is a hyperparameter.

14. The method implemented in a computer according to any one of claims 1 to 13, wherein the model response includes an executable program.

15. The method implemented on a computer according to any one of claims 1 to 14, wherein the machine learning language model includes a pre-trained large-scale language model.

16. A computer-implemented method according to any one of claims 1 to 15, further comprising fine-tuning the machine-learned language model with the aforementioned previously defined query-response pairs.

17. A method implemented in a computer according to any one of claims 1 to 16, wherein the computing system provides the prompt as input for processing by the machine-learned language model, the computing system executes the machine-learned language model to process the prompt.

18. A computer system configured to carry out the method described in any one of claims 1 to 17.

19. The computer system according to claim 18, wherein the computer system includes the machine learning language model.

20. One or more non-temporary computer-readable media for storing instructions for carrying out the method according to any one of claims 1 to 17.

Images