Efficient use of tools through language models

Abstract

A computer system is provided that enables a machine-learned language model to select the correct structural tool to be utilized when addressing challenging tasks. In particular, the current language model relies seriously on internal knowledge to resolve all downstream tasks, which results in hallucination and no-basis answers. In contrast, it is proposed herein a new tool use framework for a language model that enables the language model to intelligently route queries to the most relevant predefined skills.

Description

[0001] Related applications

[0002] This application claims priority and benefit to U.S. Provisional Patent Application No. 63 / 456,300, filed March 31, 2023. U.S. Provisional Patent Application No. 63 / 456,300 is hereby incorporated by reference in its entirety. Technical Field

[0003] This disclosure generally relates to the use of machine learning for language modeling. More specifically, this disclosure relates to a computational system that generates suggestions for responding to queries using structured tools to enable machine learning language models to efficiently use these suggestions. Background Technology

[0004] Natural Language Processing (NLP) has seen rapid development in recent years, largely thanks to learning-based algorithms and other improvements in machine or “neural” learning. A specific task within NLP is contextual text generation. In contextual text generation, the agent (e.g., a machine learning model) is tasked with generating output text from a given context. In some example approaches to contextual text generation, text-to-text models read the input context text and then directly produce the output text. An example of a contextual text generation task is a question-answering task, where the input context contains a question, and the expected output text is the answer to that question.

[0005] Current state-of-the-art contextual text generation models are often based on transformer-based neural models, some of which are commonly referred to as “large language models” (or LLMs). However, these so-called large language models suffer from several drawbacks. Specifically, while pre-trained large language models exhibit significant intelligence, their knowledge is limited only to information contained in (and learned from) their training dataset and / or information introduced within the contextual text input. Consequently, their knowledge of factual information is very limited and often stagnant in time. Therefore, when asked to produce outputs containing factual information, the model often hallucinates incorrect facts or supplies outdated information. Reliance on incorrect factual information can lead to inefficiencies, where incorrect actions (e.g., computational actions) are taken and require correction or other remediation, resulting in redundant and unnecessary use of resources (e.g., computational resources).

[0006] The inability of current language models to access external knowledge is just one example of a larger challenge associated with such models. Specifically, existing language models are largely unable to select, access, or utilize various structural tools, such as computational tools (e.g., calculators), information retrieval tools (e.g., stored data, search engines, knowledge graphs), and / or programming tools (e.g., compilers, programming language interpreters, etc.). Access to these tools would significantly improve a language model's ability to handle more diverse and complex queries. Summary of the Invention

[0007] Various aspects and advantages of embodiments of this disclosure will be set forth in part in the description which follows, or may be learned from the description or by practice of the embodiments.

[0008] One example aspect of this disclosure relates to a computer-implemented method for performing contextual text generation. The method includes obtaining data describing a query by a computing system including one or more computing devices. The method includes generating a query embedding for the query by the computing system, wherein the query embedding is expressed within a latent embedding space. The method includes performing a similarity search by the computing system within the latent embedding space for the query embedding to identify one or more previously defined embeddings associated with one or more previously defined query-response pairs. The method includes generating a prompt by the computing system based on the query and the one or more previously defined query-response pairs. The method includes providing the prompt by the computing system as input for processing by a machine learning language model. The method includes receiving a model response to the query output by the machine learning language model based on the processing of the prompt by the computing system.

[0009] Other aspects of this disclosure relate to various systems, devices, non-transitory computer-readable media, user interfaces, and electronic devices.

[0010] These and other features, aspects, and advantages of the various embodiments of this disclosure will be better understood with reference to the following description and the appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the disclosure and, together with the description, serve to illustrate the relevant principles. Attached Figure Description

[0011] Referring to the accompanying drawings, a detailed discussion of embodiments is set forth in this specification for those skilled in the art, in which:

[0012] Figure 1 A block diagram depicting an example data stream for generating a system response to a query, according to an example embodiment of the present disclosure.

[0013] Figure 2A graphical representation of an example process for generating a system response to a query, according to an example embodiment of the present disclosure.

[0014] Figure 3 This is a flowchart illustrating an example method for training a machine learning model according to an example implementation of an aspect of this disclosure;

[0015] Figure 4 This is a block diagram of an example processing flow for using a machine learning model to process input to generate output, based on an example implementation of aspects of this disclosure.

[0016] Figure 5 This is a block diagram of an example sequence processing model based on an example implementation of aspects of this disclosure;

[0017] Figure 6 This is a block diagram of an example technique for filling an example input sequence for processing by a sequence processing model, based on an example implementation of aspects of this disclosure;

[0018] Figure 7 This is a block diagram of an example model development platform based on an example implementation of aspects of this disclosure;

[0019] Figure 8 This is a block diagram of an example training workflow for training a machine learning model, based on an example implementation of aspects of this disclosure.

[0020] Figure 9 This is a block diagram of an inference system for performing inference by operating one or more machine learning models, based on an example implementation of aspects of this disclosure.

[0021] Figure 10 This is a block diagram of an example networked computing system based on an example implementation of aspects of this disclosure;

[0022] Figure 11 This is a block diagram of an example computing device implementing an aspect of this disclosure; and

[0023] Figure 12 This is a block diagram of an example computing device that implements aspects of this disclosure.

[0024] The repeated reference numerals across multiple figures are intended to identify the same features in various implementations. Detailed Implementation

[0025] SUMMARY

[0026] An exemplary aspect of this disclosure relates to a computer system that enables machine learning language models to select the correct structural tools to utilize when solving challenging tasks. Specifically, current language models heavily rely on internal knowledge to solve all downstream tasks, leading to hallucinations and unfounded answers. In contrast, this disclosure proposes a novel tool-use framework for language models that enables them to intelligently route queries to the most relevant predefined skills.

[0027] As an example, a computational system can generate an embedding cache associated with a set of previously defined query-response pairs. For instance, some or all of the previously defined query-response pairs can demonstrate tool usage in a service that generates answers to queries. Then, when a new query is received, the computational system can generate a query embedding for that query and perform a similarity search in the embedding space to identify and retrieve multiple (e.g., previous) queries. The most similar previously defined query-response pair.

[0028] These retrieved, previously defined query-response pairs can be combined with new queries to generate hints for a language model targeting machine learning. The language model can then process these hints to generate a model response. For example, the model response may include one or more lexical units that invoke the use of one or more structured tools, such as computational tools (e.g., calculators), information retrieval tools (e.g., stored data, search engines, knowledge graphs), and / or programming tools (e.g., compilers, programming language interpreters, etc.). The computational system can process the model response (e.g., by executing the structured tools invoked by the model response) to generate a system response in response to the query.

[0029] More specifically, access to and proper use of different structural tools can significantly improve the ability to reliably and efficiently solve challenging problems. Therefore, enabling language models to use tools is attractive for the following reasons: 1) the authenticity of language model completions is generally more trustworthy because the execution performed by tools is deterministic rather than purely probabilistic; 2) the size of language models can be potentially reduced because domain-specific knowledge is delegated to external tools; and 3) language models using tools are largely unaffected by out-of-distribution performance degradation because, by definition, they are less dependent on the specific inputs provided.

[0030] Therefore, this disclosure proposes a novel general framework that enables language models (e.g., existing pre-trained language models) to efficiently utilize external tools. In some implementations, the proposed framework may begin with the creation of a predefined set of query-response pairs. In some implementations, these query-response pairs may be referred to as “prototypes” and may correspond to or originate from accumulated question-and-answer pairs during validation error analysis.

[0031] Next, to enable the language model to execute routing functions for tool usage, the computational system implementing the proposed framework can embed each query in the prototype set using an embedding model (e.g., a pre-trained text embedding model). Then, when a new query is received, it can be similarly embedded using the same embedding model.

[0032] The computational system can generate hints to trigger language model completion by linking one or more re-ranked prototypes, where the problem is the top-ranked prototypes most similar to the newly received query. The language model can process this prompt to generate a model response. After language model completion, the computational system can execute the generated model response, where appropriate, with the support of a "toolbox" containing the tools (e.g., data, libraries, and functions) required to successfully execute the model response.

[0033] Example experiments demonstrate that the proposed framework significantly improves the language model's ability to correctly respond to more diverse and complex arrays of queries. Specifically, example experiments show that the proposed framework outperforms state-of-the-art methods on many different tasks requiring complex processing.

[0034] The systems and methods disclosed herein offer several technical effects and benefits. As an example of such effects and benefits, the proposed method enables language models to utilize structural tools to access supplementary information, such as additional factual information. Therefore, language models can invoke and use such structural tools to access supplementary information that may be up-to-date, factual, domain-specific, client-specific, or user-specific. This improves the knowledge available to the language model when formulating text output and further enhances the system's flexibility by enabling the incorporation of diverse information sources for various use cases.

[0035] Beyond improving the quality of model output, the proposed structural tools also result in savings in computational resources such as processor usage, memory usage, and network bandwidth. Specifically, the knowledge available to previous language models was limited only to information contained in (and learned from) their training datasets and / or introduced from contextual text input. Therefore, their knowledge of factual information was very limited and often stagnant in time. Consequently, when asked to produce outputs containing factual information, the models often misinterpreted facts or supplied outdated information. Thus, retraining the entire language model was necessary to keep it up-to-date with changing real-world facts, to port the language model to new domains or user information sets, or otherwise deploy the model in new situations involving new information. Retraining the language model required significant computational resources such as processor usage, memory usage, and network bandwidth.

[0036] However, the use of the structural tools proposed in this disclosure eliminates the need to retrain the model to keep the language model up-to-date with changing real-world facts, enabling portability to new domains or user information sets, or otherwise deployment in new situations involving new information. Instead, the model can access only supplementary information (e.g., via structural tools), which can be up-to-date, factual, domain-specific, client-specific, or user-specific. Therefore, the model can be easily ported to different domains, uses, users, etc., and / or can provide responses utilizing up-to-date factual information without requiring model retraining, thus significantly saving computational resources. By encoding the context in the form of intermediate analysis that can interact with (potential external) information sources, this process helps address technical constraints in the placement of information and / or functionality.

[0037] Similarly, another example of technical effectiveness stems from the model's ability to leverage information from external sources without needing to store (e.g., in the form of learned relationships) all the information required to respond to various inputs. Specifically, past methods required storing and using large models (e.g., on user devices with limited memory and / or battery availability) that were large enough (e.g., in terms of the number of parameters) to learn and store relationships between various inputs and outputs. In contrast, some example implementations of this disclosure enable "thin" models to exist on user devices or other mobile clients or browsers. These lightweight models can leverage various infrastructure tools (e.g., cloud services) to save on battery, computation, storage, updates, etc. Therefore, smaller models with access to infrastructure tools can achieve similar or better performance than large, self-contained models, thereby saving computational resources such as memory usage, network bandwidth, and energy consumption. More generally,

[0038] As another example, the proposed technique enables machine learning models to better select the tool to use from multiple available tools. Specifically, by providing helpful examples demonstrating the correct tool usage, the model can automatically select the appropriate tool from several different available tools. When the model is more confident, it can invoke a single tool instead of several different tools. This saves computational resources because it reduces the number of interactions between the model and different tools.

[0039] As another example, the proposed model demonstrates improved interpretability. For instance, the model responses generated by the model can be examined or inspected (e.g., before or after execution) to explain or understand how the final system response was generated in response to the contextual input. Improved interpretability can lead to more efficient use of computational resources such as processor usage and memory usage. For example, a lack of interpretability in language model output can lead to a lack of confidence in or reliance on the model output, resulting in unnecessary overhead or other efforts (e.g., computerized operations) to attempt to "repeatedly verify" the authenticity or usefulness of the model output. By improving interpretability, confidence in computerized systems can be improved. Specifically, the reliability of the system response can be examined and / or evaluated to establish the system's usability for a specific task.

[0040] Exemplary embodiments of this disclosure will now be discussed in further detail with reference to the accompanying drawings.

[0041] Example model and tool usage

[0042] This disclosure relates to a process in which a language model is prompted to mimic the workflow presented in few-shot samples using contextual demonstration samples, thereby enabling the language model to solve previously unseen tasks. Specifically, some example implementations include three main phases: (a) predefining and embedding a prototype for solving the task, (b) during inference, searching for the k most relevant questions in the prototype for each new query question and re-ranking them to inject prompts, and (c) executing the generated content with the support of a preloaded toolkit to obtain the final result.

[0043] Example accumulation of prototypes from errors

[0044] One aspect of this disclosure relates to the accumulation and use of demonstrative query-response pairs, which may be referred to as “prototypes.” In some implementations, to find prototypes that elicit optimal performance, for each of several evaluation tasks, the corresponding dataset can be divided into a test set and a validation set in a certain ratio (e.g., 4:1). Additional prototypes can then be accumulated by maintaining a “prototype pool” containing prototypes of incorrect answers in each validation round.

[0045] An example of accumulating prototypes from errors can be described as Algorithm 1:

[0046] Algorithm 1: Accumulating Prototypes from Errors

[0047]

[0048] In some implementations, the prototype pool can be initialized using randomly selected questions and answers. A stopping criterion can be established for accumulation, which is: using the updated prototype pool... After rounds of validation, did the accuracy on the validation set not increase? In one example, This serves as the "patience threshold".

[0049] Example routing tool usage hints

[0050] Given several prototypes in a prototype pool (e.g., as conditional inputs included in prompts), a pre-trained language model (LM) can solve some of the problems by mimicking the workflows presented in the prototypes. Example experiments demonstrate that the success rate of the model can be significantly improved by actively selecting and inputting prototypes with workflows similar to those required to answer the query question. Therefore, one aspect of this disclosure relates to a routing system for intelligently selecting appropriate prototypes for a given query question.

[0051] Specifically, before starting the evaluation of a task, some sample systems can first pre-embed all questions from a stored prototype using an embedding system, and store these embeddings in a cache file. Then, during evaluation, each new query question will first be embedded using the same embedding system. The computational system can then use the embeddings of the prototype questions to compute a distance metric (e.g., pairwise cosine similarity). This distance metric (e.g., cosine similarity) can be referred to as a relevance score. Finally, the computational system includes a certain number of (previous) questions in the prompts used to trigger LM generation. The most similar prototype.

[0052] Example inspection execution

[0053] Once a prompt prepared by the routing module is received, the LM can generate the query question by considering the workflow presented in the prototype included in the prompt.

[0054] In some example implementations, the answers provided in the prototype are mostly code, because the tool can be used simply by executing the code. Therefore, some example implementations may include configuring and using a "toolbox" for each task. For example, the toolbox could correspond to a global memory that can load data, code snippets, and / or random parameters to help execute the code generated by the LM.

[0055] As an example, the following items can be preloaded into the toolbox: dataset files (e.g., .csv files), solver classes implementing some desired functionality (e.g., the classic Dijkastra algorithm), and packages required for successful execution (e.g., pandas, numpy, etc.). This preloaded code can be executed once before the task evaluation begins. The generated code for each problem (e.g., the model response generated by LM, referred to as...) The computing system can run In order to obtain the tested results.

[0056] In some implementations, the computing system uses To extract the value stored in the answer variable to obtain the final answer (e.g., LM is expected to mimic the answer in the prototype, which also stores the answer in the variable in the final step). middle).

[0057] Example query response system

[0058] Figure 1 An example query response system 110 is described to implement the above concepts. Specifically, Figure 1 A block diagram depicting an example data stream for generating a system response 136 to query 112, according to an example embodiment of the present disclosure.

[0059] like Figure 1 As shown, the computing system can obtain data describing query 112. For example, query 112 can be received from a user. For example, the query could be a natural language question.

[0060] The computing system can use the embedding model 114 to generate a query embedding 116 for the query 112. For example, the query embedding 116 can be expressed within a latent embedding space.

[0061] The computational system can perform a similarity search 120 within the latent embedding space for the query embedding 116 to identify one or more previously defined embeddings 124 associated with one or more previously defined query-response pairs 118. For example, performing the similarity search 120 may include ranking the one or more previously defined query-response pairs 118. For example, the previously defined query-response pairs 118 may be ranked based on a distance metric (e.g., pairwise cosine similarity) evaluated between the embedding 122 for each pair 118 and the query embedding 116. In some implementations, performing the similarity search 120 may include identifying the previous embeddings of the previously defined embeddings 122 from the embedding space. One set, of which This is a hyperparameter. In other implementations, it can return all embeddings with a distance metric less than a certain threshold.

[0062] More specifically, the computing system can use the embedding model 114 to generate an embedding 122 for a previously defined query-response pair 118. For example, this process can be completed offline before the query 112 is received, and the embedding 122 can be stored in a database for use when the query 112 is received. For example, in some implementations, the embedding 122 can be generated only from the query portion of the previously defined query-response pair 118. According to aspects of this disclosure, some or all of the query-response pairs in the previously defined query-response pair 118 can demonstrate the use of one or more structural tools 138.

[0063] In some implementations, certain subsets of the previously defined query-response pairs 18 may be associated with different structural tools 138. For example, a first pair may be an example of use of a first tool, while a second pair may be an example of use of a second tool. The user associated with query 112 may have access to or be otherwise permitted to use some, but not all, of the structural tools 138. In some implementations, the similarity search 120 may be restricted to searching only for previously defined embeddings 124 generated from query-response pairs 118 associated with structural tools 138 that the user has access to or may use. Alternatively, the user may be able to select one or more structural tools 138 (e.g., via a user interface). In some implementations, the similarity search 120 may be restricted to searching only for previously defined embeddings 124 generated from query-response pairs 18 associated with one or more structural tools 138 selected by the user.

[0064] The computing system can generate a hint 128 based on query 112 and one or more previously defined query-response pairs 124 identified by similarity search 120 (e.g., by performing hint construction 126). As an example, the computing system can concatenate query 112 with one or more previously defined query-response pairs 124 identified by similarity search 120.

[0065] The computing system can provide the prompt 128 as input for processing by the machine learning language model 130. For example, the language model 130 can be a large, pre-trained language model, such as, for example, BERT, LaMDA, PaLM, etc. The computing system can receive the model response 132 to query 112, output by the machine learning language model 130 based on the processing of the prompt 128.

[0066] The computing system can execute response execution 134 to process the model response 132 to generate a system response 136. The system response 136 can be provided to the user as output. For example, the system response 136 could be a natural language answer to a natural language question.

[0067] According to aspects of this disclosure, in some implementations, model response 132 may include one or more lexical units that, when executed by a computational system (e.g., shown as response execution 134), cause structuring tool 138 to retrieve or transform information. Therefore, in some implementations, executing response execution 134 to process model response 132 may include executing one or more lexical units to cause structuring tool 138 to retrieve or transform the information. System response 136 may be based at least in part on the information retrieved or transformed by structuring tool 138 in response to the execution of one or more lexical units.

[0068] In some implementations, a toolbox 140 may be provided for use by the structural tool 138. For example, the toolbox 140 may include any knowledge, data, or information that enables the structural tool to retrieve or transform information as requested or instructed by the model in response to 132.

[0069] To provide an illustrative example, in some implementations, model response 132 may include instructions expressed in a computer language (e.g., an executable computer program), and structural tool 138 may be a programming language interpreter configured to execute instructions expressed in a computer language. In some such implementations, toolbox 140 may include pre-loaded information, such as one or more libraries or datasets associated with a computer language. One or more libraries or datasets may be provided in a shared computing environment with a programming language interpreter. As an example, the computer language may be the Python computer language.

[0070] As other examples, structure tool 138 may be, include, or utilize: a database lookup for accessing additional information from a database; an API call for requesting and receiving additional information via an application programming interface (API); a query service for querying results from a search engine, knowledge graph, or digital assistant; a calculator tool; or other computing tools.

[0071] In some implementations, the language model 130 can optionally be fine-tuned on a previously defined query-response pair 118 before using the machine learning model 130, such as... Figure 1 As shown.

[0072] Example illustration

[0073] Figure 2 A graphical representation depicts a specific example process for generating a system response to a query according to an example embodiment of this disclosure. As shown in section (a), the computational system may first accumulate a set of question-answer pairs (referred to as a “prototype pool”) from validation errors. As shown in section (b), the computational system may embed questions from the prototype pool, and each time a new question is queried in the LM, the computational system searches for the most similar question in the review manual. The question re-ranks the corresponding prototypes and links them together as a hint injection. As shown in section (c), the hint-triggered LM generation can be executed in an environment (e.g., via a Python interpreter) that preloads data, libraries, code snippets, etc., that can support a particular task. Finally, the answer can be collected from runtime variables.

[0074] More specifically, Figure 2 A graphical diagram illustrating an example process for generating a system response to a query according to an example embodiment of the present disclosure. Figure 2 The process described illustrates the interaction between a predefined query-response pair (called a prototype) and a new user query, which prompts the generation of a system response using a language model and resource toolkit developed through machine learning.

[0075] In Phase 1(a), the process involves selecting relevant prototypes (e.g., Prototype 1, Prototype 4, and Prototype 8) based on a relevance score indicating the degree of similarity to a new user query. Each prototype contains a question and an answer, where the answer can be in the form of non-executable natural language and / or an executable program. For example, Prototype 4, with a relevance score of 0.6, provides an answer in executable code to determine the number of flights operated by Delta Air Lines, while Prototype 1, with a higher relevance score of 0.9, demonstrates using the DijkstraSolver class to find the cheapest flight under specific constraints.

[0076] Phase 2(b) involves combining new user queries with re-ranked prototypes to form hints that are injected into the language model. The example user query expresses the expectation of finding the lowest price for a flight from SFO to JFK, preferably Delta Airlines, with a maximum of two layovers. Re-ranked prototypes are selected to help the language model understand the context and requirements of the query by providing examples of similar problems and their solutions.

[0077] In the final stage (c), the language model processes new queries and combined hints from the selected prototype to compose a model response. This response leverages the composition and planning skills demonstrated in the prototype, such as filtering a dataset of Delta Airlines flights and using planning algorithms to find the cheapest flight option within given constraints. The model response, including executable code, is then run using a toolkit for flight booking. This toolkit contains necessary resources such as the flight dataset, internal code snippets like the DijkstraSolver class, and external libraries such as pandas. Execution produces a system response with the lowest flight price, thus confirming the successful application of the method and tools in resolving user queries.

[0078] This interactive execution process serves as an example embodiment of the proposed technology, demonstrating how a computing system can efficiently utilize structural tools to generate accurate and up-to-date responses to complex queries by leveraging language models from machine learning and predefined query-response prototypes.

[0079] Example method

[0080] Figure 3 A flowchart depicts a method 300 for training one or more machine learning models according to aspects of this disclosure. For example, an example machine learning model may include {references to the claimed model}.

[0081] One or more portions of Example Method 300 may be implemented by a computing system (such as a computing system described, for example, with reference to other diagrams) including one or more computing devices. Each corresponding portion of Example Method 300 may be executed by any one (or any combination) of one or more computing devices. Furthermore, one or more portions of Example Method 300 may be implemented on the hardware components of the apparatus described herein, for example, to train one or more systems or models. Figure 3 For illustrative and discussion purposes, elements executed in a specific order are depicted. Those skilled in the art will understand using the disclosure provided herein that elements of any of the methods discussed herein can be adapted, rearranged, extended, omitted, combined, or modified in various ways without departing from the scope of this disclosure. Figure 3The description is for illustrative purposes only and with reference to elements / terms described with respect to other systems and figures, and is not intended to be limiting. One or more portions of example method 300 may be performed additionally or alternatively by other systems.

[0082] At 302, example method 300 may include obtaining training instances. The training dataset may include multiple training instances partitioned across multiple datasets (e.g., training datasets, validation datasets, or test datasets). Training instances may be labeled or unlabeled. Although referred to as "training" instances in example method 300, it should be understood that runtime inference may form training instances when a model is trained (e.g., online training / learning) using an evaluation of the model's performance on that runtime instance. Example data types of training instances and various tasks associated with them are described throughout this disclosure.

[0083] At 304, example method 300 may include using one or more machine learning models to process training instances to generate output. This output may be obtained directly from the one or more machine learning models, or it may be a downstream result of a processing operation chain that includes the outputs of the one or more machine learning models.

[0084] At 306, example method 300 may include receiving an evaluation signal associated with the output. The evaluation signal can be obtained using a loss function. Various losses can be used, such as mean squared error, likelihood loss, cross-entropy loss, hinge loss, contrastive loss, or various other loss functions. The evaluation signal can be computed using known ground truth labels (e.g., supervised learning), predicted or estimated labels (e.g., semi-supervised or self-supervised learning), or unlabeled labels (e.g., unsupervised learning). The evaluation signal can be a reward (e.g., for reinforcement learning). The reward can be computed using a machine learning reward model configured to generate a reward based on the received output. The reward can also be computed using feedback data describing human feedback to the output.

[0085] At 308, example method 300 may include using an evaluation signal to update a machine learning model. For example, in some embodiments, various training or learning techniques (such as, for example, backpropagation) may be used to learn the values ​​of the parameters of the machine learning model. For example, the evaluation signal may be backpropagated from the output (or another source of the evaluation signal) through the machine learning model to update one or more parameters of the model (e.g., based on the gradient of the evaluation signal relative to the parameter values). For example, a system containing one or more machine learning models may be trained in an end-to-end manner. Gradient descent techniques may be used to iteratively update the parameters over multiple training iterations. In some implementations, performing error backpropagation may include performing truncated backpropagation over time. Example method 300 may include implementing various generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization ability of the model being trained.

[0086] In some implementations, example method 300 can be implemented to train a machine learning model from an initial state to a fully trained state (e.g., when the model exhibits a desired performance profile, such as based on accuracy, precision, recall, etc.).

[0087] In some implementations, example method 300 can be implemented for specific stages of the training process. For example, in some implementations, example method 300 can be implemented for pre-training a machine learning model. Pre-training can include, for example, large-scale training on potentially noisy data to achieve a broad performance level base across multiple tasks / data types. In some implementations, example method 300 can be implemented for fine-tuning a machine learning model. Fine-tuning can include, for example, smaller-scale training on higher quality (e.g., labeled, curated, etc.) data. Fine-tuning can affect all or part of the parameters of the machine learning model. For example, parts of the machine learning model can be “frozen” at certain training stages. For example, parameters associated with the embedding space can be “frozen” during fine-tuning (e.g., to preserve information learned from a broader domain than that present in the fine-tuned dataset). Example fine-tuning methods include reinforcement learning. Reinforcement learning can be based on user feedback on model performance during use.

[0088] Example machine learning model

[0089] Figure 4 This is a block diagram of an example processing flow for using machine learning model 1 to process input 2 to generate output 3.

[0090] The machine learning model 1 can be or includes one or more machine learning models or model components. Example machine learning models can include neural networks (e.g., deep neural networks). Example machine learning models can include non-linear or linear models. Example machine learning models can use other architectures to replace or attach to neural networks. Example machine learning models can include decision tree-based models, support vector machines, hidden Markov models, Bayesian networks, linear regression models, k-means clustering models, etc.

[0091] Example neural networks can include feedforward neural networks, recurrent neural networks (RNNs) (including long short-term memory (LSTM) based RNNs), convolutional neural networks (CNNs), diffusion models, generative adversarial networks, or other forms of neural networks. Example neural networks can also be deep neural networks. Some example machine learning models may utilize attention mechanisms, such as self-attention. For example, some example machine learning models may include multi-head self-attention models.

[0092] Machine learning model 1 may include one or more instances of the same model configured to operate on data from input 2. Machine learning model 1 may also include a collection of different models that can interact collaboratively to process data from input 2. For example, machine learning model 1 may employ a hybrid expert structure. See, for example, Zhou et al., Mixture-of-Experts with Expert Choice Routing, arXiv:2202.09368v2 (October 14, 2022).

[0093] Input 2 can typically include or otherwise represent various types of data. Input 2 can include one type or many different types of data. Output 3 can be data of the same type as input 2 or data of a different type. Output 3 can include one type or many different types of data.

[0094] Example data types for input 2 or output 3 include natural language text data, software code data (e.g., source code, object code, machine code, or any other form of computer-readable instruction or programming language), machine code data (e.g., binary code, assembly code, or other forms of machine-readable instruction that can be executed directly by a computer's central processing unit), assembly code data (e.g., a low-level programming language that uses a symbolic representation of machine code instructions to program the processing unit), genetic data or other chemical or biochemical data, image data, audio data, audiovisual data, tactile data, biometric data, medical data, financial data, statistical data, geographic data, astronomical data, historical data, and generally sensor data (e.g., digital or analog values, such as voltage or other absolute or relative level measurements from real or artificial inputs, such as from audio sensors, light sensors, displacement sensors, etc.). Data can be raw or processed and can be in any format or mode.

[0095] In multimodal input 2 or output 3, example combinations of data types include image data and audio data, image data and natural language data, natural language data and software code data, image data and biometric data, sensor data and medical data, etc. It should be understood that any combination of data types in input 2 or output 3 can exist.

[0096] Example input 2 may include one or more data types, such as the example data types indicated above. Example output 3 may include one or more data types, such as the example data types indicated above. The data type of input 2 may be the same as or different from the data type of output 3. It should be understood that the example data types indicated above are provided for illustrative purposes only. The data types contemplated within the scope of this disclosure are not limited to those examples indicated above.

[0097] Example machine learning sequence processing model

[0098] Figure 5This is a block diagram illustrating an example implementation of an example machine learning model configured to process information sequences. For example, an example implementation of machine learning model 1 could include a sequence processing model 4. The example system could pass input 2 to sequence processing model 4. Sequence processing model 4 could include one or more machine learning components. Sequence processing model 4 could process the data from input 2 to obtain input sequence 5. Input sequence 5 could include one or more input elements 5-1, 5-2, ..., 5-M, etc., obtained from input 2. Sequence processing model 4 could use a prediction layer 6 to process input sequence 5 to generate output sequence 7. Output sequence 7 could include one or more output elements 7-1, 7-2, ..., 7-N, etc., generated based on input sequence 5. The system could generate output 3 based on output sequence 7.

[0099] Sequence processing models 4 can include one or more machine learning model components configured to ingest, generate, or otherwise infer sequences of information. For example, some example sequence processing models in the text domain are referred to as “large language models” or LLMs. See, for example, the PaLM 2 technology report, Google, https: / / ai.google / static / documents / palm2techreport.pdf (nd). Other example sequence processing models can operate in other domains, such as the image domain (see, for example, Dosovitskiy et al., An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, arXiv:2010.11929v2 (June 3, 2021); the audio domain (see, for example, Agostinelli et al., MusicLM: Generating Music From Text, arXiv:2301.11325v1 (January 26, 2023); and the biochemical domain (see, for example, Jumper et al., Highly accurate protein structure prediction with AlphaFold, 596 Nature 583 (August 26, 2021)). Sequence processing model 4 can process one or more types of data simultaneously. Sequence processing model 4 can include relatively large models (e.g., more parameters, computationally intensive, etc.), relatively small models (e.g., fewer parameters, computationally lightweight, etc.), or both.

[0100] Generally, sequence processing model 4 can use data from input 2 to obtain input sequence 5. For example, input sequence 5 may include a representation of the data from input 2 in a format understood by sequence processing model 4. One or more machine learning components of sequence processing model 4 may ingest data from input 2, parse the data into fragments compatible with the processing architecture of sequence processing model 4 (e.g., via "lexicalization"), and project the fragments into the input space associated with prediction layer 6 (e.g., via "embedding").

[0101] Sequence processing model 4 can ingest data from input 2 and parse the data into a sequence of elements to obtain input sequence 5. For example, a portion of the input data from input 2 can be decomposed into segments, which together represent the content of that portion of the input data. The segments can provide the elements of the sequence.

[0102] In some cases, elements 5-1, 5-2, ..., 5-M can represent building blocks used to capture or express meaningful information in a specific data domain. For example, elements can describe "atomic units" across one or more domains. For instance, for a text input source, elements can correspond to a group of one or more word or sub-word components (such as a set of one or more characters).

[0103] For example, elements 5-1, 5-2, ..., 5-M can represent tokens obtained using a tokenizer. For instance, a tokenizer can process a given portion of the input source and output a series of tokens representing that portion of the input source (e.g., corresponding to the input elements 5-1, 5-2, ..., 5-M). Various methods can be used for tokenization. For example, byte-pair encoding (BPE) techniques can be used to tokenize text input sources. See, for example, Kudo et al., *SentencePiece: A simple and language-independent subword tokenizer and detokenizer for Neural Text Processing*, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (System Demonstrations), pp. 66–71 (October 31–November 4, 2018), https: / / aclanthology.org / D18-2012.pdf. Lexicalization of image-based input sources can be achieved by extracting and serializing patches from images.

[0104] Generally, any data type can be serialized and processed into the input sequence 5. It should be understood that... Figure 5 The elements 5-1, 5-2, ..., 5-M described in the text can be lexical units or their embedded representations.

[0105] Prediction layer 6 can predict one or more output elements 7-1, 7-2, ..., 7-N based on the input elements. Prediction layer 6 may include one or more machine learning model architectures, such as one or more learned parameter layers that manipulate and transform the input to extract higher-order meaning from the input elements 5-1, 5-2, ..., 5-M and the relationships between the input elements. For example, in this way, example prediction layer 6 can predict new output elements given the context provided by the input sequence 5.

[0106] Prediction layer 6 can evaluate the associations between parts of the input sequence 5 and specific output elements. These associations can inform the prediction of the probability of a specific output occurring after the input context. For example, consider the text fragment “The carpenter's toolbox was small and heavy. It was full of ___.” Example prediction layer 6 can identify that “It” refers to “toolbox” by determining the relationships between the corresponding embeddings. Example prediction layer 6 can also link “It” to attributes of the toolbox, such as “small” and “heavy”. Based on these associations, prediction layer 6 can, for example, assign a higher probability to the word “nails” than to the word “sawdust”.

[0107] Transformers are example architectures that can be used in prediction layer 4. See, for example, Vaswani et al., AttentionIsAllYouNeed, arXiv:1706.03762v7 (August 2, 2023). Transformers are examples of machine learning model architectures that use attention mechanisms to compute associations between items within a context window. The context window can include a sequence containing an input sequence 5 and potentially one or more output elements 7-1, 7-2, ..., 7-N. A transformer block can include one or more attention layers and one or more post-attention layers (e.g., feedforward layers, such as multilayer perceptrons).

[0108] Prediction layer 6 can include other machine learning model architectures that are attached to or replace the transformer-based architecture. For example, recurrent neural networks (RNNs) and long short-term memory (LSTM) models, as well as convolutional neural networks (CNNs) can also be used. In general, prediction layer 6 can utilize various kinds of artificial neural networks that are capable of understanding or generating sequences of information.

[0109] The output sequence 7 may include or otherwise represent the same or different data type as the input sequence 5. For example, the input sequence 5 may represent text data, and the output sequence 7 may represent text data. The input sequence 5 may represent image, audio, or audiovisual data, and the output sequence 7 may represent text data (e.g., describing image, audio, or audiovisual data). It should be understood that the prediction layer 6 and any other gap model components of the sequence processing model 4 can be configured to receive multiple data types from the input sequence 5 and output multiple data types from the output sequence 7.

[0110] Output sequence 7 can have various relationships with input sequence 5. Output sequence 7 can be a continuation of input sequence 5. Output sequence 7 can be complementary to input sequence 5. Output sequence 7 can transform, alter, enhance, or otherwise modify input sequence 5. Output sequence 7 can answer, evaluate, acknowledge, or otherwise respond to input sequence 5. Output sequence 7 can implement (or describe instructions for implementation) the instructions provided via input sequence 5.

[0111] Output sequence 7 can be generated autoregressively. For example, for some applications, the output of one or more prediction layers 6 can be passed through one or more output layers (e.g., softmax layers) to obtain a probability distribution of an output vocabulary (e.g., a text or symbol vocabulary) conditioned on the set of input elements in the context window. In this way, output sequence 7 can be generated autoregressively by sampling possible next output elements, adding that element to the context window, regenerating the probability distribution based on the updated context window, and sampling possible next output elements.

[0112] Output sequence 7 can also be generated non-autoregressively. For example, multiple output elements of output sequence 7 can be predicted together without explicit order conditions. See, for example, Saharia et al., Non-Autoregressive Machine Translation with Latent Alignments, arXiv:2004.07437v3 (November 16, 2020).

[0113] Output sequence 7 may include one or more parts or elements. In the example content generation configuration, output sequence 7 may include multiple elements corresponding to multiple parts of the generated output sequence (e.g., text sentences, values ​​of discrete waveforms, computer code, etc.). In the example classification configuration, output sequence 7 may include a single element associated with the classification output. For example, the output "vocabulary" may include the set of classes to which the input sequence will be classified. For example, a vision transformer block may pass latent state information to a multilayer perceptron, which outputs possible class values ​​associated with the input image.

[0114] Figure 6 This is a block diagram of an example technique for populating example input sequence 8. Input sequence 8 may include various functional elements forming part of the model infrastructure, such as element 8-0 obtained from task indicator 9, which signals to any model processing input sequence 8 that a specific task is being performed (e.g., helping to adapt the model's performance to that specific task). Input sequence 8 may include various data elements from different data modalities. For example, input modality 10-1 may include one data modality. Data-to-sequence model 11-1 may process the data from input modality 10-1 to project the data into a format compatible with input sequence 8 (e.g., determining one or more vectors of dimensions based on the dimensions of input sequence 8) to obtain elements 8-1, 8-2, 8-3. Another input modality 10-2 may include a different data modality. Data-to-sequence model 11-2 may project the data from input modality 10-2 into a format compatible with input sequence 8 to obtain elements 8-4, 8-5, 8-6. Another input modality 10-3 may include yet another different data modality. The data to sequence model 11-3 can project the data from the input mode 10-3 into a format compatible with the input sequence 8 to obtain elements 8-7, 8-8, and 8-9.

[0115] Input sequence 8 may be the same as or different from input sequence 5. Input sequence 8 can be a multimodal input sequence containing elements representing data from different modalities using a common dimensional representation. For example, the embedding space may have P dimensions. Input sequence 8 can be configured to contain multiple elements with P dimensions. For example, in this way, example implementations can facilitate information extraction and reasoning across diverse data modalities by projecting data onto elements in the same embedding space for comparison, combination, or other computations between them.

[0116] For example, elements 8-0, ..., 8-9 can indicate specific locations within a multidimensional embedding space. Some elements can map to a discrete set of locations in the embedding space. For example, elements corresponding to discrete members in a predefined lexicon can map to discrete locations in the embedding space associated with those lexicons. Other elements can be continuously distributed across the embedding space. For example, some data types can be decomposed into continuously defined parts (e.g., image patches), which can be described using continuously distributed locations within the embedding space.

[0117] In some implementations, the expressive power of the embedding space may not be limited to meanings associated with any particular set of lexical units or other building blocks. For example, a contiguous embedding space can encode a series of higher-order information. A single piece of information (e.g., a lexical unit) can be mapped to a specific point in the space: for example, the lexical unit of the word "dog" can be projected to an embedding value that points to a specific location in the embedding space associated with canine-related information. Similarly, an image patch of a dog on grass can be projected into the embedding space. In some implementations, the projection of the image of a dog can be similar to the projection of the word "dog," and also similar to the projection of the word "grass," but potentially different from both. In some implementations, the projection of the image patch may not be perfectly aligned with any single projection of a single word. In some implementations, the projection of the image patch can be aligned with a combination of the projections of the words "dog" and "grass." For example, in this way, a higher-order embedding space can encode information independent of the data modalities in which it expresses information.

[0118] Task indicator 9 may include a model or model component configured to identify the task being performed and inject the input value represented by element 8-0 into input sequence 8, which signals which task is being performed. For example, the input value may be provided as a data type associated with an input modality and projected together with that modality (e.g., the input value may be a text task label embedded along with other text data in the input; the input value may be a pixel-based task representation embedded along with other image data in the input; etc.). The input value may be provided as a data type different from or at least independent of other inputs. For example, the input value represented by element 8-0 may be learned within a continuous embedding space.

[0119] Input modes 10⁻¹, 10⁻², and 10⁻³ can be associated with a variety of different data types (e.g., as described above with respect to input 2 and output 3).

[0120] Data-to-sequence models 11-1, 11-2, and 11-3 can be the same as or different from each other. Data-to-sequence models 11-1, 11-2, and 11-3 can be adapted to each corresponding input modality 10-1, 10-2, and 10-3. For example, a text data-to-sequence model can subdivide a portion of the input text and project these subdivisions onto elements in input sequence 8 (e.g., elements 8-1, 8-2, 8-3, etc.). An image data-to-sequence model can subdivide the input image and project these subdivisions onto elements in input sequence 8 (e.g., elements 8-4, 8-5, 8-6, etc.). A data-to-sequence model of any data type can subdivide the input of that arbitrary data type and project these subdivisions onto elements in input sequence 8 (e.g., elements 8-7, 8-8, 8-9, etc.).

[0121] Data-to-sequence models 11-1, 11-2, and 11-3 can form part of the machine learning sequence processing model 4. Data-to-sequence models 11-1, 11-2, and 11-3 can be trained jointly with the machine learning sequence processing model 4 or independently of it. Data-to-sequence models 11-1, 11-2, and 11-3 can be trained end-to-end with the machine learning sequence processing model 4.

[0122] Example machine learning model development platform

[0123] Figure 7 This is a block diagram of an example model development platform 12, which facilitates the creation, adaptation, and refinement of example machine learning models (e.g., machine learning model 1, sequence processing model 4, etc.). The model development platform 12 can provide several different toolkits that developer systems can use to develop new or adapted machine learning models.

[0124] The model development platform 12 can provide one or more model libraries 13 containing building blocks for new models. Model libraries 13 can include one or more pre-trained base models 13-1, which can provide support for processing capabilities across a variety of tasks. Model libraries 13 can include one or more pre-trained expert models 13-2, which can focus on performance in specific professional domains. Model libraries 13 can include various model primitives 13-3, which can provide low-level architectures or components (optionally pre-trained), which can be assembled in various arrangements as needed.

[0125] The model development platform 12 can receive selections of various model components 14. The model development platform 12 can transfer the selected model components 14 to the workbench 15, which combines the selected model components 14 into a development model 16.

[0126] Workbench 15 can facilitate further refinement and adaptation of the development model 16 by utilizing multiple different toolkits integrated with the model development platform 12. For example, workbench 15 can facilitate the use of model alignment toolkit 17 to align the development model 16 with expected performance profiles for various tasks.

[0127] The model alignment toolkit 17 can provide a variety of tools to cause the development model 16 to generate outputs aligned with desired behavioral characteristics. Alignment can include improving the accuracy, precision, recall, etc., of the model output. Alignment can include enforcing output styles, patterns, or other preferred characteristics of the model output. Alignment can be general or domain-specific. For example, the pre-trained base model 13-1 can start from initial performance levels across multiple domains. Alignment of the pre-trained base model 13-1 can include improving the performance of a specific information or task domain (e.g., even at the expense of the performance of another information or task domain).

[0128] The model alignment toolkit 17 can integrate one or more datasets 17-1 used to align development models 16. Selected datasets 17-1 may include labeled or unlabeled training data. Datasets 17-1 can be obtained from public domain datasets. Datasets 17-1 may also be obtained from private datasets associated with one or more developer systems for aligning custom machine learning models tailored to private use cases.

[0129] The pre-training pipeline 17-2 may include a machine learning model training workflow configured to update the development model 16 on a large-scale, potentially noisy dataset. For example, pre-training may utilize unsupervised learning techniques (e.g., denoising, etc.) to process a large number of training instances to update model parameters from an initial state and achieve the desired baseline performance. The pre-training pipeline 17-2 may utilize the unlabeled dataset from dataset 17-1 to perform pre-training. Workbench 15 may implement the pre-training pipeline 17-2 to pre-train the development model 16.

[0130] Fine-tuning pipeline 17-3 may include a machine learning model training workflow configured to refine the model parameters of development model 16 using higher-quality data. Fine-tuning pipeline 17-3 can update development model 16 through supervised training using the labeled dataset from dataset 17-1. Fine-tuning pipeline 17-3 can also update development model 16 through reinforcement learning using reward signals from user feedback. Workbench 15 enables fine-tuning pipeline 17-3 to fine-tune development model 16.

[0131] Hint library 17-4 may include a set of inputs configured to induce behavior aligned with desired performance criteria. Hint library 17-4 may include few-shot hints (e.g., inputs that provide examples of the desired model output for addition to the desired runtime query), chain-of-thought hints (e.g., inputs that provide step-by-step reasoning within a paradigm to facilitate thorough reasoning by the model), and so on.

[0132] Sample hints can be retrieved from the available repository in Hint Library 17-4. One or more developer systems can contribute sample hints using Workbench 15.

[0133] In some implementations, pre-trained or fine-tuned models can achieve satisfactory performance even when there are no paradigms in the input. For example, zero-shot hints can include inputs lacking paradigms. Zero-shot hints can be within the domain of the training dataset or outside the training domain.

[0134] Hint library 17-4 may include one or more hint engineering tools. Hint engineering tools can provide a workflow for retrieving or learning optimized hint values. Hint engineering tools can facilitate the direct learning of hint values ​​(e.g., input element values) based on one or more training iterations. Workbench 15 can implement the hint engineering tools within development model 16.

[0135] Hint library 17-4 may include a pipeline for hint generation. For example, input can be generated using development model 16 itself or other machine learning models. In this way, for example, a first model can process information about the task and output input for a second model to process in order to perform steps of the task. The second model may be the same as or different from the first model. Workbench 15 can implement the hint generation pipeline within development model 16.

[0136] Hint library 17-4 may include a pipeline for context injection. For example, if additional context is provided for performing a specific task, the performance of development model 16 on that task can be improved. Hint library 17-4 may include software components configured to identify desired context, retrieve context from external sources (e.g., databases, sensors, etc.), and add the context to input hints. Workbench 15 can implement the context injection pipeline in development model 16.

[0137] Although the various training examples described herein regarding model development platform 12 involve "pre-training" and "fine-tuning," it should be understood that the model alignment toolkit 17 generally supports a wide range of training techniques suitable for training a broad variety of machine learning models. Example training techniques may correspond to the example training methods 300 described above.

[0138] The model development platform 12 may include a model plug-in toolkit 18. The model plug-in toolkit 18 may include a variety of tools configured to enhance the functionality of machine learning models by integrating them with other systems, devices, and software components. For example, machine learning models can use tools to improve performance quality where appropriate. For instance, deterministic tasks can be offloaded to dedicated tools instead of performing tasks probabilistically when the risk of error increases. For example, instead of autoregressively predicting solutions to a system of equations, the machine learning model can identify the tools invoked to obtain solutions and pass the system of equations to the appropriate tool. This tool may be a conventional equation solver that operates deterministically to solve the system of equations. The tool's output may be returned in response to the original query. In this way, the tool uses tools that allow some example models to focus on the strengths of the machine learning model—e.g., understanding the intent in unstructured requests for a task—while enhancing model performance by offloading certain tasks to more focused tools to mechanically apply deterministic algorithms to well-defined problems.

[0139] The model plugin toolkit 18 may include a validation tool 18-1. The validation tool 18-1 may include tools that can parse and validate the output of a machine learning model. The validation tool 18-1 may include engineered heuristics that establish certain thresholds applied to the model output. For example, the validation tool 18-1 may base the output of the machine learning model on a structured data source (ground) (e.g., to mitigate "illusion").

[0140] The model plugin toolkit 18 may include a toolkit 18-2 for implementing one or more tools, which may include scripts or other executable code that can be executed with the development model 16. The toolkit 18-2 may include one or more inputs to the model implementation tools configured to cause machine learning (e.g., few-shot hints that induce the model to output tool calls with correct syntax). For example, the toolkit 18-2 may include, for instance, fine-tuned training data for training the model to use the tools.

[0141] The model plugin toolkit 18 may include interfaces for calling external application programming interfaces (APIs) 18-3. For example, attached to or replacing the direct implementation of tool calls or tool code using development model 16, development model 16 may be aligned with output instructions that initiate API calls to send or retrieve data via external systems.

[0142] The model plugin toolkit 18 can be integrated with the hint library 17-4 to build a catalog of available tools for use with the development model 16. For example, the model can receive a catalog of available tools as input, and the model can generate output that selects a tool from the available tools and initiates a tool call for using said tool.

[0143] Model development platform 12 may include a suite of computational optimization tools 19 for optimizing the computational performance of development model 16. For example, tools for model compression 19-1 may allow development model 16 to be reduced in size while maintaining the desired performance level. For example, model compression 19-1 may include quantization workflows, weight pruning, and sparsity techniques. Tools for hardware acceleration 19-2 may facilitate the configuration of model storage and execution formats for optimal operation on different hardware resources. For example, hardware acceleration 19-2 may include tools for optimally sharding the model for distributed processing across multiple processing units to increase bandwidth, reduce uniform memory requirements, etc. Tools for distillation 19-3 may provide training on a lighter model based on knowledge encoded in development model 16. For example, development model 16 may be a high-performance, large-scale machine learning model optimized using model development platform 12. To obtain a lightweight model for operation in resource-constrained environments, the smaller model may be a “student model” that learns from and imitates development model 16 as the “teacher model.” In this way, for example, the investment in learning and developing the parameters and configuration of model 16 can be efficiently transferred to a smaller model for more efficient inference.

[0144] Workbench 15 may implement one or more of the toolkits implemented in model development platform 12, or may not implement any toolkits. Workbench 15 may output output model 20 based on development model 16. Output model 20 may be a deployment version of development model 16. Output model 20 may be a development or training checkpoint of development model 16. Output model 20 may be a distilled version, compressed version, or otherwise optimized version of development model 16.

[0145] Figure 8 This is a block diagram of an example training process for training a development model 16 for machine learning. One or more parts of the example training process can be implemented by a computing system (such as the computing system described, for example, with reference to other figures) that includes one or more computing devices. Each corresponding part of the example training process can be executed by any one (or any combination of) of the one or more computing devices. Furthermore, one or more parts of the example training process can be implemented on the hardware components of the apparatus described herein, for example, to train one or more systems or models. Figure 8For illustrative and discussion purposes, elements executed in a specific order are depicted. Those skilled in the art will understand using the disclosure provided herein that elements of any of the methods discussed herein can be adapted, rearranged, extended, omitted, combined, or modified in various ways without departing from the scope of this disclosure. Figure 8 The descriptions of elements / terms are for illustrative purposes only and are not intended to be limiting. One or more parts of the example training process may be additionally or alternatively performed by other systems.

[0146] Initially, development model 16 can be kept in its initial state as initialization model 21. Development model 16 can be initialized using weight values. The initial weight values ​​can be random or based on an initialization pattern. The initial weight values ​​can be based on previous pre-training for the same or different models.

[0147] The initialization model 21 can undergo pre-training in the pre-training phase 22. The pre-training phase 22 can be implemented using one or more pre-training pipelines 17-2 on data from dataset 17-1. For example, if the initialization model 21 has already been pre-trained (e.g., the development model 16 contains, is, or is based on a pre-trained base model or expert model), pre-training can be omitted.

[0148] The pre-trained model 23 can then be a new version of the development model 16, which can remain as the development model 16 or a new development model. If the development model 16 has already been pre-trained, the pre-trained model 23 can be in its initial state. The pre-trained model 23 can undergo fine-tuning in the fine-tuning phase 24. The fine-tuning phase 24 can be implemented using one or more fine-tuning pipelines 17-3 on data from dataset 17-1. For example, fine-tuning can be omitted if the pre-trained model has satisfactory performance, if the model has already been fine-tuned, or if other tuning methods are preferred.

[0149] The fine-tuned model 29 can then be a new version of the development model 16, which can remain as the development model 16 or a new development model. If the development model 16 has already been fine-tuned, the fine-tuned model 29 can be in its initial state. The fine-tuned model 29 can undergo refinement using user feedback 26. For example, refinement using user feedback 26 can include reinforcement learning, optionally based on human feedback from human users of the fine-tuned model 25. Since reinforcement learning can be a form of fine-tuning, it is understood that the fine-tuning phase 24 can include phases for refinement using user feedback 26. Refinement using user feedback 26 can produce a refined model 27. The refined model 27 can be output to the downstream system 28 for deployment or further development.

[0150] In some implementations, computational optimization operations can be applied before, during, or after each stage. For example, initializing model 21 can undergo computational optimization 29-1 before pre-training stage 22 (e.g., using computational optimization toolkit 19). Pre-trained model 23 can undergo computational optimization 29-2 before fine-tuning stage 24 (e.g., using computational optimization toolkit 19). Fine-tuned model 25 can undergo computational optimization 29-3 before refinement using user feedback 26 (e.g., using computational optimization toolkit 19). Refined model 27 can undergo computational optimization 29-4 before output to downstream system 28 (e.g., using computational optimization toolkit 19). Computational optimizations 29-1, ..., 29-4 can be all the same, all different, or include at least some different optimization techniques.

[0151] Example machine learning model inference system

[0152] Figure 9 This is a block diagram of an inference system used to operate one or more machine learning models 1 to perform inference (e.g., for training, for deployment, etc.). Model host 31 can receive machine learning model 1. Model host 31 can host one or more model instances 31-1, which can be one or more instances of one or more models. Model host 31 can use available computing resources 31-2 associated with model host 31 to host model instances 31-1.

[0153] Model host 31 can perform inference on behalf of one or more clients 32. Client 32 can transmit input request 33 to model host 31. Using input request 33, model host 31 can obtain input 2 to feed into machine learning model 1. Machine learning model 1 can process input 2 to generate output 3. Using output 3, model host 31 can return output payload 34 in response to input request 33 from client 32. Output payload 34 can include or be based on output 3.

[0154] Model host 31 can utilize various other resources and tools to enhance the inference task. For example, model host 31 can communicate with tool interface 35 to facilitate the use of tools by model instance 31-1. Tool interface 35 may include local or remote APIs. Tool interface 35 may include integrated scripts or other software functions. Model host 31 can use online learning interface 36 to facilitate continuous improvement of machine learning model 1. For example, online learning interface 36 can be used within a reinforcement learning loop to retrieve user feedback on the inference served by model host 31. Model host 31 can access runtime data source 37 for enhancing input 2 with additional contextual information. For example, runtime data source 37 may include knowledge graph 37-1 that facilitates structured information retrieval for information associated with input request 33 (e.g., search engine service). Runtime data source 37 may include public or private, external or local database 37-2 that can store information associated with input request 33 for enhancing input 2. The runtime data source 37 may include account data 37-3, which can be retrieved in association with the user account corresponding to the client 32 to customize the behavior of the model host 31 accordingly.

[0155] The model host 31 may be implemented by one or more computing devices or systems. The client 2 may be implemented by one or more computing devices or systems, which may include computing devices or systems shared with the model host 31.

[0156] For example, model host 31 can operate on a server system that provides machine learning services (e.g., via a local area network or wide area network) to client devices operating client 32. The client device can be an end-user device used by an individual. The client device can also be a server system that operates client 32 to provide various functionalities as services to downstream end-user devices.

[0157] In some implementations, model host 31 may operate on the same device or system as client 32. Model host 31 may be a machine learning service that runs on the device to provide machine learning capabilities to one or more applications operating on the client device, which may include the application implementing client 32. Model host 31 and client 32 may be part of the same application. For example, model host 31 may be a subroutine or method implemented as part of the application, and client 32 may be another subroutine or method that uses model host 31 to perform inference functions within the application. It should be understood that model host 31 and client 32 may have various different configurations.

[0158] Model instance 31-1 may include one or more machine learning models that can be used to perform inference. Model instance 31-1 may include weights or other model components stored on / in persistent storage, temporarily cached, or loaded into high-speed memory. Model instance 31-1 may include multiple instances of the same model (e.g., for parallel execution of more requests on the same model). Model instance 31-1 may include instances of different models. Model instance 31-1 may include cached intermediate states of active or inactive models, which are used to accelerate inference for those models. For example, an inference session with a particular model can generate a significant amount of computational results that can be reused for future inference runs (e.g., using a KV cache for a transformer-based model). These computational results can be stored in association with the inference session, allowing for more efficient execution when the session resumes.

[0159] Computing resource 31-2 may include one or more processors (central processing unit, graphics processing unit, tensor processing unit, machine learning accelerator, etc.) connected to one or more memory devices. Computing resource 31-2 may include a dynamic pool of available resources shared with other processes. Computing resource 31-2 may include a memory device large enough to fit an entire model instance into a single memory instance. Computing resource 31-2 may also shard model instances across multiple memory devices (e.g., using data parallelization or tensor parallelization). Doing so can increase parallelization or execute large models using multiple memory devices that, individually, might not be able to fit the entire model into memory.

[0160] Input request 33 may include data for input 2. Model host 31 can process input request 33 to obtain input 2. Input 2 can be obtained directly from input request 33 or retrieved using input request 33. Input request 33 can be submitted to model host 31 via API.

[0161] Model host 31 can perform inference on multiple batch input requests 33 in parallel. For example, model instance 31-1 can be configured with an input structure having batch dimensions. Individual inputs 2 can be distributed across batch dimensions (e.g., rows of an array). Individual inputs 2 can include completely different contexts. Individual inputs 2 can be multiple inference steps for the same task. Individual inputs 2 can be interleaved in the input structure, such that any given inference loop can operate on different parts of the corresponding inputs 2. In this way, for example, model host 31 can perform inference on batches in parallel, such that output 3 can also contain batch dimensions and return the inference results of batch inputs 2 in parallel. In this way, for example, multiple batch input requests 33 can be processed in parallel to achieve higher throughput of output payload 34.

[0162] The output payload 34 may include or be based on the output 3 from the machine learning model 1. The model host 31 may process the output 3 to obtain the output payload 34. This may include chaining multiple rounds of inference (e.g., iteratively, recursively, across the same or different models) to obtain the final output of the task to be returned in the output payload 34. The output payload 34 may be transferred to the client 32 via an API.

[0163] Online learning interface 36 can facilitate reinforcement learning for machine learning model 1. Online learning interface 36 can facilitate reinforcement learning using human feedback (RLHF). Online learning interface 36 can facilitate federated learning for machine learning model 1.

[0164] Model host 31 can execute machine learning model 1 to perform inference for various tasks using various types of data. For example, various inputs 2 and outputs 3 can be used for various different tasks. In some implementations, input 2 may be or otherwise represent image data. Machine learning model 1 can process image data to generate outputs. As an example, machine learning model 1 can process image data to generate image recognition outputs (e.g., image data identification, latent embedding of image data, encoded representation of image data, hashing of image data, etc.). As another example, machine learning model 1 can process image data to generate image segmentation outputs. As another example, machine learning model 1 can process image data to generate image classification outputs. As another example, machine learning model 1 can process image data to generate image data modification outputs (e.g., image data alterations, etc.). As another example, machine learning model 1 can process image data to generate encoded image data outputs (e.g., encoded and / or compressed representations of image data, etc.). As another example, machine learning model 1 can process image data to generate upgraded image data outputs. As another example, machine learning model 1 can process image data to generate predictive outputs.

[0165] In some implementations, the task is a computer vision task. In some cases, the input 2 includes pixel data from one or more images, and the task is an image processing task. For example, an image processing task could be image classification, where the output is a set of scores, each score corresponding to a different object class and representing the probability that one or more images depict an object belonging to that object class. An image processing task could be object detection, where the image processing output identifies one or more regions in one or more images, and for each region, identifies the probability that the region depicts an object of interest. As another example, an image processing task could be image segmentation, where the image processing output defines a corresponding probability for each of a predetermined set of categories for each pixel in one or more images. For example, this set of categories could be foreground and background. As another example, this set of categories could be object classes. As another example, an image processing task could be depth estimation, where the image processing output defines a corresponding depth value for each pixel in one or more images. As another example, an image processing task could be motion estimation, where the network input includes multiple images, and the image processing output defines the motion of the scene depicted at that pixel between the images in the network input for each pixel in one of the input images.

[0166] In some implementations, input 2 can be or otherwise represent natural language data. Machine learning model 1 can process the natural language data to generate output. As an example, machine learning model 1 can process natural language data to generate language-encoded output. As another example, machine learning model 1 can process natural language data to generate latent text embedding output. As another example, machine learning model 1 can process natural language data to generate transformation output. As another example, machine learning model 1 can process natural language data to generate classification output. As another example, machine learning model 1 can process natural language data to generate text segmentation output. As another example, machine learning model 1 can process natural language data to generate semantic intent output. As another example, machine learning model 1 can process natural language data to generate upgraded text or natural language output (e.g., text or natural language data of higher quality than the input text or natural language). As another example, machine learning model 1 can process natural language data to generate predictive output (e.g., one or more predicted subsequent parts of natural language content).

[0167] In some implementations, input 2 can be or otherwise represent speech data (e.g., data describing spoken natural language, such as audio data, text data, etc.). Machine learning model 1 can process the speech data to generate output. As an example, machine learning model 1 can process speech data to generate speech recognition output. As another example, machine learning model 1 can process speech data to generate speech conversion output. As another example, machine learning model 1 can process speech data to generate latent embedding output. As another example, machine learning model 1 can process speech data to generate encoded speech output (e.g., encoded and / or compressed representations of speech data, etc.). As another example, machine learning model 1 can process speech data to generate upgraded speech output (e.g., speech data of higher quality than the input speech data, etc.). As another example, machine learning model 1 can process speech data to generate text representation output (e.g., a text representation of the input speech data, etc.). As another example, machine learning model 1 can process speech data to generate predictive output.

[0168] In some implementations, input 2 can be or otherwise represent latent encoded data (e.g., a latent space representation of the input). Machine learning model 1 can process the latent encoded data to generate an output. As an example, machine learning model 1 can process the latent encoded data to generate an identification output. As another example, machine learning model 1 can process the latent encoded data to generate a reconstruction output. As another example, machine learning model 1 can process the latent encoded data to generate a search output. As another example, machine learning model 1 can process the latent encoded data to generate a re-clustering output. As yet another example, machine learning model 1 can process the latent encoded data to generate a prediction output.

[0169] In some implementations, input 2 may be or otherwise represent statistical data. Statistical data may be, represent, or otherwise include data calculated and / or computed from another data source. Machine learning model 1 can process statistical data to generate output. As an example, machine learning model 1 can process statistical data to generate identification output. As another example, machine learning model 1 can process statistical data to generate prediction output. As another example, machine learning model 1 can process statistical data to generate classification output. As another example, machine learning model 1 can process statistical data to generate segmentation output. As another example, machine learning model 1 can process statistical data to generate visualization output. As another example, machine learning model 1 can process statistical data to generate diagnostic output.

[0170] In some implementations, input 2 can be or otherwise represent sensor data. Machine learning model 1 can process the sensor data to generate output. As an example, machine learning model 1 can process sensor data to generate identification output. As another example, machine learning model 1 can process sensor data to generate prediction output. As another example, machine learning model 1 can process sensor data to generate classification output. As another example, machine learning model 1 can process sensor data to generate segmentation output. As another example, machine learning model 1 can process sensor data to generate visualization output. As another example, machine learning model 1 can process sensor data to generate diagnostic output. As another example, machine learning model 1 can process sensor data to generate detection output.

[0171] In some implementations, the machine learning model 1 can be configured to perform tasks including encoding input data to achieve reliable and / or efficient transmission or storage (and / or corresponding decoding). For example, the task could be an audio compression task. The input could include audio data, and the output could include compressed audio data. In another example, the input includes visual data (e.g., one or more images or videos), the output includes compressed visual data, and the task is a visual data compression task. In another example, the task could include generating embeddings for input data (e.g., input audio or visual data). In some cases, the input includes audio data representing spoken utterances, and the task is a speech recognition task. The output could include text output mapped to spoken utterances. In some cases, the task includes encrypting or decrypting the input data. In some cases, the task includes microprocessor performance tasks such as branch prediction or memory address translation.

[0172] In some implementations, the task is a generative task, and the machine learning model 1 can be configured to output content generated in view of input 2. For example, input 2 can be, or otherwise represent, data of one or more modalities that encode the context used to generate additional content.

[0173] In some implementations, the task can be a text completion task. Machine learning model 1 can be configured to process input 2, representing text data, and generate output 3, which represents additional text data that completes the text sequence including input 2. For example, machine learning model 1 can be configured to generate output 3 to complete a sentence, paragraph, or section of text following a portion of the text represented by input 2.

[0174] In some implementations, the task can be an instruction-following task. The machine learning model 1 can be configured to process input 2 representing instructions for performing a function and generate output 3 that advances towards a goal satisfying the instruction function (e.g., at least one step of a multi-step process for performing the function). Output 3 can represent data of the same or different modality as input 2. For example, input 2 can represent text data (e.g., natural language instructions for a task to be performed), and the machine learning model 1 can process input 2 to generate output 3 representing text data in response to the instructions (e.g., a natural language response, a programming language response, a machine language response, etc.). Input 2 can represent image data (e.g., image-based instructions for a task to be performed, optionally accompanied by text instructions), and the machine learning model 1 can process input 2 to generate output 3 representing text data in response to the instructions (e.g., a natural language response, a programming language response, a machine language response, etc.). One or more outputs 3 can be generated iteratively or recursively to sequentially process and complete steps toward completing the requested function. For example, the initial output can be executed by an external system or processed by the machine learning model 1 to complete the initial steps of performing the function. Multiple steps can be performed, in which the final output is obtained in response to the initial instruction.

[0175] In some implementations, the task can be a question-answering task. A machine learning model 1 can be configured to process input 2 representing a question to be answered and generate output 3 that advances towards the goal of returning an answer to the question (e.g., at least one step in a multi-step process for performing the function). Output 3 can represent data of the same or different modality as input 2. For example, input 2 can represent text data (e.g., natural language instructions for a task to be performed), and machine learning model 1 can process input 2 to generate output 3 representing text data in response to the question (e.g., a natural language response, a programming language response, a machine language response, etc.). Input 2 can represent image data (e.g., image-based instructions for a task to be performed, optionally accompanied by text instructions), and machine learning model 1 can process input 2 to generate output 3 representing text data in response to the question (e.g., a natural language response, a programming language response, a machine language response, etc.). One or more outputs 3 can be generated iteratively or recursively to sequentially process and complete the steps leading to an answer to the question. For example, the initial output can be executed by an external system or processed by a machine learning model 1 to complete the initial steps to obtain an answer to the question (e.g., querying a database, performing calculations, executing scripts, etc.). Multiple steps can be performed, in which a final output in response to the question is obtained.

[0176] In some implementations, the task can be an image generation task. Machine learning model 1 can be configured to process input 2, which represents context regarding a desired portion of the image content. Context can include text data, image data, audio data, etc. Machine learning model 1 can be configured to generate output 3, which represents image data depicting the image in relation to that context. For example, machine learning model 1 can be configured to generate pixel data of an image. The values ​​of the channels associated with pixels in the pixel data can be selected based on context (e.g., based on probabilities determined according to the context).

[0177] In some implementations, the task can be an audio generation task. Machine learning model 1 can be configured to process input 2, which represents context regarding a desired portion of the audio content. Context can include text data, image data, audio data, etc. Machine learning model 1 can be configured to generate output 3, which represents audio data associated with that context. For example, machine learning model 1 can be configured to generate waveform data in the form of an image (e.g., a spectrogram). The values ​​of channels associated with pixels in the image can be selected based on the context. Machine learning model 1 can be configured to generate waveform data in the form of a sequence of discrete samples of a continuous waveform. The values ​​of the sequence can be selected based on the context (e.g., based on probabilities determined according to the context).

[0178] In some implementations, the task can be a data generation task. Machine learning model 1 can be configured to process input 2, which represents context regarding a desired portion of the data (e.g., data from various data domains, such as sensor data, image data, multimodal data, statistical data, etc.). For example, the desired data can be synthetic data used to train other machine learning models. The context can include any data type. Machine learning model 1 can be configured to generate output 3, which represents data aligned with the desired data. For example, machine learning model 1 can be configured to generate data values ​​to populate a dataset. The values ​​of data objects can be selected based on context (e.g., based on probabilities determined according to the context).

[0179] Example computing system and device

[0180] Figure 10This is a block diagram of an example networked computing system capable of implementing aspects of the exemplary implementations of this disclosure. The system may include multiple computing devices and systems communicatively coupled via network 49. Example computing device 50 is described to provide an example of a computing device capable of implementing any aspect of this disclosure (e.g., implementing model host 31, client 32, or both). Example server computing system 60 is described as an example of a server computing system capable of implementing any aspect of this disclosure (e.g., implementing model host 31, client 32, or both). Computing device 50 and server computing system 60 may interact collaboratively (e.g., via network 49) to implement any aspect of this disclosure (e.g., implementing model host 31, client 32, or both). Model development platform system 70 is an example system of model development platform 12 that can host or serve models for developing machine learning models. Third-party system 80 is an example system that any of computing device 50, server computing system 60, or model development platform system 70 can interact with when implementing various aspects of this disclosure (e.g., using third-party tools, accessing third-party databases or other resources, etc.).

[0181] Network 49 can be any type of communication network, such as a local area network (e.g., intranet), a wide area network (e.g., the Internet), or some combination thereof, and can include any number of wired or wireless links. Generally, communication via network 49 can use a wide variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), or protection schemes (e.g., VPN, Secure HTTP, SSL) via any type of wired or wireless connection. Network 49 can also be implemented via a system bus. For example, Figure 10 One or more devices or systems may be located in the same place as, contained in, or otherwise integrated into one or more other devices or systems.

[0182] Computing device 50 can be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop computer), a mobile computing device (e.g., a smartphone or tablet computer), a game console or controller, a wearable computing device, an embedded computing device, a server computing device, a virtual machine operating on a host device, or any other type of computing device. Computing device 50 can be a client computing device. Computing device 50 can be an end-user computing device. Computing device 50 can be a computing device that provides services to an end user (who may use another computing device to interact with computing device 50).

[0183] Computing device 50 may include one or more processors 51 and memory 52. ​​Processor 51 may be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and may be a single processor or multiple processors operatively connected. Memory 52 may include one or more non-transitory computer-readable storage media, such as HBM, RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. Memory 52 may store data 53 and instructions 54, which may be executed by processor 51 to cause user computing device 50 to perform operations. These operations may implement any or more features described herein. These operations may implement the example methods and techniques described herein.

[0184] The computing device 50 may also include one or more input components for receiving user input. For example, the user input component may be a touch-sensitive component (e.g., a touch-sensitive display or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or stylus). The touch-sensitive component may be used to implement a virtual keyboard. Other example user input components include a microphone, camera, LiDAR, physical keyboard or other buttons, or other components through which the user can provide input.

[0185] The computing device 50 may store or include one or more machine learning models 55. The machine learning model 55 may include one or more machine learning models 1, such as sequence processing model 4. The machine learning model 55 may include one or more model instances 31-1. The machine learning model 55 may be received from a server computing system 60, a model development platform system 70, a third-party system 80 (e.g., an application distribution platform), or developed locally on the computing device 50. The machine learning model 55 may be loaded into memory 52 and used by the processor 51 or otherwise implemented. The computing device 50 may implement multiple parallel instances of the machine learning model 55.

[0186] Server computing system 60 may include one or more processors 61 and memory 62. Processor 61 may be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and may be a single processor or multiple processors operatively connected. Memory 62 may include one or more non-transitory computer-readable storage media, such as HBM, RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. Memory 62 may store data 63 and instructions 64, which may be executed by processor 61 to cause server computing system 60 to perform operations. These operations may implement any or more features described herein. These operations may implement the exemplary methods and techniques described herein.

[0187] In some implementations, the server computing system 60 includes one or more server computing devices or is otherwise implemented by one or more server computing devices. In instances where the server computing system 60 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0188] Server computing system 60 may store or otherwise include one or more machine learning models 65. Machine learning model 65 may be the same as or different from machine learning model 55. Machine learning model 65 may include one or more machine learning models 1, such as sequence processing model 4. Machine learning model 65 may include one or more model instances 31-1. Machine learning model 65 may be received from computing device 50, model development platform system 70, third-party system 80, or developed locally on server computing system 60. Machine learning model 65 may be loaded into memory 62 and used by processor 61 or otherwise implemented. Server computing system 60 may implement multiple parallel instances of machine learning model 65.

[0189] In the example configuration, the machine learning model 65 may be included in or otherwise stored and implemented by the server computing system 60 to establish a client-server relationship with the computing device 50 for servicing model inference. For example, the server computing system 60 may implement the model host 31 on behalf of client 32 on the computing device 50. For example, the machine learning model 65 may be implemented by the server computing system 60 as part of a web service (e.g., a remote machine learning model hosting service, such as an online interface for performing machine learning model operations on the server computing system 60 over a network). For example, the server computing system 60 may communicate with the computing device 50 via a local intranet or internet connection. For example, the computing device 50 may be a workstation or endpoint communicating with the server computing system 60, where the implementation of the machine learning model 65 is managed by the server computing system 60 to remotely perform inference (e.g., for runtime or training operations), and where output is returned (e.g., projected, streamed, etc.) to the computing device 50. The machine learning model 65 may work cooperating or interoperating with the machine learning model 55 on the computing device 50 to perform various tasks.

[0190] The model development platform system 70 may include one or more processors 71 and memory 72. Processor 71 may be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and may be a single processor or multiple processors operatively connected. Memory 72 may include one or more non-transitory computer-readable storage media, such as HBM, RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. Memory 72 may store data 73 and instructions 74, which may be executed by processor 71 to cause the model development platform system 70 to perform operations. These operations may implement any one or more features described herein. These operations may implement the example methods and techniques described herein. Example operations include the functionality described herein with respect to model development platform 12. This functionality and other functionalities may be implemented by developer tools 75.

[0191] The third-party system 80 may include one or more processors 81 and memory 82. Processor 81 may be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and may be a single processor or multiple processors operatively connected. Memory 82 may include one or more non-transitory computer-readable storage media, such as HBM, RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. Memory 82 may store data 83 and instructions 84, which may be executed by processor 81 to cause the third-party system 80 to perform operations. These operations may implement any one or more features described herein. These operations may implement the example methods and techniques described herein. Example operations include the functionality described herein with respect to tools and other external resources (e.g., third-party resource 85) invoked when training machine learning models 1, 4, 16, 20, 55, 65, etc., or performing inference using the machine learning model.

[0192] Figure 10An example arrangement of a computing system that can be used to implement the present disclosure is shown. Other computing system configurations may also be used. For example, in some implementations, one or both of computing system 50 or server computing system 60 may implement all or part of the operation of model development platform system 70. For example, computing system 50 or server computing system 60 may implement developer tool 75 (or extensions thereof) to develop, update / train, or refine machine learning models 1, 4, 16, 20, 55, 65, etc., using one or more techniques described herein with respect to model alignment toolkit 17. In this way, for example, computing system 50 or server computing system 60 may develop, update / train, or refine machine learning models based on local datasets (e.g., for model personalization / customization, as permitted by user data preference selection).

[0193] Figure 11 This is a block diagram of an example computing device 98 implemented according to an example embodiment of the present disclosure. The computing device 98 may be a user computing device or a server computing device (e.g., computing device 50, server computing system 60, etc.). The computing device 98 may implement model host 31. For example, the computing device 98 may include multiple applications (e.g., applications 1 to N). Each application may contain its own machine learning library and machine learning models. For example, each application may include a machine learning model. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc. Figure 11 As shown, each application can communicate with multiple other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, or additional components. In some implementations, each application can communicate with each device component using an API (e.g., a public API). In other implementations, the API used by each application is application-specific.

[0194] Figure 12 This is a block diagram of an example computing device 99 implemented according to an example embodiment of the present disclosure. Computing device 99 may be the same as or different from computing device 98. Computing device 99 may be a user computing device or a server computing device (e.g., computing device 50, server computing system 60, etc.). Computing device 98 may implement model host 31. For example, computing device 99 may include multiple applications (e.g., applications 1 to N). Each application may communicate with a central intelligence layer. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc. In some implementations, each application may use an API (e.g., a common API across all applications) to communicate with the central intelligence layer (and the model stored therein).

[0195] The central intelligence layer can include multiple machine learning models. For example, such as... Figure 12 As shown, a corresponding machine learning model can be provided for each application, and this corresponding machine learning model is managed by a central intelligent layer. In other implementations, two or more applications can share a single machine learning model. For example, in some implementations, the central intelligent layer can provide a single model for all applications. In some implementations, the central intelligent layer is included within the operating system of the computing device 99 or otherwise implemented by the operating system.

[0196] The central intelligence layer can communicate with the central device data layer. The central device data layer can be a centralized data storage repository for computing device 99. For example... Figure 12 As shown, the central device data layer can communicate with multiple other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0197] ADDITIONAL DISCLOSURE

[0198] This paper discusses technologies related to servers, databases, software applications, and other computer-based systems, as well as the actions taken and the information sent to and from such systems. The inherent flexibility of computer-based systems allows for a wide range of possible configurations, combinations, and divisions of tasks and functions between and within components. For example, the processes discussed herein can be implemented using a single device or component, or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0199] While the subject matter has been described in detail with respect to various specific example embodiments, each example is provided by way of illustration and not limitation. Modifications, variations, and equivalents of such embodiments will be readily apparent to those skilled in the art upon understanding the foregoing. Therefore, this disclosure does not exclude such modifications, alterations, or additions to the subject matter that will be readily understood by those of ordinary skill in the art. For example, features shown or described as part of one embodiment may be used with another embodiment to produce yet another embodiment. Therefore, it is intended that this disclosure cover such modifications, variations, and equivalents.

[0200] The aspects of this disclosure have been described with reference to illustrative embodiments thereof. Any and all features of the appended claims may be combined or rearranged in any possible manner, including combinations of claims not expressly listed together, as the illustrative claims dependencies listed herein should not be construed as limiting the scope of possible combinations of features disclosed herein. Therefore, the scope of this disclosure is illustrative rather than limiting, and this disclosure does not exclude such modifications, alterations, or additions to the subject matter that will be readily understood by one of ordinary skill in the art. Furthermore, terms are described herein using lists of illustrative elements connected by conjunctions such as “and,” “or,” “but,” etc. It should be understood that such conjunctions are provided for illustrative purposes only. For example, a sequence of terms and other items connected by a specific conjunction such as “or” may refer to “and / or,” “at least one of,” “any combination,” etc., of the illustrative elements listed therein. Terms such as “based on” should be understood as “at least partially based on.”

[0201] The term "capable" should be understood as referring to the possibility of a feature in various implementations, rather than a capability that must exist in every implementation. For example, the phrase "X can perform Y" should be understood as indicating that in various implementations, X may be configured to perform Y, rather than indicating that X must always be able to perform Y in every instance. It should be understood that in various implementations, X may not be able to perform Y and is still within the scope of this disclosure.

[0202] The term "may" should be understood as referring to the possibility of a feature in various implementations, rather than specifying a capability that must exist in every implementation. For example, the phrase "X can perform Y" should be understood as indicating that in various implementations, X may be configured to perform Y, rather than indicating that X must always be able to perform Y in every instance. It should be understood that in various implementations, X may not be able to perform Y and is still within the scope of this disclosure.

Claims

1. A method for automating the use of computer-based tools based on learned similarity, the method comprising: The data describing the query is obtained from a computing system comprising one or more computing devices; The computing system generates a query embedding for the query, wherein the query embedding is expressed in a latent embedding space; The computing system performs a similarity search within the potential embedding space for the query embedding to identify one or more previously defined embeddings associated with one or more previously defined query-response pairs; The computing system generates a prompt based on the query and the one or more previously defined query-response pairs; The computational system provides the prompts as input for processing by a language model used in machine learning; and The computing system receives the model response to the query, which is output by the language model of the machine learning based on the processing of the prompt.

2. The computer-implemented method as described in any of the preceding claims, wherein, The model response includes one or more lexical units, which, when executed by the computing system, cause structural tools to retrieve or transform information.

3. The computer-implemented method as described in claim 2, further comprising: The computing system processes the model response to generate a system response; Processing the model response includes executing one or more lexical units to cause the structural tool to retrieve or transform the information; and The system response is at least in part based on the information retrieved or transformed by the structural tool in response to the execution of the one or more lexical terms.

4. The computer-implemented method as described in claim 3, wherein, The query includes a natural language question, and the system response includes a natural language answer to the natural language question.

5. The computer-implemented method as described in any one of claims 2 to 4, wherein, The structural tools include a programming language interpreter configured to execute instructions expressed in a computer language.

6. The computer-implemented method of claim 5, further comprising: The computing system preloads one or more libraries or datasets associated with the computer language into the computer environment associated with the programming language interpreter.

7. The computer-implemented method as described in claim 5 or 6, wherein, The computer languages ​​mentioned include the Python computer language.

8. The computer-implemented method as described in any one of claims 2 to 7, wherein, The structural tools include: Database lookup used to access additional information from the database; API calls used to request and receive additional information via the Application Programming Interface (API); A query service that retrieves results from search engines, knowledge graphs, or digital assistants.

9. The computer-implemented method as described in any of the preceding claims, wherein, The computational system generates the prompt based on the query and the one or more previously defined query-response pairs by concatenating the query with the one or more previously defined query-response pairs.

10. The computer-implemented method as described in any of the preceding claims, wherein, The computational system generates the prompt based on the query and the one or more previously defined query-response pairs, including ranking the one or more previously defined query-response pairs.

11. The computer-implemented method as described in any of the preceding claims, wherein: At least one of the previously defined query-response pairs demonstrates the use of the structuring tool; and The processing of the prompts by the language model of the machine learning includes the use of the structural tools.

12. The computer-implemented method as described in any of the preceding claims, further comprising, before obtaining the data describing the query: The computing system generates the previously defined embedding for each of the previously defined query-response pairs, at least in part based on the query portion of each of the one or more previously defined query-response pairs.

13. The computer-implemented method as described in any of the preceding claims, wherein, The computing system performing the similarity search for the query embedding within the potential embedding space includes identifying the previous embeddings from the embedding space. One set, of which It's a hyperparameter.

14. The computer-implemented method as described in any of the preceding claims, wherein, The model response includes an executable program.

15. The computer-implemented method as described in any of the preceding claims, wherein, The machine learning language model includes a large, pre-trained language model.

16. The computer-implemented method as described in any of the preceding claims, further comprising: The language model of the machine learning is fine-tuned on the previously defined query-response pairs.

17. The computer-implemented method as described in any of the preceding claims, wherein, Providing the prompt by the computing system as input for processing by the language model of the machine learning includes: the computing system running the language model of the machine learning to process the prompt.

18. A computer system configured to perform the method as described in any of the preceding claims.

19. The computer system of claim 18, wherein, The computer system includes the language model for machine learning.

20. One or more non-transitory computer-readable media, the one or more non-transitory computer-readable media storing instructions for performing the method as claimed in any one of claims 1 to 17.

Images