Machine learning-based agent for text editing

Abstract

For example, a computing system is provided that includes a machine learning agent which may be required to perform various editing tasks in a word processing application or other text generation environment. The machine learning agent may include one or more machine learning-based language models trained on a set of training data that provides examples of editing performed on long-form text.

Description

Technical Field

[0001] This disclosure generally relates to machine learning agents for performing text editing. More specifically, this disclosure relates to computing systems that can include machine learning agents that can be required to perform various editing tasks, for example, in a word processing application or other text generation environment. The machine learning agent can include one or more trained machine learning language models based on a set of training data that provides examples of edits performed on long-form text.

Background Art

[0002] Various approaches have been proposed for performing text editing using machine learning models. Many of these proposed approaches are limited to performing individual tasks on sentence-level text. Since sentence-level models are limited to the information of a single sentence without surrounding context, they may result in proposals that are inconsistent with the surrounding context, leading to low semantic and logical coherence at the paragraph or document level. Further, these sentence-level models typically do not function well when the input text is longer than a few sentences. In other approaches, performing document-level editing using so-called "large language models" or "LLMs" has been proposed. However, general-purpose LLMs are usually very large (e.g., with respect to the number of parameters and the amount of computation required), resulting in significant computational costs and often making them undesirable for use in production.

Summary of the Invention

[0003] Aspects and advantages of embodiments of this disclosure will be described in part in the following description, or can be learned from the description, or can be learned through the practice of the embodiments.

[0004] One exemplary embodiment relates to a computing system for automated text editing. The computing system includes a machine learning agent, which includes one or more machine learning models configured to perform one or more text editing tasks. The computing system includes a text generation application, and the machine learning agent includes an addressable entity that can be addressed within the text generation application. The computing system is configured to perform an operation, which includes the computing system obtaining data describing annotations entered by a user in the text generation application, the annotations corresponding to a set of text content created within the text generation application, and the annotations including a natural language text editing request addressed to the machine learning agent; the computing system executing the machine learning agent in response to the annotations, causing the machine learning agent to generate one or more edits to the set of text content based on the natural language text editing request; and the computing system updating the set of text content based on the one or more edits generated by the machine learning agent.

[0005] In some embodiments, annotations include comments displayed in the margins of the graphical user interface of a text generation application. In some embodiments, annotations are addressed to a machine learning agent using natural language. In some embodiments, annotations are addressed to a machine learning agent using specific symbols that designate the machine learning agent as the recipient of the annotation. In some embodiments, a set of text content includes two or more sentences. In some embodiments, a set of text content includes an entire document being edited by a user in a text generation application. In some embodiments, a set of text content includes a subset of a document being edited by a user in a text generation application, and the set of text content is selected by the user. In some embodiments, running a machine learning agent by a computing system includes the computing system generating model prompts based on the set of text content and natural language text editing requests, and the computing system processing the model prompts by one or more machine learning models of the machine learning agent to generate one or more edits as outputs of one or more machine learning models. In some embodiments, generating a model prompt by a computing system based on a set of text content and a natural language text editing request involves the computing system combining (1) a set of text content, (2) one or more contextual portions adjacent to the set of text content in a document being edited by a user in a text generation application, and (3) a natural language text editing request to form a model prompt.In some embodiments, the operation further includes the computing system running a machine learning agent to cause the machine learning agent to generate a second annotation in response to an annotation input by the user, and the computing system inserting the second annotation into a text generation application.

[0006] Other exemplary embodiments relate to computer-implemented methods for training language models. The method includes obtaining a training tuple by a computing system including one or more computing devices, which includes data describing a natural language text editing instruction, a set of text content, and one or more ground truth edits performed on the set of text content in relation to the natural language text editing instruction, wherein the set of text content includes two or more sentences. The method includes processing the natural language text editing instruction and the set of text content with a language model by the computing system to generate one or more predictive edits on the set of text content in response to the natural language text editing instruction. The method includes evaluating a loss function by the computing system that generates a loss value based on a comparison of one or more predictive edits with one or more ground truth edits. The method includes modifying one or more parameter values ​​of the language model by the computing system based on the loss function.

[0007] In some embodiments, the training tuple includes comments and edit history obtained from a text generation application in which the language model is deployed. In some embodiments, the training tuple includes edit history obtained from a large text edit corpus. In some embodiments, the large text edit corpus includes edits to an online collaboratively edited encyclopedia. In some embodiments, the natural language instruction includes a summary of one or more ground truth edits. In some embodiments, the natural language instruction includes a modified version of a summary of one or more ground truth edits, and the modified version of the summary is generated by a second language model different from the original version of the summary of one or more ground truth edits. In some embodiments, the natural language instruction is selected by a reward language model. In some embodiments, the natural language instruction is filtered based on natural language inference (NLI) score, length, or edit distance. The NLI score can indicate an inference relationship between two (e.g., ordered) texts, such as pre-edit and post-edit texts. The inference relationship can indicate, for example, implication, contradiction, or neutrality. In some embodiments, the natural language instruction includes a synthetic prompt.

[0008] Other aspects of this disclosure cover a variety of systems, apparatus, non-temporary computer-readable media, user interfaces, and electronic devices.

[0009] These and other features, aspects and advantages of the various embodiments of this disclosure will be better understood by referring to the following description and the appended claims. The appended drawings incorporated herein and forming part of this specification illustrate exemplary embodiments of this disclosure and, together with the description, serve to illustrate the relevant principles.

[0010] A detailed description of embodiments intended for those skilled in the art is given herein with reference to the accompanying drawings. [Brief explanation of the drawing]

[0011] [Figure 1] A block diagram of an exemplary computing system according to an exemplary embodiment of the present disclosure is shown. [Figure 2A] This describes a sequence of exemplary graphical user interfaces illustrating exemplary interactions with exemplary machine learning agents according to exemplary embodiments of the present disclosure. [Figure 2B] This describes a sequence of exemplary graphical user interfaces illustrating exemplary interactions with exemplary machine learning agents according to exemplary embodiments of the present disclosure. [Figure 2C] This describes a sequence of exemplary graphical user interfaces illustrating exemplary interactions with exemplary machine learning agents according to exemplary embodiments of the present disclosure. [Figure 2D] This describes a sequence of exemplary graphical user interfaces illustrating exemplary interactions with exemplary machine learning agents according to exemplary embodiments of the present disclosure. [Figure 2E] This describes a sequence of exemplary graphical user interfaces illustrating exemplary interactions with exemplary machine learning agents according to exemplary embodiments of the present disclosure. [Figure 2F] This describes a sequence of exemplary graphical user interfaces illustrating exemplary interactions with exemplary machine learning agents according to exemplary embodiments of the present disclosure. [Figure 2G] This describes a sequence of exemplary graphical user interfaces illustrating exemplary interactions with exemplary machine learning agents according to exemplary embodiments of the present disclosure. [Figure 2H] This describes a sequence of exemplary graphical user interfaces illustrating exemplary interactions with exemplary machine learning agents according to exemplary embodiments of the present disclosure. [Figure 2I]This describes a sequence of exemplary graphical user interfaces illustrating exemplary interactions with exemplary machine learning agents according to exemplary embodiments of the present disclosure. [Figure 3A] A block diagram of an exemplary machine learning-trained language model according to an exemplary embodiment of this disclosure is shown. [Figure 3B] A block diagram of an exemplary machine learning-trained language model according to an exemplary embodiment of this disclosure is shown. [Figure 4A] This disclosure shows a block diagram illustrating an exemplary training process for a machine learning-prepared language model according to an exemplary embodiment of this disclosure. [Figure 4B] This disclosure shows a block diagram illustrating an exemplary training process for a machine learning-prepared language model according to an exemplary embodiment of this disclosure. [Figure 5] This flowchart illustrates an exemplary method for training a machine learning model according to exemplary embodiments of the aspects of this disclosure. [Figure 6] This is a block diagram of an exemplary processing flow for processing inputs(multiple) and generating outputs(multiple) using machine learning models(multiple) according to exemplary embodiments of the aspects of this disclosure. [Figure 7] This is a block diagram of an exemplary sequence processing model according to an exemplary embodiment of the aspects of the present disclosure. [Figure 8] This is an exemplary block diagram of the technique for filling in an exemplary input sequence for processing by a sequence processing model according to an exemplary embodiment of an aspect of the present disclosure. [Figure 9] This is a block diagram of an exemplary model development platform according to an exemplary embodiment of the aspects of the present disclosure. [Figure 10] This is a block diagram of an exemplary training workflow for training a machine learning model according to exemplary embodiments of the aspects of the present disclosure. [Figure 11] This is a block diagram of an inference system for operating one or more machine learning models to perform inference, according to an exemplary embodiment of an aspect of the present disclosure. [Figure 12] FIG. 1 is a block diagram of an exemplary network computing system according to an exemplary embodiment of an aspect of the present disclosure. [Figure 13] FIG. 2 is a block diagram of an exemplary computing device according to an exemplary embodiment of an aspect of the present disclosure. [Figure 14] FIG. 3 is a block diagram of an exemplary computing device according to an exemplary embodiment of an aspect of the present disclosure. DETAILED DESCRIPTION

[0012] Reference numbers repeated throughout the several drawings are intended to identify the same or like features in the various embodiments.

[0013] SUMMARY Generally, the present disclosure is directed to a machine learning agent included in or provided as a feature of a text generation application such as a web-based collaborative document processing application or other productivity tool that enables text generation. The machine learning agent may be required to perform various text editing tasks, including co-authoring tasks for document-level content. Specifically, according to one aspect of the present disclosure, the machine learning agent may be treated as an addressable entity within the document processing application. As an addressable entity, the machine learning agent can be called or requested to provide edits to a set of text content created within the document processing application.

[0014] For example, a user can add comments or other annotations to a set of text content. The user can address the comment to a machine learning agent (e.g., by explicitly marking the comment as directed to the agent, or by otherwise addressing the agent within the comment). User-generated comments may also contain edit requests expressed in natural language. Accordingly, the machine learning agent can edit the set of text content as requested by the comment. In some cases, the machine learning agent may also respond to the user's original comment with its own responsive comment (e.g., a description of the edit performed). In this way, a user can call on the machine learning agent to take action via comments or other annotations, and the machine learning agent can respond by performing the requested edit.

[0015] Other aspects of this disclosure relate to techniques for training a machine learning agent to perform the editing tasks described herein. Specifically, the machine learning agent may include one or more trained language models. The language model may be trained on a set of training data that provides examples of edits performed on long-form text. The term “long-form text” may refer to a set of text content containing two or more sentences. The training data may include examples of text before and after editing, and may also include corresponding natural language editing commands that describe exemplary edits (e.g., request). Additionally or alternatively, the training data may include natural language comments (e.g., edit descriptions) that respond to the original natural language editing commands. By training (e.g., fine-tuning) the language model on this training data, it may be possible to enable the model to edit long-form text in response to user-generated editing requests.

[0016] Training data can be created from several different sources. For example, training data can be created from annotation and edit history from the same document processing application or other productivity tool on which the machine learning agent is deployed. Another example is that training data can be created from edit history from a large text corpus, such as the edit history of an online collaboratively edited encyclopedia. For each edit in the text corpus's edit history, an edit summary may be available. This summary may be included in the training dataset as exemplary edit instructions. The edit summary may be included as is, or it may be modified or improved by applying human changes or edits made by a separate language model. More generally, several pre- or post-processing techniques may be performed on the training data to improve the performance of language models trained on it. This may generate training data that enables one or more language models to perform generalized text editing on long-form text content.

[0017] The system and method of this disclosure offer several technical effects and advantages. For example, this disclosure provides a general-purpose editing model that is not constrained to a specific task, but rather capable of performing any number of different types of editing operations permitted on the training dataset. Another exemplary technical effect is that this disclosure reduces the training cost and the amount of labeled data required to develop features in the content-symbiotic domain. In particular, the proposed model can quickly adapt to new features and generate training data for any downstream task. The reduction in training cost can translate to a reduction in the use of computational resources, such as processor usage, memory usage, and / or network bandwidth usage. Yet another exemplary technical effect is that this disclosure extends state-of-the-art LLM techniques to long-form editing tasks, such as document-level editing tasks.

[0018] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the drawings.

[0019] An exemplary system for automated text editing Figure 1 shows a block diagram of an exemplary computing system 100 according to an exemplary embodiment of the present disclosure. The computing system 100 includes a text generation application 112. The text generation application 112 may be a software program or application that enables a user to create, edit, format, and manage text content on a computer (for example, to be stored in or contained in a file or document). The text generation application 112 may provide tools and functions for creating, editing, and formatting text, as well as options for saving, printing, and sharing documents.

[0020] The text generation application 112 can be any number of different applications that allow one or more users to create text content.

[0021] For example, the text generation application 112 could be a word processing application or other text editor. The document processing application can be used to create and edit text documents and can provide a variety of functions for creating and formatting text content, including text editing, formatting, style, and layout options.

[0022] In other examples, the text generation application 112 could be a blogging platform or a social media platform. On these platforms, users may be able to create text content in the form of posts, tweets, updates, and articles.

[0023] In another example, the text generation application 112 could be an email client. The email client could allow the user to create text content in email format.

[0024] In another example, the text generation application 112 could serve as a platform for generating and editing programming language content. The application 112 may be used by programmers, developers, and other professionals to create and edit computer code, scripts, or other programming language content.

[0025] In another example, the text generation application 112 could be a note-taking application. A note-taking application may allow the user to create and organize text content in the form of notes, lists, and memos.

[0026] In yet another example, the text generation application 112 may be or include a spreadsheet software application, a presentation software application, a task or project management tool, a time management tool, or a cloud storage and file sharing tool.

[0027] The text generation application 112 can receive one or more user inputs 114 that result in the creation or editing of a set of text content 116. The inputs 114 may originate from one user or several different users. The inputs 114 may be received through user interaction with a graphical user interface. For example, in some embodiments, the text generation application 112 may be a web-based collaborative document processing application that enables one or more users to interact and collaborate over the internet to create or edit a set of text content 116 in real time, regardless of their physical location.

[0028] Therefore, a user(s) can use the editing tools provided by the text generation application 112 to provide input(s) 114 for making changes to a set of text content, including changes to text, formatting, and other elements. These changes may include adding, deleting, or modifying text, formatting text (e.g., changing font style, size, or color), and adding or modifying other elements such as images, tables, or hyperlinks.

[0029] The text generation application 112 may also allow users to generate comments 117 or other annotations on or about a set of text content 116. For example, a user can leave a comment 117 on the set of text content 116 to provide feedback, suggestions, or clarifications. The comment 117 can be displayed as an annotation or note in the document's margins, or as a pop-up box, and / or can be seen by other users who have access to the document. Users can respond to, resolve, or edit comments as needed.

[0030] According to aspects of this disclosure, the computing system 100 may also include a machine learning agent 118. The machine learning agent 118 is or may include a computer program or system that uses machine learning algorithms and techniques to perform tasks or make decisions autonomously or semi-autonomously based on input data and feedback from its environment. The machine learning agent 118 may be designed to learn from experience, adapt to changing conditions, and optimize its performance over time without being explicitly programmed for a particular task. Although the machine learning agent 118 is shown separately from the text generation application 112 in Figure 1, in other cases the machine learning agent 118 may be included in or a feature of the text generation application 112. In other cases the machine learning agent 118 may exchange data with the text generation application 112 using an application programming interface (API).

[0031] The machine learning agent 118 can interact with the text generation application 112. For example, the machine learning agent 118 can receive data describing a set of text content 116 and / or any associated annotations.

[0032] The machine learning agent 118 may include and implement one or more pre-trained language models 120. In some embodiments, the pre-trained language models 120 are, or may be referred to as, so-called “large-scale language models.” In other embodiments, the pre-trained language models 120 are, or may be referred to as, so-called “large-scale language models” (LMMs). A large-scale language model refers to a powerful machine learning model that can process and generate text on a large scale. These models are typically trained (e.g., “pre-trained”) on large amounts of data, often millions or billions of sentences, to learn patterns, structures, and nuances of natural language. A large-scale multimodal model includes a language model that can also process one or more additional forms of data other than language or text data. For example, additional data modalities may be image data (e.g., including video data), audio data, sensor data, time-series data, and / or other modalities of data. In embodiments where a model (e.g., a language model) processes a sequence of inputs and / or generates a sequence of outputs, such a model may be referred to as a “sequence-processing model.”

[0033] Language models (or multiple models)120 can be trained using a variety of techniques or architectures, such as recurrent neural networks (RNNs), transformers, or other machine learning algorithms. Training may be performed in two stages: (1) pre-training on a large unlabeled text corpus, followed by (2) fine-tuning on specific tasks, such as short-form and / or long-form editing tasks. Language models 120 can be trained to handle a wide range of natural language processing (NLP) tasks, including text generation, text completion, sentiment analysis, machine translation, and question answering. Examples of large language models include BERT (Bidirectional Encoder Representation from Transformer) and T5 (Text-to-Text Transmitter). An additional example is the Gemini family of LMMs.

[0034] According to aspects of this disclosure, the machine learning agent 118 may be required to perform a variety of text editing tasks, including co-creation tasks of document-level content. More specifically, according to one aspect of this disclosure, the machine learning agent 118 may be treated as an addressable entity of the text generation application 112. As an addressable entity, the machine learning agent 118 can be invoked or requested to provide edits to a set of text content 116 created within the text generation application 112.

[0035] As described above, users can add comments 117 or other annotations to a set of text content 116. For example, in the context of a text generation application 112, a comment can refer to a type of annotation or note that can be added to a set of text content 116, providing additional information, feedback, or suggestions without modifying the set of text content 116 itself. Comments 117 can be used for collaborative purposes, allowing multiple users to provide input and feedback on the document. They may appear as margins or pop-up boxes within the document and can be viewed, edited, and deleted as needed. Comments 117 may identify the person who added the comment, along with a timestamp, in order to identify the commenter and provide context.

[0036] According to the aspects of this disclosure, a user can address comment 117 to machine learning agent 118. For example, a user can explicitly mark comment 117 as directed to machine learning agent 118. For example, a user can mark comment 117 as "@" directed to machine learning agent 118 (or use any other symbol to formally specify the recipient of the comment). The "@" feature, commonly known as the "at sign" or "mention" feature, allows a user to refer to or tag another user (or agent 118) by using the "@" symbol followed by the username or handle of the person the user wishes to notify or refer to.

[0037] In another example, a user can informally address comment 117 to machine learning agent 118. A filter implemented using computer logic can detect that machine learning agent 118 is the intended recipient of the comment.

[0038] User-generated comments (117) may include edit requests expressed in natural language. Natural language refers to the language used by humans for communication, including oral and written forms of expression. Edit requests can generally convey the need for text editing, or more specifically, describe desired areas of improvement such as flow, quality, readability, effectiveness, consistency, grammar, punctuation, conciseness, engagement, and / or assertiveness. Natural language edit requests may also request a specific tone or describe an intended audience.

[0039] In response to a comment 117 addressed to the machine learning agent 118, the machine learning agent 118 can edit a set of text content 116 as requested by the comment 117. For example, the machine learning agent 118 can process the comment 117 and the set of text content 116 using one or more language models 120 to generate edited text. In some embodiments, the agent 118 can interact with a text generation application 112 to directly replace the existing set of text content 116 with edited content generated by the model 120. In other cases, the user may be given the opportunity to review the edited content and choose whether or not to replace the existing set of text content 116 with edited content generated by the model 120.

[0040] In some implementations, the edited content generated by Model 120 may be a complete set of replacement text. In some embodiments, the edited content generated by Model 120 may describe specific edits to an existing set of text content 116. In some implementations, the edited content generated by Model 120 may be provided in a clean format (without explicitly indicated edits or changes). In some embodiments, the edited content generated by Model 120 may be provided in markup format (with edits or changes indicated using visual cues such as strikethroughs, underlines, highlighting, and / or font color).

[0041] In some cases, the machine learning agent 118 may also respond to the user's original comment 117 with its own responsive comment 117 (which could be, for example, a description of the edits made). For example, a new comment 117 generated by the machine learning agent 118 may be provided in a comment chain or other comment history related to a set of text content 116.

[0042] Therefore, a user can call the machine learning agent 118 to take action via comment 117 or other annotations, and the machine learning agent 118 can respond by performing the requested editing operation. As an example of this process, Figures 2A-I depict a sequence of exemplary graphical user interfaces showing exemplary interaction with an exemplary machine learning agent according to exemplary embodiments of the present disclosure.

[0043] As shown in Figure 2A, the text generation application receives user input that generates an initial set of text content ("We are holding a summit in Zurich. Zurich is a great place to visit.").

[0044] As shown in Figure 2B, the user is highlighting a specific portion of a set of text content ("Zurich is a great place to visit"). The user then created a new comment using a graphical user interface.

[0045] As shown in Figure 2C, the user addresses the comment to a machine learning agent (in this example, the agent is referred to using the exemplary name "CoComposer").

[0046] As shown in Figure 2D, the user is entering a natural language explanation of the requested edit ("Could you elaborate on this a bit more?").

[0047] As shown in Figure 2E, in response to a user's comment, the machine learning agent edits the document to add additional text content (for example, content describing Zurich). The machine learning agent also provides an additional comment to the comment thread ("Expanded. Take a look.").

[0048] As shown in Figure 2F, both the user and the machine learning agent add further comments and respond to each other.

[0049] As shown in Figure 2G, the user has added additional text content ("I hope the summit will be a good one").

[0050] As shown in Figure 2H, the user again addresses the comment to the machine learning agent, including an additional natural language editing request ("Please check if this is OK").

[0051] As shown in Figure 2I, the machine learning agent edits the text selected by the user into updated or edited text ("Absolutely, the summit is going to be the best!"). The machine learning agent also adds a responsive comment ("Changed").

[0052] Example inference process Figure 3A shows a block diagram of an exemplary inference process of a machine learning-trained language model 120 according to an exemplary embodiment of the present disclosure. As shown in Figure 3A, a set of natural language instructions 350 and text content 352 may be provided as input to the machine learning-trained language model 120.

[0053] In some embodiments, the set of text content 352 may include the entire document being edited by the text generation application. In other embodiments, the set of text content 352 may include one or more subparts of the document. For example, the subpart of the document may be a part selected by the user (e.g., using highlighting). In some embodiments, one or more sentences surrounding the user-selected part(s) may also be included in the set of text content 352 to provide additional context for the instruction 350.

[0054] In some embodiments, a natural language instruction 350 and a set of text content 352 can be concatenated to form a model prompt provided as input. In some implementations, the model prompt may be structured according to a template.

[0055] In response to the input, the machine-learned language model 120 can generate one or more edits 360 for a set of text content 352. In some implementations, the edits 360 generated by the model 120 can be a complete set of replacement text. In some embodiments, the edits 360 generated by the model 120 can describe a specific edit to the existing set of text content 352. In some implementations, the edits 360 generated by the model 120 may be provided in a clean format (without explicitly indicated edits or changes). In some embodiments, the edits 360 generated by the model 120 may be provided in markup format (with edits or changes indicated using visual cues such as strikethroughs, underlines, highlighting, and / or font color).

[0056] Figure 3B shows a block diagram of another exemplary inference process of the machine-learned language model 120 according to an exemplary embodiment of the present disclosure. Figure 3B is very similar to Figure 3A, except that in Figure 3B the model 120 also outputs a responsive comment 362. In some embodiments, the edit(s) 360 and the responsive comment 362 may be generated by the model 120 in a single inference run. In other embodiments, the edit(s) 360 and the responsive comment 362 may be generated by the model 120 in different inference runs. For example, two prompts may be created, one prompt requesting the model 120 to perform the requested edit(s) 360. A second prompt may request the model 120 to generate a responsive comment 362 as a summary of the edit(s) 360.

[0057] Exemplary training process Figures 4A and 4B show block diagrams of an exemplary training process for a machine learning-trained language model 120 according to an exemplary embodiment of the present disclosure. As shown in Figure 4A, the computing system can acquire a training tuple 430. The training tuple 430 may include a natural language instruction 450, a set of pre-edited text 434, and a set of edited text 432.

[0058] The natural language instructions 450 and the pre-edited text 432 may be provided as input to the machine learning language model 120. Accordingly, the machine learning language model 120 can generate a set of predicted texts 460.

[0059] The computing system can evaluate a loss function 470 that generates a loss value based on a comparison between the edited text 432 and the predicted text 460. Based on the loss function 470, the computing system can update one or more parameter values ​​of the machine learning language model 120. For example, the loss function 470 can be backpropagated through the machine learning language model 120.

[0060] The training tuple 430 can be created from several different sources. For example, the training tuple 430 can be created from annotation and edit history from the same text generation application or other productivity tool on which the machine learning model 120 is deployed. For instance, the annotation and edit history may include, or be derived from, a record or log of comments added, edited, resolved, or deleted in several documents during a collaborative process. The history may include a record of comments and responsive edits and / or responsive comments.

[0061] As another example, training tuple 430 can be created from edit histories from a large text corpus, such as the edit history of an online collaboratively edited encyclopedia. For each edit in the text corpus's edit history, an edit summary may be available. This summary may be included in training tuple 430 as a natural language instruction 450. The edit summary may be included in training tuple 430 as is, or it may be modified or improved by applying human changes or edits made by a separate language model.

[0062] More generally, some pre- or post-processing techniques can also be performed on the training tuple 430 to improve the performance of the language model 120 trained on it. This can generate a training tuple 430 that enables the language model 120 to be trained to perform generalized text editing of long-form text content.

[0063] Figure 4B shows a training approach similar to the approach in Figure 4A. However, in Figure 4B, the training tuple 430 further includes the ground truth response 480. Furthermore, model 120 can generate a predicted response 490. The loss function 470 can further compare the ground truth response 480 with the predicted response 490. A single loss function 470 is used, but in other cases, multiple different loss functions may be used.

[0064] Exemplary Method Figure 5 shows a flowchart of a method 500 for training one or more machine learning models according to an aspect of the present disclosure. For example, an exemplary machine learning model may include a language model such as a large language model or a large multimodal model that can handle language and one or more additional modalities.

[0065] One or more parts of Exemplary Method 500 may be implemented by a computing system including one or more computing devices, such as the computing system described with reference to other figures. Each part of Exemplary Method 500 can be implemented by any one (or any combination) of one or more computing devices. Furthermore, one or more parts of Exemplary Method 500 may be implemented in the hardware components of the device(s) described herein, for example, to train one or more systems or models. Figure 5 shows the elements to be implemented in a particular order for illustrative and explanatory purposes. Those skilled in the art will understand, by using the disclosure provided herein, that any element of the methods described herein can be adapted, rearranged, extended, omitted, combined, or modified in various ways without departing from the scope of this disclosure. Figure 5 is illustrated for illustrative purposes only, with reference to elements / terms described in relation to other systems and drawings, and is not intended to limit it. One or more parts of Exemplary Method 500 may be implemented by other systems, additionally or alternatively.

[0066] In 502, Exemplary Method 500 may include obtaining training instances. A training dataset may include multiple training instances that are split across multiple datasets (e.g., a training dataset, a validation dataset, or a test dataset). Training instances may or may not be labeled. Although referred to as “training” instances in Exemplary Method 500, it should be understood that a training instance can also be formed if runtime estimation trains the model using an evaluation of the model’s performance against that runtime instance (e.g., online training / learning). Exemplary data types for training instances and various tasks associated therewith are described throughout this disclosure.

[0067] In 504, an exemplary method 500 may include processing a training instance using one or more machine-trained models to produce an output. The output may be obtained directly from one or more machine-trained models, or it may be the final result of a chain of processing operations that include the outputs of one or more machine-trained models.

[0068] In 506, exemplary method 500 may include receiving an evaluation signal associated with an output. The evaluation signal may be obtained using a loss function. Various methods for determining the loss can be used, such as mean squared error, likelihood loss, cross-entropy loss, hinge loss, contrastive loss, or various other loss functions. The evaluation signal can be computed using known ground truth labels (e.g., supervised learning), predicted or estimated labels (e.g., semi-supervised or self-supervised learning), or unlabeled (e.g., unsupervised learning). The evaluation signal may be a reward (e.g., for reinforcement learning). The reward can be computed using a machine-learned reward model configured to generate a reward based on the received output(s). The reward may be computed using feedback data describing human feedback on the output(s). In other examples, the reward model may provide a reward based on the edit ratio between input and output and / or the natural language inference score between input and output. In another example, a reward model may be trained to predict rewards based on training data that includes a training quality score generated based on the edit ratio between inputs and outputs and / or the natural language inference score between inputs and outputs.

[0069] In 508, exemplary method 500 may include updating a machine-trained model using an evaluation signal. For example, the parameter values ​​of a machine-trained model(s) may be learned using various training or learning techniques, such as backpropagation, in some embodiments. For example, an evaluation signal may be backpropagated from the output (or another source of the evaluation signal) through the machine-trained model(s) to update one or more parameters of the model(s) (for example, based on the gradient of the parameter(s) of the evaluation signal). For example, a system(s) containing one or more machine-trained models may be trained in an end-to-end manner. Gradient descent can be used to iteratively update parameters over several training iterations. In some embodiments, performing backpropagation may include performing time-truncate backpropagation. Exemplary method 500 includes implementing several generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization ability of the model being trained.

[0070] In some embodiments, exemplary method 500 may be implemented to train a machine learning model from an initialized state to a fully trained state (for example, when the model exhibits a desired performance profile based on accuracy, precision, recall, etc.).

[0071] In some embodiments, exemplary method 500 may be implemented for a specific stage of the training procedure. For example, in some embodiments, exemplary method 500 may be implemented to pre-train a machine learning model. Pre-training may include, for example, large-scale training on potentially noisy data to achieve a broad base of performance levels across various task / data types. In some embodiments, exemplary method 500 may be implemented to fine-tune a machine learning model. Fine-tuning may include, for example, smaller-scale training on higher-quality data (e.g., labeled, curated, etc.). Fine-tuning may affect all or some of the parameters of the machine learning model. For example, different parts of the machine learning model may be “frozen” during a particular training stage. For example, parameters associated with the embedding space may be “frozen” during fine-tuning (e.g., to retain information learned from a broader domain than present in the fine-tuning dataset). Exemplary fine-tuning techniques include reinforcement learning. Reinforcement learning may be based on user feedback on the model's performance in use.

[0072] Exemplary machine learning model Figure 6 is a block diagram of an exemplary processing flow for processing input(s) 2 using machine learning model(s) 1 to produce output(s) 3.

[0073] A machine learning model(s) may be one or more machine learning models or model components, or may include them. An exemplary machine learning model may include a neural network (e.g., a deep neural network). An exemplary machine learning model may include a nonlinear or linear model. An exemplary machine learning model may use other architectures instead of, or in addition to, a neural network. An exemplary machine learning model may include decision tree-based models, support vector machines, hidden Markov models, Bayesian networks, linear regression models, k-means clustering models, etc.

[0074] Exemplary neural networks can include feedforward neural networks, recurrent neural networks (RNNs) including long-short-term memory (LSTM) based recurrent neural networks, convolutional neural networks (CNNs), distributed models, generative adversarial networks, or other forms of neural networks. Exemplary neural networks can also be deep neural networks. Some exemplary machine-learned models can utilize attention mechanisms such as self-attention. For example, some exemplary machine-learned models can include multi-head self-attention models.

[0075] A machine learning model(s) 1 may include one or more instances of the same model configured to work on data from input(s) 2. A machine learning model(s) 1 may include an ensemble of different models that can cooperate and interact to process data from input(s) 2. For example, a machine learning model(s) 1 may employ a mixed-expert structure. See, for example, Zhou et al., Mixture-of-Experts with Expert Choice Routing, ARXIV:2202.09368v2 (October 14, 2022).

[0076] Input(s) 2 can generally contain or represent various types of data. Input(s) 2 can contain one type or many different types of data. Output(s) 3 can contain the same type(s) of data as Input(s) 2 or different types of data. Output(s) 3 can contain one type or many different types of data.

[0077] Exemplary data types for input(s)2 or output(s)3 include natural language text data, software code data (e.g., source code, object code, machine code, or other forms of computer-readable instructions or programming languages), machine code data (e.g., binary code, assembly code, or other forms of machine-readable instructions that can be directly executed by a computer's central processing unit), assembly code data (e.g., low-level programming languages ​​that program a processing unit using symbolic representations of machine code instructions), genetic data or other chemical or biochemical data, image data, audio data, audiovisual data, tactile data, biometric data, medical data, financial data, statistical data, geographic data, astronomical data, historical data, and general sensor data (e.g., digital or analog values ​​such as voltage or other absolute or relative level measurements from real or artificial inputs such as audio sensors, light sensors, and displacement sensors). Data may be raw or processed, and may be in any format or schema.

[0078] In multimodal input 2 or output 3, exemplary combinations of data types include image data and audio data, image data and natural language data, natural language data and software code data, image data and biometric data, sensor data and medical data, etc. Please understand that any combination of data types is possible in input 2 or output 3.

[0079] An exemplary input 2 may include one or more data types, such as the exemplary data types described above. An exemplary output 3 may include one or more data types, such as the exemplary data types described above. The data types of input 2 may be the same as or different from the data types of output 3. It should be understood that the exemplary data types described above are provided for illustrative purposes only. The data types contemplated within the scope of this disclosure are not limited to the examples given above. Exemplary machine learning-based sequence processing models

[0080] Figure 7 is a block diagram of an exemplary implementation of an exemplary machine learning model configured to process a sequence of information. For example, an exemplary embodiment of machine learning model 1 may include machine learning sequence processing model 4. The exemplary system can pass input 2 to sequence processing model 4. Sequence processing model 4 may include one or more machine learning components. Sequence processing model 4 can process data from input 2 to obtain input array 5. Input array 5 may include one or more input elements 5-1, 5-2, ..., 5-M, etc., obtained from input 2. Sequence processing model 4 can process input sequence 5 using prediction layer 6 to generate output sequence 7. Output sequence 7 may include one or more output elements 7-1, 7-2, ..., 7-N, etc., generated based on input sequence 5. The system may generate output 3 based on output sequence 7.

[0081] A sequence processing model (or multiple models) 4 may include one or more machine learning model components configured to ingest, generate, or otherwise infer sequences of information. For example, some exemplary sequence processing models in the text domain are referred to as “Large-Scale Language Models,” or LLMs. See, for example, the PaLM 2 Technical Report, GOOGLE, https: / / ai.google / static / documents / palm2techreport.pdf (nd). Other exemplary sequence processing models can operate in other domains, such as the image domain. For example, see Dosovitskiy et al., An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, ARXIV:2010.11929v2 (June 3, 2021); for the speech domain, see Agostinelli et al., MusicLM: Generating Music From Text, ARXIV:2301.11325v1 (January 26, 2023); and for the biochemistry field, see Jumper et al., Highly accurate protein structure prediction with AlphaFold, 596 Nature 583 (August 26, 2021). The sequencing model(s)4 may process one or more types of data simultaneously. The sequencing model(s)4 may include relatively large models (e.g., more parameters, higher computational cost), relatively small models (e.g., fewer parameters, lighter computation), or both.

[0082] Generally, a sequence processing model 4 can obtain an input array 5 using data from input 2. For example, input array 5 may contain a representation of the data from input 2 in a format understood by the sequence processing model 4. One or more machine learning-trained components of the sequence processing model 4 can take data from input 2, parse the data (e.g., via "tokenization") in a part compatible with the processing architecture of the sequence processing model 4, and project the part (e.g., via "embedding") into the input space related to the prediction layer 6.

[0083] Sequence processing model 4 can take data from input 2, parse the data into a sequence of elements, and obtain input sequence 5. For example, a portion of the input data from input 2 can be decomposed into parts that collectively represent the content of a portion of the input data. These parts may provide elements of an array.

[0084] Elements 5-1, 5-2, ..., 5-M can, in some cases, represent constituent units for capturing or representing meaningful information in a particular data domain. For example, an element can describe an “atomic unit” spanning one or more domains. For instance, in the case of a text input source(s), an element might correspond to a group of one or more word or subword components, such as one or more sets of characters.

[0085] For example, elements 5-1, 5-2, ..., 5-M may represent tokens obtained using a tokenizer. For example, a tokenizer may process a given portion of an input source and output a set of tokens representing that portion of the input source (e.g., corresponding to input elements 5-1, 5-2, ..., 5-M). Various approaches to tokenization can be used. For example, a text input source(s) may be tokenized using byte-pair coding (BPE) techniques. See, for example, Kudo et al., SentencePiece: A simple and language-independent subword tokenizer and detokenizer for Neural Text Processing, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (System Demonstrations), pages 66-71 (October 31-November 4, 2018), https: / / aclanthology.org / D18-2012.pdf. Image-based input sources can be tokenized by extracting and serializing patches from the images.

[0086] In general, any data type can be serialized and processed into input sequence 5. It should be understood that the elements 5-1, 5-2, ..., 5-M shown in Figure 7 may be tokens or their embedded representations.

[0087] The prediction layer(s) 6 can predict one or more output elements 7-1, 7-2, ..., 7-N based on the input elements. The prediction layer(s) 6 may include one or more trained machine learning model architectures, such as one or more layers of trained parameters that manipulate and transform the input(s), extracting higher-order meanings and relationships between the input(s) 5-1, 5-2, ..., 5-M. In this way, for example, the exemplary prediction layer(s) 6 can predict new output(s) taking into account the context provided by the input sequence 5.

[0088] The predictive layer(s) 6 can evaluate associations between parts of the input sequence 5 and specific output elements. These associations can provide information for predicting the likelihood that a particular output will follow the input context. For example, consider the text excerpt, "The carpenter's toolbox was small and heavy. It was full of ___." The example predictive layer(s) 6 can identify that "It" refers to "toolbox" by determining the relationships between each embedding. The example predictive layer(s) 6 can also link "It" to attributes of the toolbox such as "small" and "heavy." Based on these associations, the predictive layer(s) 6 can assign a higher probability to the word "nails" than to the word "sawdust," for example.

[0089] A transformer is an exemplary architecture that can be used in a prediction layer 4(or more). See, for example, Vaswani et al., Attention Is All You Need, ARXIV:1706.03762v7 (August 2, 2023). A transformer is an example of a machine learning model architecture that uses an attention mechanism to compute associations between items in a context window. A context window can include an input sequence 5 and a sequence that may contain one or more output elements 7-1, 7-2, ..., 7-N. A transformer block can include one or more attention layers and one or more post-attention layers (e.g., feedforward layers such as multilayer perceptrons).

[0090] The prediction layer(s) 6 may include, in addition to or instead of, a converter-based architecture, other machine learning model architectures. For example, not only convolutional neural networks (CNNs), but also recurrent neural networks (RNNs) and long-term short-term memory (LSTM) models may be used. In general, the prediction layer(s) 6 can leverage various types of artificial neural networks that can understand or generate sequences of information.

[0091] Output sequence 7 may contain the same or different data types as input sequence 5, or represent them in other ways. For example, input sequence 5 may represent text data, and output sequence 7 may represent text data. Input sequence 5 may represent image, audio, or audiovisual data, and output sequence 7 may represent text data (e.g., describing image, audio, or audiovisual data). It will be understood that any other intervening model component of the prediction layer 6 and the sequence processing model 4 may be configured to receive various data types of input sequence 5 and output various data types of output sequence 7.

[0092] Output sequence 7 can have various relationships with input sequence 5. Output sequence 7 can be a continuation of input sequence 5. Output sequence 7 can be complementary to input sequence 5. Output sequence 7 can translate, transform, extend, or otherwise modify input sequence 5. Output sequence 7 can answer, evaluate, confirm, or otherwise respond to input sequence 5. Output sequence 7 can implement (or write instructions to implement) instructions provided via input sequence 5.

[0093] The output sequence 7 can be generated autoregressively. For example, in some applications, the output of one or more prediction layers 6 can pass through one or more output layers (e.g., softmax layers) to obtain a probability distribution of output vocabulary (e.g., text vocabulary or symbolic vocabulary) conditioned on a set of input elements in a context window. In this way, the output sequence 7 can be generated autoregressively, for example, by sampling the elements most likely to be output next, adding those elements to the context window, regenerating the probability distribution based on the updated context window, sampling the elements most likely to be output next, and so on.

[0094] Output sequence 7 can also be generated non-autoregressively. For example, multiple output elements of output sequence 7 can be predicted together without explicit sequential conditioning of each other. See, for example, Saharia et al., Non-Autoregressive Machine Translation with Latent Alignments, ARXIV:2004.07437v3 (November 16, 2020).

[0095] Output sequence 7 can contain one or more parts or elements. In an exemplary content generation configuration, output sequence 7 can contain multiple elements (e.g., sentences of text, discretized waveform values, computer code, etc.) corresponding to multiple parts of the generated output sequence. In an exemplary classification configuration, output sequence 7 can contain a single element associated with the classification output. For example, the output "Vocabulary" can contain a set of classes to which the input sequence is classified. For example, a visual transducer block can pass latent state information to a multilayer perceptron that outputs the most likely class values ​​associated with the input image.

[0096] Figure 8 is a block diagram of an exemplary technique for filling an exemplary input sequence 8. The input sequence 8 may include various functional elements that constitute part of the model infrastructure, such as element 8-0 obtained from a task indicator 9 that signals that a particular task is being performed (for example, to help fit the performance of the model(s) to that particular task) in any model(s) that processes the input sequence 8. The input sequence 8 may include various data elements from different data modalities. For example, input modality 10-1 may include one modality of data. The data-versus-sequence model 11-1 may process the data from input modality 10-1 and project that data into a format compatible with input sequence 8 (for example, one or more vectors dimensioned according to the dimensions of input sequence 8) to obtain elements 8-1, 8-2, and 8-3. Other input modalities 10-2 may include different data modalities. The data-to-sequence model 11-2 can project data from input modality 10-2 into a format compatible with input sequence 8, obtaining elements 8-4, 8-5, and 8-6. Other input modalities 10-3 may include yet other different data modalities. The data-to-sequence model 11-3 can project data from input modality 10-3 into a format compatible with input sequence 8, obtaining elements 8-7, 8-8, and 8-9.

[0097] Input sequence 8 may be the same as or different from input sequence 5. Input sequence 8 can be a multimodal input sequence containing elements that represent data from different modalities using a common dimensional representation. For example, the embedding space may have P dimensions. Input sequence 8 can be configured to contain multiple elements having P dimensions. In this way, for example, exemplary embodiments can facilitate information extraction and inference across diverse data modalities by projecting data onto elements in the same embedding space for comparison, combination, or other computations between them.

[0098] For example, elements 8-0, ..., 8-9 can represent specific locations within a multidimensional embedding space. Some elements can be mapped to a discrete set of locations within the embedding space. For instance, elements corresponding to individual members of a given vocabulary of tokens can be mapped to discrete locations within the embedding space associated with those tokens. Other elements can be contiguously distributed throughout the embedding space. For example, some data types can be decomposed into contiguously defined parts (e.g., image patches) that can be described using contiguously distributed locations within the embedding space.

[0099] In some embodiments, the expressiveness of the embedding space is not limited to the meaning associated with any particular set of tokens or other constituent units. For example, a continuous embedding space can encode a spectrum of higher-order information. Individual pieces of information (e.g., tokens) can be mapped to specific points within that space; for example, the token for the word "dog" can be projected onto an embedding value that points to a specific location within the embedding space associated with dog-related information. Similarly, an image patch of a dog on grass can also be projected onto the embedding space. In some embodiments, the projection of the dog image may be similar to the projection of the word "dog," while also having similarity to the projection of the word "grass," and at the same time, both may be different. In some embodiments, the projection of the image patch may not exactly match any single projection of either of the single words. In some embodiments, the projection of the image patch can match a combination of projections of the words "dog" and "grass." In this way, for example, a higher-order embedding space can encode information that can be independent of the data modality in which the information is represented.

[0100] The task indicator 9 may include a model or model component configured to identify the task being executed and inject input values ​​represented by elements 8-0, which signal which task is being executed, into the input sequence 8. For example, the input values ​​may be provided as a data type associated with an input modality and projected along that input modality (e.g., the input values ​​may be text task labels embedded in the input along with other text data, or pixel-based representations of tasks embedded in the input along with other image data, etc.). The input values ​​may be provided as a data type that is different from, or at least independent of, other inputs(s). For example, the input values ​​represented by elements 8-0 may be learned in a contiguous embedding space.

[0101] Input modalities 10-1, 10-2, and 10-3 can be associated with various different data types (for example, as described above for input(s) 2 and output(s) 3).

[0102] The data-to-sequence models 11-1, 11-2, and 11-3 may be the same or different from each other. The data-to-sequence models 11-1, 11-2, and 11-3 can be adapted to their respective input modalities 10-1, 10-2, and 10-3. For example, a text data-to-sequence model can subdivide a portion of the input text and project the subdivided portion onto an element(s) in the input sequence 8 (e.g., elements 8-1, 8-2, 8-3, etc.). An image data-to-sequence model can subdivide an input image and project the subdivided portion onto an element(s) in the input sequence 8 (e.g., elements 8-4, 8-5, 8-6, etc.). A data-to-sequence model for any data type can subdivide the input of that data type and project the subdivided portion onto an element(s) in the input sequence 8 (e.g., elements 8-7, 8-8, 8-9, etc.).

[0103] The data-to-sequence models 11-1, 11-2, and 11-3 can form part of a machine learning-based sequence processing model 4(or more). The data-to-sequence models 11-1, 11-2, and 11-3 can be trained in conjunction with the machine learning-based sequence processing model 4(or more) or independently of it. The data-to-sequence models 11-1, 11-2, and 11-3 can be trained end-to-end using the machine learning-based sequence processing model 4(or more).

[0104] Exemplary machine learning model development platform Figure 9 is a block diagram of an exemplary model development platform 12 that can facilitate the creation, adaptation, and improvement of exemplary machine learning models (e.g., machine learning model 1, sequence processing model 4, etc.). The model development platform 12 can provide several different toolkits that developer systems can employ in the development of new or adapted machine learning models.

[0105] The model development platform 12 may provide one or more model libraries 13 containing components for new models. The model library 13 may include one or more pre-trained foundational models 13-1 that can provide a backbone of processing power across various tasks. The model library 13 may also include one or more pre-trained expert models 13-2 that can focus on performance in a specific domain of expertise. The model library 13 may also include various model primitives 13-3 that can provide low-level architectures or (optionally pre-trained) components that can be assembled into various configurations as needed.

[0106] The model development platform 12 can receive a selection of various model components 14. The model development platform 12 can pass the selected model components 14 to the workbench 15, which then incorporates the selected model components 14 into the development model 16.

[0107] The workbench 15 can facilitate further refinement and adaptation of the development model 16 by leveraging several different toolkits integrated with the model development platform 12. For example, the workbench 15 can facilitate the alignment of the development model 16 to a desired performance profile for various tasks by using the model alignment toolkit 17.

[0108] The model alignment toolkit 17 can provide several tools for causing the development model 16 to produce aligned outputs with desired behavioral characteristics. Alignment may include improving the accuracy, precision, recall, etc., of the model output. Alignment may include enforcing an output style, schema, or other preferred characteristics of the model output. Alignment may be general or domain-specific. For example, a pre-trained base model 13-1 can start with an initial level of performance across multiple domains. Alignment of the pre-trained base model 13-1 may include improving performance in a specific domain of information or task (for example, at the expense of performance in other domains of information or task).

[0109] The model alignment toolkit 17 can integrate one or more datasets 17-1 for aligning the development model 16. The selected datasets 17-1 may include labeled or unlabeled training data. The datasets 17-1 may be obtained from public domain datasets. Alternatively, the datasets 17-1 may be obtained from private datasets associated with one or more developer systems for aligning special machine learning models customized for private use cases.

[0110] The pre-training pipeline 17-2 may include a machine learning model training workflow configured to update the development model 16 across a large and potentially noisy dataset. For example, pre-training may utilize unsupervised learning techniques (e.g., denoising) to process a large number of training instances to update model parameters from an initialized state and achieve the desired baseline performance. The pre-training pipeline 17-2 may perform pre-training using an unlabeled dataset of dataset 17-1. The workbench 15 may implement the pre-training pipeline 17-2 to pre-train the development model 16.

[0111] The fine-tuning pipeline 17-3 may include a machine learning model training workflow configured to improve the model parameters of the development model 16 with higher quality data. The fine-tuning pipeline 17-3 can update the development model 16 by performing supervised training using the labeled dataset(s) of dataset(s) 17-1. The fine-tuning pipeline 17-3 can update the development model 16 by performing reinforcement learning using reward signals from user feedback signals. The workbench 15 can implement the fine-tuning pipeline 17-3 to fine-tune the development model 16.

[0112] The prompt library 17-4 may include a set of inputs configured to elicit behavior aligned to a desired performance criterion. The prompt library 17-4 may include short-shot prompts (e.g., inputs that provide examples of desired model outputs to prepend to a desired runtime query), thought chain prompts (e.g., inputs that provide step-by-step reasoning in examples to facilitate thorough reasoning by the model), and so on.

[0113] Exemplary prompts can be obtained from the available repositories of the prompt library 17-4. Exemplary prompts may be provided by one or more developer systems using the workbench 15.

[0114] In some embodiments, a pre-trained or fine-tuned model can achieve satisfactory performance even when the input lacks examples. For example, a zero-shot prompt may contain an input lacking examples. A zero-shot prompt may be located inside or outside the training domain(s) within the training dataset.

[0115] The prompt library 17-4 may include one or more prompt engineering tools. The prompt engineering tools can provide workflows for obtaining or learning optimized prompt values. The prompt engineering tools can facilitate direct learning of prompt values ​​(e.g., input element values) based on one or more training iterations. The workbench 15 can implement the prompt engineering tools in the development model 16.

[0116] The prompt library 17-4 can include a pipeline for prompt generation. For example, inputs can be generated using the development model 16 itself or other machine learning models. In this way, for example, the first model can process information about a task and output an input that the second model processes to carry out the steps of the task. The second model may be the same as or different from the first model. The workbench 15 can implement the prompt generation pipeline in the development model 16.

[0117] The prompt library 17-4 may include a pipeline for context injection. For example, the performance of the development model 16 for a particular task can be improved if additional context is provided for performing that task. The prompt library 17-4 may include a software component configured to identify a desired context, retrieve the context from an external source (e.g., a database, a sensor, etc.), and add the context to the input prompt. The workbench 15 can implement a pipeline for context injection in the development model 16.

[0118] While the various training examples described herein with respect to the model development platform 12 refer to "pre-training" and "fine-tuning," it should be understood that the model alignment toolkit 17 can generally support a variety of training techniques adapted for training various machine learning models. Exemplary training techniques can correspond to the aforementioned exemplary training methods 500.

[0119] The model development platform 12 may include a model plugin toolkit 18. The model plugin toolkit 18 may include a variety of tools configured to extend the functionality of machine-learned models by integrating them with other systems, devices, and software components. For example, machine-learned models can use tools as needed to improve the quality of their performance. For instance, deterministic tasks can be offloaded to specialized tools instead of probabilistically performing tasks with a higher risk of error. For example, instead of autoregressively predicting the solution to a system of equations, a machine-learned model can recognize which tool to call to obtain the solution and pass the system of equations to the appropriate tool. The tool could be a conventional equality system solver that can act deterministically to solve the system of equations. The output of the tool may be returned in response to the original query. In this way, by using tools, some exemplary models may be able to enhance their performance by focusing on the advantages of machine-learned models, such as understanding the intent of unstructured requests to tasks, while at the same time offloading specific tasks to more centralized tools for mechanically applying deterministic algorithms to well-defined problems.

[0120] The Model Plugin Toolkit 18 may include a validation tool 18-1. The validation tool 18-1 may include a tool that can analyze and verify the output(s) of a machine learning model. The validation tool 18-1 may include engineered heuristics that establish specific thresholds to apply to the model output. For example, the validation tool 18-1 may justify the output of a machine learning model to a structured data source (e.g., to mitigate hallucination).

[0121] The model plugin toolkit 18 may include a tool package 18-2 for implementing one or more tools, which may include scripts or other executable code that can be run alongside the development model 16. The tool package 18-2 may include one or more inputs configured to cause a machine-trained model(s) to implement the tool (e.g., a few-shot prompts that guide the model to output tool calls in the appropriate syntax). The tool package 18-2 may include, for example, fine-tuning training data for training the model to use the tool.

[0122] The model plug-in toolkit 18 may include an interface for calling an external application programming interface (API) 18-3. For example, in addition to directly implementing tool calls or tool code in the development model 16, or instead, the development model 16 may be aligned to output instructions that initiate API calls to send or retrieve data via an external system.

[0123] The model plugin toolkit 18 can integrate with the prompt library 17-4 to build a catalog of available tools for use in the development model 16. For example, the model can receive a catalog of available tools as input, and the model can select a tool from the available tools and generate an output that initiates a tool call to use that tool.

[0124] The model development platform 12 may include a computational optimization toolkit 19 for optimizing the computational performance of the development model 16. For example, tools for model compression 19-1 allow for reducing the size of the development model 16 while maintaining a desired level of performance. For example, model compression 19-1 may include quantization workflows, weight pruning, and sparsification techniques. Tools for hardware acceleration 19-2 can facilitate the configuration of model storage and executables to work optimally with different hardware resources. For example, hardware acceleration 19-2 may include tools for optimally sharing the model for distributed processing across multiple processing units due to increased bandwidth, reduced integrated memory requirements, etc. Tools for distillation 19-3 can provide training for a lighter model based on the knowledge encoded in the development model 16. For example, the development model 16 may be a high-performance, large-scale machine learning model optimized using the model development platform 12. To obtain a lightweight model for execution in resource-constrained environments, the smaller model may be a "student model," which learns by mimicking the development model 16 as a "teacher model." In this way, for example, input for learning the parameters and configuration of development model 16 can be efficiently transferred to a smaller model for more efficient inference.

[0125] Workbench 15 may implement one or more of the toolkits implemented in the model development platform 12, or it may not implement any of them. Workbench 15 can output an output model 20 based on the development model 16. The output model 20 may be an expanded version of the development model 16. The output model 20 may be a development or training checkpoint of the development model 16. The output model 20 may be a distilled, compressed, or otherwise optimized version of the development model 16.

[0126] Figure 10 is a block diagram of an exemplary training flow for training a machine learning-prepared development model 16. One or more parts of the exemplary training flow may be implemented by a computing system including one or more computing devices, such as the computing system described with reference to other figures. Each part of the exemplary training flow can be executed by any one (or any combination) of one or more computing devices. Furthermore, one or more parts of the exemplary training flow may be implemented on the hardware components of the device described herein, for example, to train one or more systems or models. Figure 10 shows the elements to be executed in a particular order for illustrative and explanatory purposes. Those skilled in the art will understand, by using the disclosures provided herein, that any element of the methods described herein can be adapted, rearranged, extended, omitted, combined, or modified in various ways without departing from the scope of this disclosure. Figure 10 is described for illustrative purposes with reference to elements / terms described in relation to other systems and drawings, and is not intended to limit it. One or more parts of the exemplary training flow may be implemented by other systems, additionally or alternatively.

[0127] First, the development model 16 may persist in its initial state as the initialized model 21. The development model 16 can be initialized with weight values. The initial weight values ​​can be random or based on an initialization schema. The initial weight values ​​can be based on prior pre-training on the same or different models.

[0128] The initialized model 21 can undergo pre-training in the pre-training stage 22. The pre-training stage 22 can be implemented using one or more pre-training pipelines 17-2 on data from dataset 17-1(or more). For example, if the initialized model 21 is already pre-trained (e.g., the development model 16 includes or is based on a pre-trained foundational or expert model), pre-training can be omitted.

[0129] The pre-trained model 23 can then become a new version of the development model 16, which can persist as development model 16 or as a new development model. The pre-trained model 23 can be set to an initial state if the development model 16 has already been pre-trained. The pre-trained model 23 can be fine-tuned in the fine-tuning stage 24. The fine-tuning stage 24 can be implemented using one or more fine-tuning pipelines 17-3 on data from dataset 17-1(or more). For example, fine-tuning can be omitted if the performance of the pre-trained model is sufficient, if the model has already been fine-tuned, or if other tuning methods are preferred.

[0130] The finely tuned model 29 can then become a new version of the development model 16, which can persist as development model 16 or as a new development model. The finely tuned model 29 can be returned to its initial state if the development model 16 is already finely tuned. The finely tuned model 29 can undergo improvements based on user feedback 26. For example, improvements based on user feedback 26 may include reinforcement learning based on human feedback from human users of the finely tuned model 25, which is of optional nature. Since reinforcement learning can be a form of fine-tuning, it should be understood that the fine-tuning stage 24 may include a stage for improvement with user feedback 26. Improvements based on user feedback 26 may result in the creation of an improved model 27. The improved model 27 can be output to a downstream system 28 for deployment or further development.

[0131] In some embodiments, computational optimization operations may be applied before, during, or after each stage. For example, an initialized model 21 may undergo computational optimization 29-1 (e.g., using the computational optimization toolkit 19) before the pre-training stage 22. A pre-trained model 23 may undergo computational optimization 29-2 (e.g., using the computational optimization toolkit 19) before the fine-tuning stage 24. A fine-tuned model 25 may undergo computational optimization 29-3 (e.g., using the computational optimization toolkit 19) before being improved by user feedback 26. An improved model 27 may undergo computational optimization 29-4 (e.g., using the computational optimization toolkit 19) before outputting to a downstream system 28. Computational optimizations 29-1, ..., 29-4 (or more) may all be the same, all be different, or include at least some different optimization techniques.

[0132] Exemplary machine learning-prepared model inference system Figure 11 is a block diagram of an inference system for operating one or more machine learning models 1 to perform inference (e.g., training and deployment). A model host 31 can receive one or more machine learning models 1. A model host 31 can host one or more model instances 31-1(or more), which may be one or more instances of one or more models. A model host 31 can host model instances 31-1(or more) using available compute resources 31-2 associated with the model host 31.

[0133] The model host 31 can perform inference on behalf of one or more clients 32. Clients 32 can send input requests 33 to the model host 31. Using the input requests 33, the model host 31 can obtain inputs 2 for input to a pre-trained model 1. The pre-trained model 1 can process inputs 2 to produce outputs 3. Using outputs 3, the model host 31 can return an output payload 34 to respond to the input requests 33 from the clients 32. The output payload 34 may contain or be based on outputs 3.

[0134] The model host 31 can extend its inference tasks by utilizing various other resources and tools. For example, the model host 31 can communicate with a tool interface 35 to facilitate the use of tools by model instances 31-1(1) or more. The tool interface 35 may include local or remote APIs. The tool interface 35 may include integrated scripts or other software functions. The model host 31 can involve online learning interfaces 36(36) or more to facilitate the continuous improvement of the machine-learned model(1) or more. For example, the online learning interfaces 36(36) or more can be used within a reinforcement learning loop to obtain user feedback on the inferences provided by the model host 31. The model host 31 can access runtime data sources(37) or more to extend the input(2) or more with additional contextual information. For example, the runtime data sources(37) or more may include a knowledge graph 37-1 that facilitates the retrieval of structured information for information associated with input requests(33) or more (e.g., a search engine service). The runtime data source(s) 37 may include public or private, external or local database(s) 37-2 that can store information associated with input requests(s) 33 to extend the input(s) 2. The runtime data source(s) 37 may also include account data(s) 37-3, which is retrieved in association with a user account corresponding to a client(s) 32, and can be used to customize the behavior of the model host(s) 31 accordingly.

[0135] Model host 31 can be implemented by one or more computing devices or systems. Client(s) 2 can be implemented by one or more computing devices or systems, which may include computing devices or systems shared with model host 31.

[0136] For example, the model host 31 may operate as a server system that provides machine learning services to client devices 32 that run on it (for example, via a local or wide area network). Client devices 32 may be end-user devices used by individuals. Client devices 32 may be server systems that run on them and provide various functions as services to downstream end-user devices.

[0137] In some embodiments, the model host 31 can operate on the same device or system as the client(s) 32. The model host 31 may be a machine learning service that runs on the device and provides machine learning capabilities to one or more applications running on the client device, which may include the application implementation client(s) 32. The model host 31 may be part of the same application as the client(s) 32. For example, the model host 31 may be a subroutine or method implemented by one part of the application, and the client(s) 32 may be other subroutines or methods that engage with the model host 31 to perform inference capabilities within the application. It should be understood that the model host 31 and the client(s) 32 can have a variety of different configurations.

[0138] A model instance 31-1(or more) can contain one or more machine-trained models available for performing inference. A model instance 31-1(or more) can contain weights or other model components that are stored in persistent storage, temporarily cached, or loaded into high-speed memory. A model instance 31-1(or more) can contain multiple instances(or more) of the same model (for example, to run more requests in parallel on the same model). A model instance 31-1(or more) can contain instances(or more) of different models(or more). A model instance 31-1(or more) can contain cached intermediate states of active or inactive models(or more) used to speed up inference for those models. For example, an inference session with a particular model can generate a significant amount of computational results that can be reused for future inference runs (for example, using a KV cache for a converter-based model). These computational results can be stored associated with that inference session so that the session can run more efficiently when it is resumed.

[0139] Computational resources 31-2 may include one or more processors (such as a central processing unit, a graphical processing unit, a tensor processing unit, or a machine learning accelerator) connected to one or more memory devices. Computational resources 31-2 may include a dynamic pool of available resources shared with other processes. Computational resources 31-2 may include a memory device large enough to accommodate an entire model instance in a single memory instance. Computational resources 31-2 may also share a model instance across multiple memory devices (for example, using data parallelism or tensor parallelism). This may be done to enhance parallelism or to run large models using multiple memory devices where the entire model cannot fit in memory on individual memory devices.

[0140] Input request 33 may contain data for input(s) 2. Model host 31 can process input request 33 to retrieve input(s) 2. Input(s) 2 can be retrieved directly from input request 33 or by using input request 33. Input request 33 may be submitted to model host 31 via API.

[0141] The model host 31 can perform inference in parallel across batches of input requests 33. For example, a model instance 31-1 can consist of an input structure having a batch dimension. Separate inputs 2 can be distributed across the batch dimension (e.g., rows of an array). Separate inputs 2 can contain completely different contexts. Separate inputs 2 can be multiple inference steps of the same task. Separate inputs 2 can be arranged alternately within the input structure, so that any given inference cycle can act on different parts of each input 2. In this way, for example, the model host 31 can perform inference in batches in parallel, and as a result, the output 3 can also contain the batch dimension and return the inference results of the batched inputs 2 in parallel. In this way, for example, batches of input requests 33 can be processed in parallel for higher throughput of the output payload 34.

[0142] The output payload 34 may contain or be based on the output(s) 3 from machine learning model(s) 1. The model host 31 can process the output(s) 3 to obtain the output payload 34. This may involve chaining multiple inferences (e.g., iteratively, recursively, across the same model(s) or different models(s)) to arrive at the final output of the task returned in the output payload 34. The output payload 34 can be sent to client(s) 32 via the API.

[0143] The online learning interface(s) 36 can facilitate reinforcement learning of the machine learning model(s) 1. The online learning interface(s) 36 can facilitate reinforcement learning with human feedback (RLHF). The online learning interface(s) 36 can facilitate federated learning of the machine learning model(s) 1.

[0144] The model host 31 can run a pre-trained model 1 to perform inference on various tasks using various types of data. For example, various different inputs 2 and outputs 3 can be used for various different tasks. In some embodiments, input 2 may be image data or can be represented in other ways. The pre-trained model 1 can process image data to generate outputs. For example, the pre-trained model 1 can process image data to generate image recognition outputs (e.g., recognition of image data, latent embedding of image data, coded representation of image data, hash of image data, etc.). As another example, the pre-trained model 1 can process image data to generate image segmentation outputs. As yet another example, the pre-trained model 1 can process image data to generate image classification outputs. As yet another example, the pre-trained model 1 can process image data to generate image data modification outputs (e.g., modification of image data, etc.). As another example, machine learning model(s) 1 can process image data to generate encoded image data output (e.g., encoded and / or compressed representations of the image data). As yet another example, machine learning model(s) 1 can process image data to generate upscaled image data output. As yet another example, machine learning model(s) 1 can process image data to generate predictive output.

[0145] In some embodiments, the task is a computer vision task. In some cases, input 2 includes pixel data from one or more images, and the task is an image processing task. For example, the image processing task may be image classification, and the output is a set of scores, each corresponding to a different object class, representing the likelihood that one or more images depict an object belonging to that object class. The image processing task may be object detection, and the image processing output identifies one or more regions within one or more images, and for each region, the likelihood that the region depicts an object of interest. As another example, the image processing task may be image segmentation, and the image processing output defines, for each pixel in one or more images, the likelihood for each category within a given set of categories. For example, the set of categories may be foreground and background. As yet another example, the set of categories may be object classes. As yet another example, the image processing task may be depth estimation, and the image processing output defines, for each pixel in one or more images, the respective depth value. As another example, the image processing task could be motion estimation, where the network input includes multiple images, and the image processing output defines the motion of the scene depicted between the images in the network input, for each pixel in the input images.

[0146] In some embodiments, input 2 can be natural language data or represent natural language data in other ways. A machine learning model 1 can process the natural language data to produce an output. For example, machine learning model 1 can process natural language data to produce a language coding output. Another example is machine learning model 1 can process natural language data to produce a latent text embedding output. Another example is machine learning model 1 can process natural language data to produce a translation output. Another example is machine learning model 1 can process natural language data to produce a classification output. Another example is machine learning model 1 can process natural language data to produce a text segmentation output. Another example is machine learning model 1 can process natural language data to produce a semantic intent output. As another example, a machine learning model(s)1 can process natural language data to produce upscaled text or natural language output (e.g., text or natural language data of higher quality than the input text or natural language). As yet another example, a machine learning model(s)1 can process natural language data to produce predictive output (e.g., one or more predicted next parts of natural language content).

[0147] In some embodiments, input 2 may be speech data (e.g., audio data, text data, or other data describing spoken natural language) or may be represented in other ways. A machine learning model 1 can process the speech data to produce an output. For example, machine learning model 1 can process the speech data to produce a speech recognition output. Another example is machine learning model 1 can process the speech data to produce a speech translation output. Another example is machine learning model 1 can process the speech data to produce a latent embedding output. Another example is machine learning model 1 can process the speech data to produce an encoded speech output (e.g., an encoded representation and / or compressed representation of the speech data). Another example is machine learning model 1 can process the speech data to produce an upscaled speech output (e.g., speech data of higher quality than the input speech data). As another example, machine learning model(s) 1 can process speech data to generate a textual representation output (e.g., a textual representation of the input speech data). As yet another example, machine learning model(s) 1 can process speech data to generate a predictive output.

[0148] In some embodiments, input 2 may be latent coded data (e.g., a latent spatial representation of the input) or may represent the latent coded data in other ways. A machine learning model 1 can process the latent coded data to produce an output. For example, machine learning model 1 can process the latent coded data to produce a recognition output. As another example, machine learning model 1 can process the latent coded data to produce a reconstruction output. As yet another example, machine learning model 1 can process the latent coded data to produce a search output. As yet another example, machine learning model 1 can process the latent coded data to produce a reclustering output. As yet another example, machine learning model 1 can process the latent coded data to produce a prediction output.

[0149] In some embodiments, input 2 may be statistical data or represent statistical data in other ways. Statistical data may be data computed and / or calculated from some other data source, or may represent it, or may otherwise include it. A machine learning model 1 may process the statistical data to produce an output. For example, machine learning model 1 may process the statistical data to produce a recognition output. Another example is machine learning model 1 may process the statistical data to produce a prediction output. Another example is machine learning model 1 may process the statistical data to produce a classification output. Another example is machine learning model 1 may process the statistical data to produce a segmentation output. Another example is machine learning model 1 may process the statistical data to produce a visualization output. Another example is machine learning model 1 may process the statistical data to produce a diagnostic output.

[0150] In some embodiments, input 2 can be sensor data or represent sensor data in other ways. A machine learning model 1 can process the sensor data to generate an output. For example, machine learning model 1 can process sensor data to generate a recognition output. Another example is machine learning model 1 can process sensor data to generate a prediction output. Another example is machine learning model 1 can process sensor data to generate a classification output. Another example is machine learning model 1 can process sensor data to generate a segmentation output. Another example is machine learning model 1 can process sensor data to generate a visualization output. Another example is machine learning model 1 can process sensor data to generate a diagnostic output. Another example is machine learning model 1 can process sensor data to generate a detection output.

[0151] In some embodiments, a machine learning model(s)1 may be configured to perform a task that includes encoding input data for reliable and / or efficient transmission or storage (and / or corresponding decryption). For example, the task may be an audio compression task. The input may include audio data, and the output may include compressed audio data. In another example, the input may include visual data (e.g., one or more images or videos), and the output may include compressed visual data, and the task is a visual data compression task. In yet another example, the task may include generating embeddings for input data (e.g., input audio or visual data). In one instance, the input may include audio data representing spoken utterances, and the task is a speech recognition task. The output may include text output mapped to the spoken utterances. In one instance, the task may include encrypting or decrypting input data. In one instance, the task may include a microprocessor performance task, such as branch prediction or memory address translation.

[0152] In some embodiments, the task is a generative task, and a machine-trained model(s)1 can be configured to output content generated considering an input(s)2. For example, input(s)2 can be data from one or more modalities that encode the context for generating additional content, or can be represented in other ways.

[0153] In some embodiments, the task may be a text completion task. A machine learning model(s) 1 can be configured to process an input(s) 2 representing text data and to produce an output(s) 3 representing additional text data to complete a text sequence containing the input(s) 2. For example, the machine learning model(s) 1 can be configured to produce an output(s) 3 to complete a sentence, paragraph, or portion of text that follows the portion of text represented by the input(s) 2.

[0154] In some embodiments, the task may be an instruction-following task. A machine-trained model(s) 1 can be configured to process input(s) 2 representing instructions to perform a function and to produce output(s) 3 that advance the goal of satisfying the instruction function (e.g., at least the steps of a multi-step procedure for performing the function). Output(s) 3 can represent the same data as input(s) 2 or data of a different modality. For example, input(s) 2 may represent text data (e.g., natural language instructions for the task to be performed), and machine-trained model(s) 1 can process input(s) 2 to produce output(s) 3 that represent text data in response to instructions (e.g., natural language responses, programming language responses, machine language responses, etc.). Input(s) 2 can represent image data (e.g., image-based instructions for a task to be performed, optionally accompanied by text instructions), and machine learning model(s) 1 can process input(s) 2 to generate output(s) 3 representing text data in response to instructions (e.g., natural language responses, programming language responses, machine language responses, etc.). One or more outputs(s) 3 can be generated iteratively or recursively to sequentially process and achieve steps toward achieving the requested function. For example, an initial output can be executed by an external system or processed by machine learning model(s) 1 to complete the initial steps toward performing the function. Multiple steps can be executed, and a final output in response to the first instruction is obtained.

[0155] In some embodiments, the task may be a question-answering task. A machine-trained model(s)1 can be configured to process input(s)2 representing a question to be answered and produce output(s)3 that advance the goal of returning an answer to the question (e.g., at least a step in a multi-step procedure for performing a function). Output(s)3 can represent data of the same or different modalities as input(s)2. For example, input(s)2 may represent text data (e.g., natural language instructions for a task to be performed), and machine-trained model(s)1 can process input(s)2 to produce output(s)3 representing text data in response to a question (e.g., a natural language response, a programming language response, a machine language response, etc.). Input(s) 2 can represent image data (e.g., image-based instructions for a task to be performed, optionally accompanied by text instructions), and machine learning model(s) 1 can process input(s) 2 to generate output(s) 3 representing text data in response to a question (e.g., natural language response, programming language response, machine language response, etc.). One or more outputs(s) 3 can be generated iteratively or recursively to sequentially process and achieve steps to achieve an answer to the question. For example, an initial output can be executed by an external system or processed by machine learning model(s) 1 to complete the first step to obtain an answer to the question (e.g., querying a database, performing a calculation, executing a script, etc.). Multiple steps can be performed, and a final output responding to the question is obtained.

[0156] In some embodiments, the task may be a text-image generation task. A machine learning model 1 can be configured to process an input 2 that represents a context about a desired portion of an image content. The context may include text data, image data, audio data, etc. The machine learning model 1 can be configured to produce an output 3 that represents image data depicting an image relevant to the context. For example, the machine learning model 1 can be configured to generate pixel data for an image. The channel values ​​associated with the pixels in the pixel data may be selected based on the context (for example, based on probabilities determined based on the context).

[0157] In some embodiments, the task may be a text-to-audio generation task. A machine-trained model 1 can be configured to process an input 2 that represents a context about a desired portion of audio content. The context may include text data, image data, audio data, etc. The machine-trained model 1 can be configured to produce an output 3 that represents an audio image related to the context. For example, the machine-trained model 1 can be configured to produce waveform data in image format (e.g., a spectrogram). The channel values ​​related to the pixels of the image may be selected based on the context. The machine-trained model 1 can be configured to produce waveform data in the form of a sequence of discrete samples of a continuous waveform. The sequence values ​​may be selected based on the context (e.g., based on probabilities determined based on the context).

[0158] In some embodiments, the task may be a data generation task. A machine learning model(s) 1 can be configured to process input(s) 2 that represent context about a desired portion of the data (e.g., data from various data domains, such as sensor data, image data, multimodal data, or statistical data). The desired data may be, for example, synthetic data for training other machine learning models. The context can include any data type(s). A machine learning model(s) 1 can be configured to produce output(s) 3 that represent data that align with the desired data. For example, a machine learning model(s) 1 can be configured to produce data values ​​to input into a dataset. The values ​​of the data objects(s) may be selected based on the context (e.g., based on probabilities determined based on the context). Exemplary computing systems and devices

[0159] Figure 12 is a block diagram of an exemplary network computing system capable of implementing an exemplary embodiment of the present disclosure. The system may include a plurality of computing devices and systems that are communicably coupled over a network 49. An exemplary computing device 50 is described to provide an example of a computing device capable of running any embodiment of the present disclosure (e.g., implementing a model host 31, a client(s) 32, or both). An exemplary server computing system 60 is described as an example of a server computing system capable of running any embodiment of the present disclosure (e.g., implementing a model host 31, a client(s) 32, or both). The computing device 50 and the server computing system(s) 60 can interact in cooperation (e.g., over the network 49) to run any embodiment of the present disclosure (e.g., implementing a model host 31, a client(s) 32, or both). A model development platform system 70 is an exemplary system that can host or supply a model development platform(s) 12 for developing machine learning models. The third-party system(s) 80 is an exemplary system(s) that any of the computing device(s) 50, server computing system(s) 60, or model development platform system(s) 70 can interact with in various aspects of the implementation of this disclosure (e.g., use of third-party tools, access to third-party databases or other resources).

[0160] Network 49 may be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or a combination thereof, and may include any number of wired or wireless links. In general, communication through Network 49 may be conducted via any type of wired or wireless connection using a wide variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encoding or formatting (e.g., HTML, XML), or protection schemes (e.g., VPN, Secure HTTP, SSL). Network 49 may also be implemented via a system bus. For example, one or more devices or systems in Figure 12 may be located in the same place as one or more other devices or systems, may be housed in one or more other devices or systems, or may be integrated in other ways.

[0161] Computing device 50 may be any type of computing device, such as a personal computing device (e.g., a laptop or desktop), a mobile computing device (e.g., a smartphone or tablet), a game console or controller, a wearable computing device, an embedded computing device, a server computing device, a virtual machine running on a host device, or any other type of computing device. Computing device 50 may be a client computing device. Computing device 50 may be an end-user computing device. Computing device 50 may be a computing device for the service itself that provides services to the end user (the end user can interact with computing device 50 using other computing devices).

[0162] The computing device 50 may include one or more processors 51 and memory 52. ​​The processor(s) 51 may be any suitable processing device (e.g., a processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and may be one processor or multiple processors connected in an operable manner. The memory 52 may include one or more non-temporary computer-readable storage media such as HBM, RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, and combinations thereof. The memory 52 may store data 53 and instructions 54, which may be executed by the processor(s) 51 to cause the computing device 50 to perform an operation. The operation may implement any one or more features described herein. The operation may implement exemplary methods and techniques described herein.

[0163] The computing device 50 may also include one or more input components that receive user input. For example, a user input component may be a touch-sensitive component (e.g., a touch-sensitive display screen or touchpad) that senses the touch of a user input object (e.g., a finger or stylus). The touch-sensitive component may serve to implement a virtual keyboard. Other exemplary user input components include a microphone, a camera, LiDAR, a physical keyboard, or other buttons, or other means by which the user can provide user input.

[0164] The computing device 50 can store or contain one or more machine learning models 55. The machine learning model 55 may contain one or more machine learning models 1, such as a sequence processing model 4. The machine learning model 55 may contain one or more model instances 31-1. The machine learning model 55 can be received from a server computing system 60, a model development platform system 70, a third-party system 80 (e.g., an application distribution platform), or developed locally on the computing device 50. The machine learning model 55 may be loaded into memory 52 and used by a processor 51, or otherwise implemented. The computing device 50 can implement multiple parallel instances of the machine learning model 55.

[0165] A server computing system (or multiple systems) 60 may include one or more processors 61 and memory 62. The processor(s) 61 may be any suitable processing device (e.g., a processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and may be one processor or multiple processors operably connected. The memory 62 may include one or more non-temporary computer-readable storage media, such as HBM, RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, and combinations thereof. The memory 62 may store data 63 and instructions 64, which may be executed by the processor(s) 61 to cause the server computing system (or multiple systems) 60 to perform operations. The operation may implement any one or more features described herein. The operation may implement exemplary methods and techniques described herein.

[0166] In some embodiments, the server computing system 60 includes one or more server computing devices, or is otherwise implemented by them. If the server computing system 60 includes multiple server computing devices, such server computing devices may operate in a sequential computing architecture, a parallel computing architecture, or any combination thereof.

[0167] The server computing system 60 may store or otherwise include one or more machine learning models 65. The machine learning models 65 may be the same as or different from the machine learning models 55. The machine learning models 65 may include one or more machine learning models 1, such as the sequence processing model 4. The machine learning models 65 may include one or more model instances 31-1. The machine learning models 65 can be received from computing devices 50, model development platform systems 70, third-party systems 80, or developed locally on the server computing system 60. The machine learning models 65 may be loaded into memory 62 and used by processors 61, or otherwise implemented. The server computing system 60 may implement multiple parallel instances of the machine learning models 65.

[0168] In an exemplary configuration, the machine learning model 65 can be contained in or otherwise stored and implemented by the server computing system 60 to establish a client-server relationship with the computing device 50 for providing model inference. For example, the server computing system 60 may implement a model host 31 on the computing device 50 on behalf of the client 32. For example, the machine learning model 65 may be implemented by the server computing system 60 as part of a web service (e.g., a remote machine learning model host service such as an online interface for performing machine learning model calculations over a network on the server computing system 60). For example, the server computing system 60 may communicate with the computing device 50 via a local intranet or internet connection. For example, the computing device 50 may be a workstation or endpoint communicating with a server computing system 60, and the implementation of the machine learning model 65 is managed by the server computing system 60 to perform inference remotely (e.g., at runtime or for training operations), and the output(s) are returned to the computing device 50 (e.g., cast, streamed, etc.). The machine learning model 65 can work collaboratively or interactively with the machine learning model 55 on the computing device 50 to perform a variety of tasks.

[0169] In some embodiments, the computing device 50 may store or include some or all of the text generation application 56. Alternatively or additionally, the server computing system 60 may store or include some or all of the text generation application 66. Thus, the text generation application can be located in the computing device 50 and / or the server computing system 60. The text generation applications 56 and / or 66 may be implemented in hardware, firmware, and / or software that control a general-purpose processor. For example, in some embodiments, the text generation applications 56 and / or 66 include program files stored in a storage device, loaded into memory, and executed by one or more processors. In other embodiments, the text generation applications 56 and / or 66 include one or more sets of computer executable instructions stored in RAM, a hard disk, or a tangible computer-readable storage medium such as an optical or magnetic medium.

[0170] The model development platform system(s) 70 may include one or more processors 71 and memory 72. The processor(s) 71 may be any suitable processing device (e.g., a processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and may be one processor or multiple processors connected in an operable manner. The memory 72 may include one or more non-temporary computer-readable storage media such as HBM, RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 72 may store data 73 and instructions 74, which may be executed by the processor(s) 71 to cause the model development platform system(s) 70 to perform operations. The operation may implement any one or more features described herein. The operation may implement exemplary methods and techniques described herein. The exemplary operation includes the functions described herein with respect to the model development platform 12. These functions and other functions may be implemented by developer tools(s) 75.

[0171] The third-party system(s) 80 may include one or more processors 81 and memory 82. The processor(s) 81 may be any suitable processing device (e.g., a processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and may be one processor or multiple processors connected in an operable manner. The memory 82 may include one or more non-temporary computer-readable storage media such as HBM, RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 82 may store data 83 and instructions 84, which may be executed by the processor(s) 81 to cause the third-party system(s) 80 to perform an operation. The operation may implement any one or more features described herein. The operation may implement exemplary methods and techniques described herein. Exemplary operations include the functionality of tools and other external resources (e.g., third-party resources)85 that are called when training or inferring using machine learning models1,4,16,20,55,65, etc., as described herein.

[0172] Figure 12 shows one exemplary configuration of a computing system that may be used to implement this disclosure. Other computing system configurations may be used similarly. For example, in some embodiments, one or both of the computing system 50 or server computing system(s) 60 may implement all or part of the operation of the model development platform system 70. For example, the computing system 50 or server computing system(s) 60 may implement developer tools(s) 75 (or extensions thereof) to develop, update / train, or improve machine-learned models 1, 4, 16, 20, 55, 65, etc., using one or more techniques described herein with respect to the model alignment toolkit 17. In this way, for example, the computing system 50 or server computing system(s) 60 may develop, update / train, or improve machine-learned models based on local datasets (for example, for model personalization / customization, to the extent permitted by user data preference selection).

[0173] Figure 13 is a block diagram of an exemplary computing device 98 implemented according to an exemplary embodiment of the present disclosure. The computing device 98 may be a user computing device or a server computing device (e.g., computing device 50, server computing system 60, etc.). The computing device 98 may implement a model host 31. For example, the computing device 98 may include several applications (e.g., applications 1 to N). Each application may include its own machine learning library and machine-trained models 1 or 1. For example, each application may include machine-trained models. Exemplary applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. As illustrated in Figure 13, each application may communicate with several other components of the computing device, such as one or more sensors, a context manager, a device state component, or additional components. In some embodiments, each application may communicate with each device component using an API (e.g., a public API). In some embodiments, the API used by each application is specific to that application.

[0174] Figure 14 is a block diagram of an exemplary computing device 99 implemented according to an exemplary embodiment of the present disclosure. Computing device 99 may be the same as or different from computing device 98. Computing device 99 may be a user computing device or a server computing device (e.g., computing device 50, server computing system 60, etc.). Computing device 98 may implement a model host 31. For example, computing device 99 may include several applications (e.g., applications 1 to N). Each application may communicate with the central intelligence layer. Exemplary applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. In some embodiments, each application may communicate with the central intelligence layer (and the model 1(s) stored therein) using an API (e.g., a common API across all applications).

[0175] The central intelligence layer can include multiple machine learning models. For example, as illustrated in Figure 14, each machine learning model may be provided for each application and managed by the central intelligence layer. In other embodiments, two or more applications may share a single machine learning model. For example, in some embodiments, the central intelligence layer may provide a single model to all applications. In some embodiments, the central intelligence layer may be contained within the operating system of the computing device 99 or otherwise implemented by the operating system of the computing device 99.

[0176] The central intelligence layer can communicate with the central device data layer. The central device data layer may be a centralized repository of data from the computing device 99. As shown in Figure 14, the central device data layer can communicate with several other components of the computing device, such as one or more sensors, a context manager, a device state component, or additional components. In some embodiments, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0177] Additional disclosures The technologies described herein refer to servers, databases, software applications, and other computer-based systems, as well as actions performed on such systems and information sent to and received from such systems. The inherent flexibility of computer-based systems allows for a wide variety of feasible configurations, combinations, and divisions of tasks and functions between components. For example, the processes described herein can be implemented using a single device or component, or multiple devices or components working together. Databases and applications can run on a single system or be distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0178] While the subject matter has been described in detail with respect to various specific exemplary embodiments, each example is provided for illustrative purposes only and does not limit the disclosure. Those skilled in the art, having attained the foregoing understanding, will readily be able to create modifications, variations, and equivalents to such embodiments. Therefore, the disclosure of the subject matter does not preclude the inclusion of such modifications, variations, or additions to the subject matter that would be readily apparent to those skilled in the art. For example, features illustrated or described as part of one embodiment may be used in other embodiments to create yet another embodiment. Thus, the disclosure is intended to extend to such modifications, variations, and equivalents.

[0179] The aspects of this disclosure have been described in relation to their exemplary embodiments. Any and all of the following claims and features should not be construed as limiting the scope of possible combinations of features disclosed herein by the dependency of the exemplary claims enumerated herein, so that combinations can be combined or rearranged in any way, including combinations of claims not expressly enumerated. Accordingly, the scope of this disclosure is illustrative and not limiting, and the disclosure of the subject matter does not preclude such modifications, variations, or additions to the subject matter, as would be readily apparent to those skilled in the art. Furthermore, terms are described herein using lists of exemplary elements joined by conjunctions such as “and,” “or,” and “but.” It should be understood that such conjunctions are provided for illustrative purposes only. For example, clauses and other sets of items joined by certain conjunctions such as “or” may refer to “and / or,” “at least one of the exemplary elements enumerated therein,” “any combination of,” etc. Terms such as “based on” should be understood as “at least partially based on.”

[0180] The term "can" should not be understood as referring to a capability that necessarily exists in all embodiments, but rather as referring to the potential of features in various embodiments. For example, the phrase "X can perform Y" should be understood as indicating that in various embodiments, X may be configured to perform Y, and not as indicating that X can always perform Y in all examples. In various embodiments, X may not be able to perform Y and may remain within the scope of this disclosure.

[0181] The term "may" should be understood not as referring to a capability necessarily present in all embodiments, but rather as referring to the possibility of features in various embodiments. For example, the phrase "X may perform Y" should be understood as indicating that in various embodiments, X may be configured to perform Y, and not as indicating that X can always perform Y in all examples. In various embodiments, X may not be able to perform Y and may remain within the scope of this disclosure.

Claims

1. A computing system for automatic text editing, A machine learning agent including one or more machine learning models configured to perform one or more text editing tasks, A text generation application comprising a machine learning agent including addressable entities that can be addressed within the text generation application, The computing system is configured to perform an operation, and the operation is The computing system obtains data describing annotations entered by a user in the text generation application, wherein the annotations correspond to a set of text content created within the text generation application, and the annotations include a natural language text editing request addressed to the machine learning agent. In response to the annotation, the computing system executes the machine learning agent to cause the machine learning agent to generate one or more edits to the set of text content based on the natural language text editing request. The computing system updates the set of text content based on one or more edits generated by the machine learning agent, A computing system that includes this.

2. The computing system according to claim 1, wherein the annotation includes comments displayed in the margins of the graphical user interface of the text generation application.

3. The computing system according to claim 1 or 2, wherein the annotation is addressed to the machine learning agent using natural language.

4. The computing system according to claim 1 or 2, wherein the annotation is addressed to the machine learning agent using a specific symbol that designates the machine learning agent as the recipient of the annotation.

5. The computing system according to any one of claims 1 to 4, wherein the set of text content includes two or more sentences.

6. The computing system according to any one of claims 1 to 5, wherein the set of text content includes an entire document being edited by the user in the text generation application.

7. The computing system according to any one of claims 1 to 5, wherein the set of text content includes a subset of documents edited by the user in the text generation application, and the set of text content is selected by the user.

8. The aforementioned computing system executes the machine learning agent, The computing system generates model prompts based on the set of text content and the natural language text editing request. The computing system includes processing the model prompts with the machine learning agent's machine learning agent's machine learning models in order to generate the one or more edits as the output of the one or more machine learning models, The computing system according to any one of claims 1 to 7.

9. The computing system generates the model prompt based on the set of text content and the natural language text editing request. The computing system according to claim 7, comprising combining (1) the set of text content, (2) one or more contextual portions adjacent to the set of text content in a document being edited by the user in the text generation application, and (3) the natural language text editing request for forming the model prompt.

10. The aforementioned operation is, The computing system executes the machine learning agent to cause it to generate a second annotation in response to the annotation input by the user, The computing system further includes inserting the second annotation into the text generation application, The computing system according to any one of claims 1 to 9.

11. A method implemented on a computer for training a language model, wherein the method is A computing system including one or more computing devices obtains a training tuple containing data describing a natural language text editing instruction, a set of text content, and one or more ground truth edits performed on the set of text content in connection with the natural language text editing instruction, wherein the set of text content contains two or more sentences. The computing system processes the natural language text editing command and the set of text content using the language model to generate one or more predictive edits for the set of text content in response to the natural language text editing command. The computing system evaluates a loss function that generates a loss value based on a comparison of one or more predictive edits and one or more ground truth edits, The computing system modifies one or more parameter values ​​of the language model based on the loss function. A method of implementation in a computer, including

12. The method implemented on a computer according to claim 11, wherein the training tuple includes comments and edit history obtained from a text generation application in which the language model is deployed.

13. The method implemented on a computer according to claim 11, wherein the training tuple includes an edit history obtained from a large text editing corpus.

14. The large text editing corpus is implemented on a computer according to claim 13, including editing into an online collaboratively edited encyclopedia.

15. A method implemented in a computer according to any one of claims 11 to 14, wherein the natural language instruction includes a summary of one or more ground truth edits.

16. A computer-implemented method according to any one of claims 11 to 14, wherein the natural language instruction includes a modified version of the summary of one or more ground truth edits, the modified version of the summary being generated by a second language model different from the original version of the summary of one or more ground truth edits.

17. A method implemented in a computer according to any one of claims 11 to 16, wherein the natural language instruction is selected by a reward language model.

18. A computer implementation of the method according to any one of claims 11 to 17, wherein the natural language instructions are filtered based on NLI score, length, or edit distance.

19. The method for implementing the natural language instruction in a computer according to any one of claims 11 to 18, comprising a synthesized prompt.

20. One or more non-temporary computer-readable media for storing together a machine learning agent as described in any one of claims 1 to 10, or a language model as described in any one of claims 11 to 19.

Images