Semantic clustering for infinite context window size for sequence processing models

Abstract

Systems and methods are provided for semantic clustering of an arbitrary long context window for a machine-learned sequence processing model. A computing system can obtain a context sequence. The computing system can determine a plurality of subsequences of the context sequence. The computing system can determine a semantic embedding for each subsequence using a machine-learned semantic embedding model. The computing system can determine a plurality of semantic clusters based on the semantic embeddings. The computing system can generate an output sequence using a machine-learned sequence generation model and based at least in part on the semantic clusters.

Description

Technical Field

[0001] This disclosure generally relates to machine learning processes and apparatuses and systems that utilize machine learning. More specifically, this disclosure relates to semantic clustering for sequence processing models that utilize machine learning. Background Technology

[0002] A computer can receive input. A computer can execute instructions to process the input using a parameterized model to generate output. A computer can receive feedback about its performance in generating output using the model. A computer can generate feedback by evaluating its performance. A computer can receive feedback from external sources. Based on this feedback, a computer can update the model's parameters to improve its performance. In this way, a computer can iteratively "learn" to generate the desired output. The resulting model is often referred to as a machine learning model.

[0003] In some instances, machine learning models can be associated with a finite maximum amount of input data that the model is configured to process. For example, a machine learning sequence processing model might have a finite context window, where inputs longer than the maximum length are effectively truncated, causing a portion of the input to be practically ignored. However, this finite input size can be associated with poorer performance (e.g., inference accuracy) in some instances, relative to machine learning models configured to process more input data. Conversely, increasing the maximum input length can be associated with very high computational costs in some instances. For example, the computational cost of training a machine learning sequence processing model might be proportional to the square of the maximum input length in some instances (e.g., O(n^2)). 2 d), where n is the maximum context length and d is the dimension of the machine learning model. For this reason, training a machine learning model to efficiently use a very large context window according to existing methods can be prohibitively expensive in some instances. Summary of the Invention

[0004] Aspects and advantages of embodiments of this disclosure will be set forth in part in the description which follows, or may be learned from the description or by practice of the embodiments.

[0005] An example aspect of this disclosure provides an example method. In some implementations, this example method may include obtaining a context sequence by one or more computing devices. The example method may include determining multiple subsequences of the context sequence by one or more computing devices. The example method may include determining multiple corresponding semantic embeddings, respectively associated with the multiple subsequences of the context sequence, by one or more computing devices using a first machine learning-based sequence processing model. The example method may include determining multiple semantic clusters by one or more computing devices. In this example method, each of the multiple semantic clusters may include one or more semantic embeddings from a plurality of semantic embeddings. The example method may include generating an output sequence by one or more computing devices using at least one of a first machine learning-based sequence processing model and a second machine learning-based sequence processing model, at least partially based on the multiple semantic clusters.

[0006] This example method may include selecting one or more subsequences from a plurality of subsequences of a context sequence, at least in part based on the plurality of semantic clusters, by one or more computing devices. This example method may also include adding data associated with the one or more subsequences to a context window of a second machine learning-based sequence processing model by one or more computing devices. In this example method, the output sequence may be generated at least in part based on the context window.

[0007] In this example method, the data associated with one or more subsequences may include at least one of the one or more subsequences.

[0008] In this example method, the data associated with one or more subsequences may include metadata associated with at least one of the one or more subsequences.

[0009] In this example method, the data associated with one or more subsequences may include a machine-generated summary of at least one of the one or more subsequences.

[0010] In this example method, the data associated with one or more subsequences may include sequence data generated by one or more computing devices based at least in part on semantic embeddings associated with at least one of the one or more subsequences.

[0011] In this example method, the data associated with one or more subsequences may include one or more corresponding semantic embeddings from a plurality of corresponding semantic embeddings.

[0012] In this example method, the data associated with one or more subsequences may include sequence data generated by one or more computing devices based at least in part on geometric transformations of semantic embeddings associated with at least one of the one or more subsequences.

[0013] In this example method, determining one or more subsequences may include selection based on one or more priorities.

[0014] In this example approach, selecting based on one or more priorities may include prioritizing clusters associated with a larger number of semantic embeddings.

[0015] In this example method, selecting based on one or more priorities may include giving preference to semantic embeddings near the center of the semantic cluster.

[0016] In this example approach, the center of a semantic cluster can be determined by averaging multiple semantic embeddings of the semantic cluster.

[0017] In this example method, one or more priorities may include priorities determined based on user input.

[0018] In this example method, selecting based on one or more priorities may include giving preference to semantic embeddings near the semantic embeddings associated with the user input.

[0019] In this example method, selecting based on one or more priorities may include giving preference to semantic embeddings from clusters that have centroids near the semantic embeddings associated with the user input.

[0020] In this example method, selecting based on one or more priorities may include prioritizing the selection of at least one subsequence from each of the plurality of semantic clusters over the selection of a second subsequence from any of the plurality of semantic clusters.

[0021] In this example method, the first machine learning model can be characterized by a first number of parameters; the second machine learning model can be characterized by a second number of parameters; and the second number can be at least five times the first number.

[0022] In this example method, the second quantity can be at least 10 times the first quantity.

[0023] In this example method, the second quantity can be at least 30 times the first quantity.

[0024] In this example method, the first machine learning model can be configured to receive a sequence of multiple lexical units as input; and generate a single semantic embedding based on at least two different lexical units among the multiple lexical units.

[0025] In this example method, the first-order machine learning model can be configured to generate a semantic embedding for each input lexical it receives. In this example method, determining the semantic embedding associated with a subsequence of the context sequence can include combining multiple semantic embeddings generated by the first-order machine learning model based on that subsequence.

[0026] In this example method, determining the multiple semantic clusters may include hierarchical clustering.

[0027] In this example method, at least one of the plurality of semantic clusters may include a plurality of subclusters. In this example method, selection based on one or more priorities may include making the selection of at least one subsequence from each of two or more subclusters preferred over the selection of a second subsequence from any of the plurality of subclusters.

[0028] In this example method, at least one of the plurality of semantic clusters may include a plurality of subclusters. In this example method, selection based on one or more priorities may include giving preference to semantic embeddings near the centroid of the subclusters among the plurality of subclusters.

[0029] In this example method, determining multiple subsequences of the context sequence may include splitting the context at least in part based on a subsequence length threshold.

[0030] In this example method, determining multiple subsequences of the context sequence may include splitting the context at least in part based on one or more delimiters or sequence boundaries.

[0031] In this example method, at least one of the one or more delimiters may include one or more numbers, letters, text, or whitespace characters.

[0032] In this example method, determining multiple subsequences of the context sequence may include using a machine learning-based model.

[0033] This example method may include adding a hint to the context window of a second machine learning model. In this example method, the hint may include one or more instructions or explanations.

[0034] Example aspects of this disclosure provide one or more example non-transitory computer-readable media storing instructions executable by one or more processors to cause a computing system to perform example operations. In some implementations, the example operations may include obtaining a context sequence. The example operations may include determining multiple subsequences of the context sequence. The example operations may include using a first machine learning-based sequence processing model to determine multiple corresponding semantic embeddings, each associated with a multiple subsequence of the context sequence. The example operations may include determining multiple semantic clusters. In the example operations, each of the multiple semantic clusters may include one or more semantic embeddings from a plurality of semantic embeddings. These example operations may include generating an output sequence based at least in part on the multiple semantic clusters using at least one of a first machine learning-based sequence processing model and a second machine learning-based sequence processing model.

[0035] An example aspect of this disclosure provides an example computing system comprising: one or more processors; and one or more example non-transitory computer-readable media storing instructions executable by the one or more processors to cause the computing system to perform example operations. In some implementations, the example operations may include obtaining a context sequence. The example operations may include determining multiple subsequences of the context sequence. The example operations may include using a first machine learning-based sequence processing model to determine multiple corresponding semantic embeddings, each associated with a plurality of subsequences of the context sequence. The example operations may include determining multiple semantic clusters. In the example operations, each of the multiple semantic clusters may include one or more semantic embeddings from a plurality of semantic embeddings. These example operations may include generating an output sequence based at least partially on the plurality of semantic clusters using at least one of a first machine learning-based sequence processing model and a second machine learning-based sequence processing model.

[0036] Other exemplary aspects of this disclosure relate to other systems, methods, apparatuses, tangible non-transitory computer-readable media, and devices for performing the functions described herein. These and other features, aspects, and advantages of the various implementations will be better understood with reference to the following description and the appended claims. The accompanying drawings, which are incorporated in and form a part of this specification, illustrate implementations of this disclosure and, together with the description, help to explain the relevant principles. Attached Figure Description

[0037] Figure 1 A block diagram of an example system implementing aspects of this disclosure is shown.

[0038] Figures 2A to 2C A diagram of an example system illustrating an example implementation of aspects of this disclosure is shown.

[0039] Figure 3 A flowchart of an example method is shown, illustrating an example implementation of an aspect of this disclosure.

[0040] Figure 4 This is a flowchart illustrating an example method for training a machine learning-based model, according to an example implementation of an aspect of this disclosure;

[0041] Figure 5 This is a block diagram of an example processing flow for using a machine learning-enabled model to process inputs to generate outputs, based on an example implementation of aspects of this disclosure.

[0042] Figure 6 This is a block diagram of an example sequence processing model based on an example implementation of aspects of this disclosure;

[0043] Figure 7 This is a block diagram of an example technique for filling an example input sequence for processing by a sequence processing model, based on an example implementation of aspects of this disclosure;

[0044] Figure 8 This is a block diagram of an example model development platform based on an example implementation of aspects of this disclosure;

[0045] Figure 9 This is a block diagram of an example training workflow for training a machine learning model, based on an example implementation of an aspect of this disclosure.

[0046] Figure 10 This is a block diagram of an inference system for operating one or more machine learning-based models to perform inference, based on an example implementation of aspects of this disclosure.

[0047] Figure 11 This is a block diagram of an example networked computing system based on an example implementation of aspects of this disclosure;

[0048] Figure 12 This is a block diagram of an example computing device implementing an aspect of this disclosure; and

[0049] Figure 13 This is a block diagram of an example computing device that implements aspects of this disclosure. Detailed Implementation

[0050] In general, this disclosure relates to systems and methods for using clusters of machine learning-based semantic embeddings with machine learning-based sequence processing models. In some instances, the provided systems and methods can facilitate the use of infinite (i.e., arbitrarily long) context sizes for machine learning models with finite context windows.

[0051] For example, an input context of arbitrary length can be split into semantic blocks (e.g., paragraphs, sentences, lines, etc.). Each block can be fed into a first sequence processing model (e.g., a lightweight embedding model associated with low computational cost) to generate one or more machine learning-based semantic embeddings associated with that block. These semantic embeddings can be clustered using a clustering algorithm to generate multiple semantic clusters. In some instances, a first or second machine learning-based sequence processing model can synthesize insights (e.g., summaries; identified patterns, topics, or entities; main ideas, suggested questions; etc.) from one or more semantic clusters. Based on multiple semantic clusters, one or more semantic blocks and / or synthesized insights can be selected to be included in a finite context window of a first, second, or third machine learning-based sequence processing model (e.g., a sequence generation model characterized by higher computational cost than the first sequence processing model). Based on the context selected for inclusion, the sequence processing model can generate one or more outputs (e.g., output sequences).

[0052] Breaking context into chunks can include various methods. In some instances, context can be broken down by one or more document boundaries (e.g., chapters, pages, paragraphs, etc.) or other sequence boundaries (e.g., songs; conversation turns; scenes or moments in an audio or video sequence; etc.). In some instances, context can be broken down by one or more delimiters (e.g., whitespace characters, such as newlines; text strings, such as tags associated with structured context, such as many-shot cues; audio delimiters, such as a period of silence; etc.). In some instances, context can be broken down into chunks based on one or more chunk length thresholds (e.g., maximum chunk length, minimum chunk length, etc.). In some instances, context can be broken down into chunks using machine learning models (e.g., machine learning-based recognition of sequence boundaries, machine learning-based recognition of sequence similarity, etc.).

[0053] Semantic embeddings for blocks can include using a lightweight sequence processing model configured to generate semantic embeddings based on lexical units or sequences. In some instances, the model can be configured to generate semantic embeddings for any sequence shorter than the maximum length. In this case, generating semantic embeddings for blocks shorter than the maximum length can essentially consist of providing the block as input to the model. Generating semantic embeddings for blocks longer than the maximum length can include, for example, splitting the block into subsequences shorter than the maximum length; generating semantic embeddings for each subsequence; and combining these semantic embeddings (e.g., via average pooling; via machine learning; etc.).

[0054] Clustering semantic embeddings can include any suitable clustering algorithm (e.g., split hierarchical clustering, centroid-based clustering (e.g., k-means), distribution-based clustering (e.g., g-means), connectivity-based clustering (e.g., agglomerated hierarchical clustering), density-based clustering, biclustering, etc.). In some instances, the target number of clusters can be determined based on the size of the finite context window of the sequence generation model used to generate the final output sequence (e.g., relative to the average length of the context block). In some instances, clustering algorithms (e.g., k-means) can be configured to generate a number of clusters equal to or approximately equal to the target number. In some instances, hierarchical clustering can be used to achieve the target number of clusters. For example, if a first clustering algorithm generates a first number of clusters greater than the target number, agglomerated hierarchical clustering can be used to merge multiple similar clusters into a single supercluster. If the first number of clusters is too small, split hierarchical clustering can be used. The hierarchical clustering process can be repeated until the target number of clusters is reached.

[0055] Selecting semantic blocks for inclusion within a finite context window can include, for example, deterministic or probabilistic selection based on one or more priorities. For instance, the selection mechanism can be configured to prioritize semantic blocks associated with large semantic clusters; semantic blocks with semantic embeddings near the center of a semantic cluster or hierarchical semantic sub-cluster; semantic blocks associated with clusters not currently represented in the finite context window; semantic blocks with semantic embeddings near semantic embeddings determined based on user input; and so on. In some instances, the selection mechanism can include hierarchical selection mechanisms, such as decision trees. In some instances, the selection mechanism can include numerical selection mechanisms, such as a weighted combination of one or more priorities. For example, in some instances, the selection mechanism can include assigning a corresponding priority score to each of a plurality of semantic blocks, where the priority score can include, for example, a weighted average of multiple priority sub-scores (e.g., distance to the nearest cluster centroid, the number of blocks in the cluster, etc.).

[0056] For example, an example selection algorithm can begin with the largest semantic cluster among multiple semantic clusters, selecting the subsequence whose semantic embedding is closest to the center of the largest semantic cluster. Next, the example selection algorithm can select from the second largest cluster, then the third largest cluster, and so on, until: (a) the context window size constraint is satisfied, or (b) at least one corresponding subsequence has been selected from each of the multiple semantic clusters. If there is remaining space in the context window after each semantic cluster has been represented, a second cluster can be selected from the largest cluster. In some instances, the second cluster may have an embedding that is second closest to the center of the largest semantic cluster, or an embedding that is closest to the center of the largest or second largest hierarchical subcluster of the largest semantic cluster. After each corresponding semantic cluster has been represented in the context window, the corresponding number of additional subsequences added from the corresponding cluster or subcluster (e.g., the largest cluster, the second largest cluster, etc.) can be based on the size of the corresponding cluster (e.g., approximately proportional to the size of the corresponding cluster).

[0057] To give another example, the example selection algorithm can begin with a semantic cluster having a center that is closest to the semantic embedding of the user input. The example algorithm can then select, for example, (a) the subsequence closest to the cluster center, or (b) the subsequence closest to the semantic embedding of the user input. Next, the example algorithm can select from the cluster that is second closest to the semantic embedding of the user input, and so on, until all clusters (or all clusters within a distance threshold of the semantic embedding of the user input, etc.) have been represented. After each corresponding semantic cluster or subcluster is represented in the context window, the corresponding number of additional subsequences added from the corresponding cluster or subcluster (e.g., the cluster closest to the user input, the second closest cluster, etc.) can be based on the distance between the center of the corresponding cluster and the semantic embedding of the user input (e.g., inversely proportional to it).

[0058] In some instances, the selected semantic block can be directly added to the limited context window of the sequence generation model. In others, additional data associated with the selected semantic block can be obtained and added to the limited context window. For example, in some instances, a machine learning-based sequence generation model can generate a sequence (e.g., a summary, extracted entities or keywords, extracted question answers, etc.) based on one or more selected blocks or clusters and add that sequence to the limited context window. In some instances, the computational system can obtain (e.g., receive, retrieve, generate) one or more metadata items (e.g., chapter number, word count, topic, etc.) associated with the selected block to add to the limited context window. In some instances, additional data not associated with the selected block can be added to the limited context window, such as prompts (e.g., "Please summarize the following information:"), etc.

[0059] The systems and methods disclosed herein provide a variety of technical effects and benefits. In one example, the provided systems and methods enable sequence processing models to efficiently utilize arbitrarily large context sequences.

[0060] Previous sequence processing models (e.g., transformers) have been associated with finite context windows in some instances. In some instances, the maximum length of the context window can be fixed by the model architecture or the training process used to train the sequence processing model. For example, in some instances, a transformer model pre-trained on a 2,000-term context might be well-suited for processing arbitrarily short contexts, but in others, it might not be architecturally meaningful for handling context windows longer than 2,000 term training contexts. In some instances, the computational cost (e.g., energy cost, etc.) of retraining such a model to support longer context windows can be prohibitively high (e.g., hundreds of millions of dollars). In some instances, a sequence processing model pre-trained on a 2,000-term context might be architecturally capable of being further trained (e.g., “fine-tuned”) to support longer context window data (e.g., 65,000 terms, etc.), but such additional training could require significant amounts of data and computational resources. Furthermore, the context window of such a model will still be limited by the size of the training context window used during the additional training process.

[0061] Advantageously, the systems and methods of this disclosure enable the use of arbitrarily long context windows without any model retraining. In this way, for example, compared to retraining-based methods (e.g., pre-training a new model, fine-tuning a small context window model on longer context data, etc.), the systems and methods of this disclosure can be associated with lower computational and energy costs. For example, in some instances, training sequence processing models to support larger context windows can cost millions or hundreds of millions of dollars. In contrast, the systems and methods provided can be efficiently combined with existing models to provide extended context windows at lower computational and energy costs.

[0062] Furthermore, in some instances, the systems and methods disclosed herein can be associated with lower inference computation costs compared to existing long context window models, even when retraining is no longer required. For example, in some instances, processing the context of n lexical units with a sequence generation model with embedding dimension d can be comparable to processing the context of n lexical units. 2The computational cost is proportional to d. Furthermore, in some instances, high-quality sequence generation can depend on a computationally expensive sequence generation model (e.g., one with a large number of parameters). Advantageously, the provided systems and methods can handle long contexts using computationally inexpensive sequence embedding models and can provide shorter, condensed contexts for processing by higher-computational-cost sequence generation models. In this way, for example, the provided systems and methods can handle contexts of a given length with lower cost and lower latency than existing systems and methods, even in instances where existing systems and methods may not require retraining.

[0063] Similarly, compared to iterative summarization methods used to provide condensed context, the systems and methods of this disclosure can be associated with lower energy costs and lower latency. For example, in some instances, iterative summarization may require repeated invocation of computationally expensive sequence generation models (e.g., large language models with hundreds of billions of parameters). For instance, high-cost models may be repeatedly invoked not only to summarize multiple semantic blocks of long contexts but also to summarize multiple summaries. In contrast, the systems and methods of this disclosure allow for the use of lightweight machine learning-based sequence embedding models and lightweight clustering systems to populate finite context windows at low computational cost. Furthermore, the provided systems and methods only require each semantic block to be embedded once. Additional data (e.g., additional books or other documents) can be added to the arbitrarily long contexts of this disclosure without re-embedding any existing data (e.g., by performing a lightweight re-clustering operation).

[0064] Furthermore, the systems and methods disclosed herein can be efficiently combined with other systems and methods for extending the length of context windows. For example, the provided systems and methods can efficiently embed and cluster semantic blocks of arbitrarily large contexts (e.g., databases or corpora with millions, hundreds of millions, billions, etc.), and select blocks to include into a finite context window of any size (e.g., thousands, hundreds of thousands, etc.). Similarly, the provided semantic clustering methods can be combined with summarization methods to select multiple summaries for a finite context window at a lower energy cost compared to existing methods.

[0065] The technical effect of the exemplary implementations disclosed herein is to improve the energy efficiency of performing operations using machine learning models, thereby improving the capabilities of computers implementing such models. For example, the exemplary implementations can provide more energy-efficient runtime execution or inference. In some scenarios, improved energy efficiency can reduce the energy consumed in performing a given task (e.g., reducing the energy consumed in maintaining the model in memory, reducing the energy consumed in performing computations within the model, etc.). In other scenarios, improved energy efficiency can accomplish more tasks within a given energy budget (e.g., a greater number of tasks, more complex tasks, or performing the same task with higher accuracy or precision, etc.).

[0066] In another example, the example implementations can provide more energy-efficient training operations or model updates. In some scenarios, improved energy efficiency can reduce the energy used to perform a given number of update iterations (e.g., reducing the energy consumed in maintaining the model in memory, reducing the energy consumed in performing computations within the model, such as calculating gradients, backpropagation loss, etc.). In some scenarios, improved energy efficiency can complete more update iterations for a given energy budget (e.g., a greater number of iterations, etc.). In some scenarios, the greater expressiveness provided by the model architecture and training techniques of this disclosure can enable obtaining a given level of functionality in fewer training iterations, thus incurring a smaller energy budget. In some scenarios, the greater expressiveness provided by the model architecture and training techniques of this disclosure can enable obtaining an extended level of functionality in a given number of training iterations, thus using a given energy budget more efficiently.

[0067] In this way, for example, the improved energy efficiency of the exemplary implementations of this disclosure can reduce the amount of pollution or other waste associated with implementing machine learning-based models and systems, thereby advancing the field of machine learning and artificial intelligence as a whole. The amount of pollution can be reduced overall (e.g., its absolute value) or on a normalized basis (e.g., energy per task, per model size, etc.). For example, by implementing more energy-efficient training or inference operations, CO2 emissions (e.g., emissions from power sources) associated with training and executing machine learning models can be reduced. By implementing more energy-efficient training or inference operations, the amount of thermal pollution in the environment (e.g., caused by processor / storage locations) can be reduced.

[0068] Various example implementations are described in this article with accompanying figures.

[0069] Example System

[0070] Figure 1This is a block diagram of an example system according to the present disclosure, wherein semantic clustering enables machine learning models with finite context windows to use arbitrarily long input contexts. A long formal context 102 can be obtained by a computational system 104 that can split the long formal context 102 into multiple context blocks 106. A lightweight sequence embedding model 108 can generate semantic block embeddings 110 based on context blocks 106. A clustering system 112 can cluster the semantic block embeddings 110 into multiple semantic clusters 114, each cluster including one or more semantic block embeddings 110. In some instances, semantic clusters 114 can be provided as input to one or more machine learning models (such as a sequence generation model 116), which can generate synthetic insights 118 based on semantic clusters 114. Based on semantic clusters 114, computational system 104 can select one or more context blocks 106 or synthetic insights 118 to use as the selected context 120. The selected context 120 can be provided as input to the sequence generation model 116, which can generate the generated output 122 based on the selected context 120.

[0071] Figure 1 Long formal context 102 is depicted. Long formal context 102 can typically include or otherwise represent various types of data. In some instances, long formal context 102 can include various types of sequence data (e.g., image data, audio data, text data, etc.). Long formal context 102 can include one or more types of data. In some instances, long formal context 102 can include multimodal data (e.g., sequence data having multiple modes such as video and audio; sequence data combined with non-sequence data or metadata; etc.). In some instances, long formal context 102 can be characterized by a context window larger than the size of the machine learning model of interest (e.g., sequence generation model 116 described below). In some instances, long formal context 102 can be stored or located in one location or distributed across multiple locations (e.g., stored in one file, document, storage location, etc.; stored in multiple files, documents, storage locations, etc.; stored on one or more computing devices; etc.). In some instances, the long formal context 102 may include one or more corpora (e.g., datasets, databases, etc.), wherein each of the one or more corpora may include one or more sequences (e.g., books, documents, audio files, images, etc.). The long formal context 102 may include data obtained in one or more ways, such as receiving from user input; retrieving from one or more computer-readable media; retrieving via an Internet search; receiving from another computing device, etc.

[0072] Figure 1 A computing system 104 is depicted. The computing system 104 may include, for example, one or more computing devices. The computing system 104 may be located on a single computing system or distributed across multiple computing systems. In some instances, the computing system 104 may correspond to... Figures 4 to 13 The computing system described (e.g., server computing system 60, etc.).

[0073] Figure 1 Context block 106 is depicted. Context block 106 can typically include or otherwise represent various types of data. In some instances, context block 106 can include or represent various types of sequential data (e.g., image data, audio data, text data, etc.). Context block 106 can include one or more types of data. In some instances, context block 106 can include multimodal data (e.g., sequential data with multiple modes such as video and audio; sequential data combined with non-sequential data or metadata; etc.). In some instances, a context block can be a subsequence of long-form context 102, a subsequence comprising long-form context 102, or otherwise corresponding to a subsequence of long-form context 102. In some instances, context block 106 can be characterized by a size smaller than the context window of the machine learning model of interest (e.g., sequence generation model 116 as described below). In some instances, context block 106 can be characterized by a size much smaller than the context window of the model of interest, such that the context window can receive multiple context blocks 106.

[0074] Context block 106 can be generated in various ways. Computation system 104 can be configured to use one or more mechanisms to generate context block 106. In some instances, context block 106 can be generated by splitting long form context 102, which can be done in various ways.

[0075] Splitting the long form context 102 may include splitting the long form context 102 based on one or more pre-existing boundaries associated with, for example, computing system 104, dataset, or computer-readable medium. For example, the long form context 102 may include multiple files or other data types, and splitting the long form context 102 may include splitting by file, etc.

[0076] Splitting the long formal context 102 may include, for example, splitting by one or more sequence boundaries. Sequence boundaries may include, for example, natural boundaries or metadata-defined boundaries. Natural boundaries may include, for example, boundaries that can be identified solely based on sequence data without referencing metadata not included in the sequence. As an illustrative example, natural boundaries may include grammatically defined boundaries between components of natural language (e.g., paragraphs, sentences, etc.). Other natural sequence boundaries may include, for example, boundaries between songs; relative silence periods in an audio sequence; boundaries between objects depicted in an image; boundaries between acts or scenes in a novel; conversation turns; and so on. Metadata-defined boundaries may include, for example, boundaries that can be defined based on metadata associated with a portion of the sequence (e.g., file components that define paragraphs in a document file, such as Visual Basic Paragraph objects or ListParagraph objects; chapter numbers associated with text sequence data located in a book or PDF file; track numbers or chapter boundaries associated with an audio or video file; and so on). In some instances, metadata-defined boundaries may include, for example, document boundaries (e.g., page boundaries, chapter boundaries, section boundaries, etc.) associated with a document from which a portion of the long formal context 102 is derived (e.g., electronic documents, physical books, etc.).

[0077] Splitting the long form context 102 may include, for example, splitting based on one or more delimiters. Delimiters may include, for example, one or more tokens of the long form context 102 (e.g., a newline character in a text sequence, a silence period in an audio sequence, a predefined sequence of pixels, etc.). For example, in some instances, the long form context 102 may include structured data (e.g., XML data, structured few-shot hints or thought chain hints configured as input to a machine learning model, etc.), and the delimiters may be sequences associated with structures associated with the structured data (e.g., XML tags, structured few-shot hint tags, such as "Question:", "Step 1:", etc.).

[0078] Splitting the long formal context 102 may include, for example, splitting based on one or more block length thresholds (such as minimum or maximum block length). In some instances, the long formal context 102 may be split based on a fixed block length, regardless of the sequence content. In other examples, splitting by length thresholds may be combined with other splitting mechanisms. For example, a maximum block length may be defined (e.g., the maximum length associated with the context window of a lightweight sequence embedding model, etc.); the long formal context 102 may be split in one or more ways described herein (e.g., splitting by paragraph); and any context block 106 longer than the maximum length may be split in various ways (e.g., split in half, split at sentence boundaries, etc.).

[0079] In some instances, splitting the long-form context 102 into context blocks 106 may include using a machine learning-enabled model. For example, in some instances, the machine learning-enabled model may be used to identify one or more natural boundaries of the sequence (e.g., a speech recognition model to identify conversation turns in an audio sequence; an image processing model to identify objects in an image; etc.). In some instances, the machine learning-enabled model may be used to determine metadata associated with the sequence (e.g., whether the audio sequence contains music, speech, or natural sounds; chapter numbers, page numbers, or figure numbers associated with a text sequence or subsequence; etc.), and the long-form context 102 may be split based on the metadata. In some instances, the machine learning-enabled model may be used to identify semantic similarity or dissimilarity between subsequences of the long-form context 102, and context blocks 106 may be partially defined by similarity or dissimilarity (e.g., paragraphs with highly dissimilar sentences may be split; two highly similar paragraphs may be merged into a single context block 106; etc.).

[0080] Figure 1 A lightweight sequence embedding model 108 is described. The lightweight sequence embedding model 108 can be or includes various different types of machine learning-based model architectures. In some instances, the lightweight sequence embedding model 108 can be or includes a sequence processing model configured to generate one or more outputs based on one or more inputs in a sequence. For example, the lightweight sequence embedding model 108 can be or includes a sequence processing model configured to generate one or more semantic embeddings based on various types of sequence data (e.g., image data, audio data, text data, multimodal data, etc.). In some instances, the lightweight sequence embedding model 108 can utilize attention mechanisms, such as self-attention. For example, the lightweight sequence embedding model 108 can be or includes a multi-head self-attention model (e.g., encoder-only, encoder-decoder, or decoder-only transformer language models, such as PaLM, etc.). In some instances, the lightweight sequence embedding model 108 can be or includes a word2vec model or a similar model configured for multi-word sequences (e.g., doc2vec, etc.). In some instances, the lightweight sequence embedding model 108 may be or include an embedding model configured for use with a lexical retrieval mechanism (e.g., XTR, etc.).

[0081] In some instances, the lightweight sequence embedding model 108 may be or include a model configured to receive a context block 106 as input and generate one or more semantic embeddings based on the context block 106. In some instances, the lightweight sequence embedding model 108 may have a sufficiently long lexical window to generate a single semantic embedding associated with the entire context block 106. In this case, the generated semantic embedding may be a semantic block embedding 110. In other instances, the lightweight sequence embedding model 108 may be configured to generate more than one semantic embedding for a given context block 106 (e.g., one semantic embedding per lexical; one semantic embedding for a subsequence of a fixed maximum length, where the length of the context block 106 is greater than the maximum length; etc.). In such instances, generating the semantic block embedding 110 may include combining multiple semantic embeddings generated by the lightweight sequence embedding model 108 (e.g., via pooling such as average pooling; combination via machine learning; concatenation; etc.) to generate a single semantic block embedding 110 associated with the context block 106.

[0082] In some instances, the lightweight sequence embedding model 108, relative to the sequence generation model 116, can be associated with lower computational costs (e.g., energy costs, latency, etc.). In some instances, the lightweight sequence embedding model 108 can be characterized by a first number of parameters (e.g., 100 million, 500 million, 1 billion, 5 billion, etc.), which is smaller than the second number of parameters associated with the sequence generation model 116. In some instances, the second number of parameters can be much larger than the first number (e.g., five times, ten times, 30 times, 50 times, 100 times, 500 times, etc.). Although... Figure 1 A lightweight sequence embedding model is described, but the lightweight sequence embedding model 108 can be replaced by a sequence embedding model with the same or greater computational cost or number of parameters as the sequence generation model 116 without exceeding the scope of this disclosure.

[0083] Figure 1 A semantic block embedding 110 is depicted. The semantic block embedding 110 can be, for example, a machine learning-based semantic embedding associated with context block 106. The semantic block embedding 110 can generally be, include, any computer-readable data type (e.g., vector, matrix, or tensor data; binary data; numerical data; etc.), or be represented by any computer-readable data type.

[0084] Figure 1Clustering system 112 is depicted. Clustering system 112 may include, for example, one or more computing devices. Clustering system 112 may reside on a single computing system or be distributed across multiple computing systems. In some instances, clustering system 112 may be computing system 104, include computing system 104, be included by computing system 104, implement computing system 104, or be implemented by computing system 104. In some instances, clustering system 112 may correspond to... Figures 4 to 13 The computing system described (e.g., server computing system 60, etc.).

[0085] Figure 1 Semantic cluster 114 is depicted. Semantic cluster 114 may include, for example, data indicating multiple semantic block embeddings 110 or context blocks 106. In some instances, semantic cluster 114 may include, for example, one or more sub-clusters, each sub-cluster having one or more context blocks 106. Semantic cluster 114 can be generated using any suitable clustering algorithm (e.g., centroid-based clustering (e.g., k-means), distribution-based clustering (e.g., g-means), connectivity-based clustering (e.g., agglomerated hierarchical clustering), dimensionality-reducing clustering (e.g., spectral clustering), density-based clustering, biclustering, split hierarchical clustering, affinity propagation, etc.). In some instances, generating semantic cluster 114 may include determining the target number of clusters (e.g., based on the size of a finite context window of sequence generation model 116). In some instances, clustering algorithms (e.g., k-means, agglomerated hierarchical clustering, split hierarchical clustering, etc.) may be configured to generate a number of clusters equal to or approximately equal to the target number. In some instances, the semantic block embedding 110 associated with the first semantic cluster 114 may be more similar to the first semantic cluster 114 than to any other semantic cluster 114, based on one or more similarity measures. For example, in some instances, the semantic block embedding 110 associated with the first semantic cluster may be closer to the centroid of the first semantic cluster 114 than to any centroid of any other semantic cluster 114, based on a measure of semantic distance (e.g., cosine distance associated with vector-based semantic embeddings). Example implementations for generating the semantic cluster 114 are further described below with reference to Figure 2.

[0086] Figure 1A sequence generation model 116 is described. The sequence generation model 116 can be or includes various different types of machine learning-based model architectures. The sequence generation model 116 can be or includes a sequence processing model configured to generate one or more outputs in a sequence based on one or more inputs. In some instances, the inputs can include sequence data (e.g., text, images, audio, multimodal data, etc.). In some instances, the sequence generation model 116 can be or includes a model configured to generate one or more outputs in a sequence based on one or more semantic embeddings (e.g., semantic embeddings generated by a lightweight sequence embedding model 108). For example, in some instances, the lightweight sequence generation model 108 and the sequence generation model 116 can each include an encoder and a decoder configured to operate on similar (e.g., identical) semantic embedding spaces. In some instances, the sequence generation model 116 can utilize attention mechanisms, such as self-attention. For example, the sequence generation model 116 can be a multi-head self-attention model (e.g., an encoder-only, encoder-decoder, or decoder-only transformer language model).

[0087] Figure 1 Synthetic insights 118 are depicted. Synthetic insights 118 can typically include or otherwise represent various types of data. Synthetic insights 118 can include one or more types of data. In some instances, synthetic insights 118 can include multimodal data (e.g., multiple types of sequence data, such as video and audio; sequence data with non-sequence data or metadata; etc.). In some instances, synthetic insights 118 can be characterized by a context window smaller than the size of the machine learning model of interest (e.g., sequence generation model 116). In some instances, synthetic insights 118 can be characterized by a size much smaller than the context window of the model of interest, such that the context window can receive multiple synthetic insights 118 or context blocks 106. Synthetic insights 118 can include, for example, sequence data generated based on one or more context blocks 106 or semantic block embeddings 110 (e.g., summaries, suggested questions, main ideas, topic descriptions, etc.). Synthetic insights 118 can include, for example, non-sequence data (e.g., metadata, such as sequence type, numerical or categorical topic identifiers, the number of times a particular term appears in the sequence, etc.). Although Figure 1 Sequence generation model 116 is described as generating synthetic insights 118, but synthetic insights 118 may be generated without using a machine learning model (e.g., using computing system 104) or using a machine learning model other than sequence generation model 116, without exceeding the scope of this disclosure.

[0088] Figure 1The selected context 120 is depicted. The selected context 120 may typically include or otherwise represent various types of data. In some instances, the selected context 120 may include or represent various types of sequential data (e.g., image data, audio data, text data, etc.). In some instances, the selected context 102 may include or represent various types of non-sequential data (e.g., metadata, synthesized insights 118, etc.). In some instances, the selected context 120 may include multimodal data (e.g., sequential data with multiple modes such as video and audio; combinations of sequential and non-sequential data; etc.). In some instances, the selected context 120 may include or represent semantic embedding data (e.g., when the sequence generation model 116 includes a model configured to generate sequential outputs from semantic embedding inputs). In some instances, the selected context 120 may include one or more context blocks 106. In some instances, the selected context 120 may include one or more synthesized insights 118.

[0089] Figure 1 A computational system 104 is depicted determining a selected context 120 based on one or more semantic clusters 114 or synthesized insights 118. Determining the selected context 120 may include, for example, a deterministic or probabilistic selection based on one or more priorities. In some instances, selection based on one or more priorities may include, for example, prioritizing context blocks 106 associated with semantic block embeddings 110 near the centroid of semantic cluster 114 according to a metric of semantic distance (e.g., cosine distance, etc.). The centroid of semantic cluster 114 may be defined in various ways, such as by averaging (e.g., arithmetic mean, geometric mean, etc.) multiple semantic block embeddings 110 associated with semantic cluster 114. In some instances, selection based on one or more priorities may include, for example, prioritizing context blocks 106 associated with semantic block embeddings 110 near the centroid of subclusters of hierarchically defined semantic cluster 114 according to a metric of semantic distance.

[0090] In some instances, selection based on one or more priorities may include prioritizing context blocks 106 associated with semantic clusters 114 having a large number of context blocks 106. In some instances, selection based on one or more priorities may include prioritizing context blocks 106 associated with semantic clusters 114 not currently represented in the selected context 120. In some instances, selection based on one or more priorities may include prioritizing context blocks 106 associated with hierarchically defined subclusters of semantic clusters 114 not currently represented in the selected context. In some instances, one or more priorities may be determined based on user input. For example, in some instances, selection based on one or more priorities may include prioritizing semantic block embeddings 110 or the centroid of semantic clusters 114 near semantic embeddings determined based on user input. In some instances, selection based on one or more priorities may include prioritizing context blocks 106 associated with high-information content (e.g., mathematical definitions based on information entropy) relative to context blocks 106 already selected for the selected context 120.

[0091] In some instances, selection based on one or more priorities may include hierarchical selection mechanisms, such as decision trees. In some instances, selection based on one or more priorities may include numerical selection mechanisms, such as weighted combinations of priorities. For example, in some instances, the selection mechanism may include assigning a corresponding priority score to each context block 106, where the priority score may include, for example, a weighted average of multiple priority sub-scores (e.g., distance to the nearest cluster or sub-cluster centroid, the number of context blocks 10 in the cluster, etc.).

[0092] In some instances, determining the selected context 120 may include generating the context sequence directly based on semantic embeddings. For example, in some instances, the centroid of semantic cluster 114 or a subcluster of semantic cluster 114 may be determined. In some instances, the centroid itself may be decoded without considering any particular context block 106 of semantic cluster 114. For example, in some instances, a machine learning-based sequence generation model may generate or otherwise decode the context sequence based on the position of the centroid in the semantic embedding space. In some instances, determining the selected context 120 may include decoding the context sequence based on semantic embeddings other than the centroid. For example, determining the selected context may include decoding the context sequence based on geometric transformations of the centroid or geometric transformations of semantic block embeddings 110. For example, geometric transformations may include transformations configured to correspond to specific semantic relations (e.g., opposition, hierarchy (such as is-a or has-a relations), analogy (such as king:queen:prince:princess) etc.).

[0093] In some instances, the selected context 120 may include one or more prompt sequences (e.g., “I’m going to give you some representative paragraphs from a book. Please summarize the book based on the following paragraphs:”, etc.). In some instances, the prompt sequence may include one or more user inputs (e.g., questions or instructions received from the user, etc.).

[0094] Figure 1 The generated output 122 is described. The generated output 122 may typically include or otherwise represent various types of data. In some instances, the generated output 122 may include or represent various types of sequential data (e.g., image data, audio data, text data, multimodal data, etc.) and non-sequential data or metadata. The generated output 122 may include one or more data types that are the same as or different from the data types of the long-form context 102, context block 106, synthesized insights 118, etc.

[0095] Figures 2A to 2C This is an illustration of an example hierarchical clustering algorithm, where clustering system 112 can group semantic block embeddings 110 into semantic clusters 114. Based on multiple unclustered semantic block embeddings 210 in the semantic embedding space 202, clustering system 112 can perform a first clustering to cluster the semantic block embeddings 210 into multiple first clusters 214a-f, each first cluster including one or more clustered semantic block embeddings 212a-f. Based on the first clusters 214a-f, clustering system 112 can perform a second clustering to create multiple second clusters 216, each second cluster including one or more first clusters. In some instances, the first cluster or second cluster 216 can be semantic cluster 114.

[0096] Figures 2A to 2CA semantic embedding space 202 is depicted. The semantic embedding space 202 may include, for example, a space of possible semantic embeddings (e.g., semantic block embedding 110) that can be generated by a machine learning model (e.g., a lightweight sequence embedding model 108) based on multiple possible input values ​​(e.g., possible values ​​of context block 106). The semantic embedding space 202 may include or represent, for example, any data type capable of representing semantic embeddings (e.g., semantic block embedding 110). In some instances, the semantic embedding space may be, for example, an n-dimensional vector space, where n may be an integer. Although the illustration in Figure 2 is presented in two dimensions for improved readability, it should be understood that in some instances, n may be greater than (e.g., much greater than) two.

[0097] Figure 2A Multiple unclustered semantic block embeddings 210 are depicted. Unclustered semantic block embeddings 210 can be, for example, semantic block embeddings 110 that have not yet been assigned to a cluster. When the clustering system 112 first obtains the semantic block embeddings 110, all semantic block embeddings 110 can be unclustered semantic block embeddings 210 in some instances.

[0098] Figures 2A to 2C Multiple clustered semantic block embeddings 212a-f are described. The clustered semantic block embeddings 212 can be, for example, semantic block embeddings 110 that have been assigned to a certain cluster.

[0099] Figures 2B to 2C Multiple first clusters 214a-f are described. In some instances, a first cluster 214 may be a semantic cluster 114, include a semantic cluster 114, be included by a semantic cluster 114, or share one or more characteristics with a semantic cluster 114.

[0100] Figure 2C Multiple second clusters 216 are described. In some instances, a second cluster 216 may be a semantic cluster 114, include a semantic cluster 114, be included by a semantic cluster 114, or share one or more characteristics with a semantic cluster 114.

[0101] Clustering system 112 can, for example, assign one or more unclustered semantic block embeddings 210 to one or more initial clusters. For example, Figure 2A Four clustered semantic block embeddings 212a-d, assigned to four corresponding first clusters 214, are depicted, where each cluster initially contains only one clustered semantic block embedding 212. In some instances, the clustering system 112 may determine one or more initially clustered semantic block embeddings 212a-d by random sampling. In some instances, the clustering system 112 may determine one or more initially clustered semantic block embeddings in another manner (e.g., based on multiple corresponding distances between the first clustered semantic block embedding 212a and multiple corresponding unclustered semantic block embeddings 210, etc.).

[0102] Based on the initially clustered semantic block embeddings 212a-d, the clustering system 112 can assign additional unclustered semantic block embeddings 210 to one or more clusters 214. In some instances, the assignment can be based on multiple corresponding distances (e.g., semantic distances) between the unclustered semantic block embeddings 210 and multiple corresponding centroids in the multiple corresponding first clusters 214a-d. In some instances, after each newly clustered semantic block embedding 212 is added to a first cluster 214, the position of the corresponding centroid in the semantic space 202 can be updated. In some instances, one or more clustered semantic block embeddings 212 can be removed from a cluster or reassigned to another cluster. In some instances, such reassignment can be based on multiple corresponding distances from the clustered semantic block embeddings to multiple updated corresponding centroids of the multiple first clusters 214.

[0103] In some instances, the final number of the first clusters 214 can be predetermined, and the initial number of first clusters 214a-d can be equal to the final number of the first clusters 214 (e.g., via k-means clustering). In other instances, the number of the first clusters 214 can be determined dynamically. For example, in some instances, the number of the first clusters 214 can be determined dynamically, at least in part, based on a target distribution of one or more first clusters 214. For example, in some instances, g-means clustering can be performed, where the number of the first clusters is dynamically selected such that the distribution of one or more first clusters 214 approximates a Gaussian distribution. In some instances, the number of the first clusters 214 can be determined dynamically based on other factors, such as one or more cluster density thresholds. Figures 2A to 2B The number of dynamically determined first clusters 214 is depicted, where 2A describes how the clustering system 112 initially selects four clustered semantic block embeddings 212a-d for four initial first clusters 214a-d (e.g., by random sampling), while Figure 2B The final number is depicted as six first clusters 214a-f.

[0104] Figure 2CMultiple second clusters 216, determined by clustering system 112 based on multiple first clusters 214a-f, are depicted. Determining the multiple second clusters 216 can include, for example, agglomerative hierarchical clustering (e.g., parhac, etc.). For example, in some instances, two or more first clusters 214a-f can be merged based on one or more similarities between them. Similarity can be based on, for example, one or more distances in the semantic embedding space 202 (e.g., the shortest minimum or shortest maximum distance between clustered semantic block embedding pairs 212a, 212b; the shortest distance between the centroids of the first clusters 214; the mean, median, or weighted average distance between clustered semantic block embedding pairs 212a, 212b; the minimum increase in variance; the minimum increase in sum of squares; etc.). In some instances, the final number of second clusters can be determined at least in part based on the size of the finite context window of sequence generation model 116. For example, the final number of second clusters can be at least in part based on the ratio between the finite context window size and the size of one or more context blocks 106. In this scenario, the second cluster can continue merging clusters until the target number of clusters for the second cluster is reached.

[0105] although Figures 2A to 2C Various example details are included for illustrative purposes, but alternative implementations are possible without departing from the scope of this disclosure. For example, the number of clusters can be greater than and less than [the specified number]. Figures 2A to 2C The quantity described in the text. Figures 2A to 2C The distances and boundaries depicted are not necessarily drawn to scale. Various clustering methods can be used to generate semantic clusters 114, regardless of whether they are... Figures 2A to 2C The text describes clustering methods such as centroid-based clustering (e.g., k-means), distribution-based clustering (e.g., g-means), connectivity-based clustering (e.g., agglomerated hierarchical clustering), dimensionality-reduced clustering (e.g., spectral clustering), density-based clustering, biclustering, split hierarchical clustering, affinity propagation, etc.). Despite... Figures 2A to 2C The first and second clusters are described, but it should be understood that the second cluster can be omitted, or a third cluster can be added, etc. For example, in some instances, agglomerative hierarchical clustering can be used alone, where each unclustered semantic block embedding 210 is initially considered as a single block “cluster” to be merged with other clusters until a target number of clusters is reached.

[0106] Example Method

[0107] Figure 3 A flowchart depicts an example method for semantic clustering according to an example embodiment of this disclosure. Although Figure 3The steps performed in a particular order are depicted for illustrative and discussion purposes, but the method of this disclosure is not limited to the order or arrangement specifically shown. The various steps of example method 300 may be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of this disclosure.

[0108] At 302, example method 300 may include obtaining a context sequence by one or more computing devices. In some instances, the computing device may be computing system 104, include computing system 104, or be included by computing system 104. In some instances, the context sequence may be long-form context 102, include long-form context 102, or be included by long-form context 102. In some instances, example method 300 may include: at 302, using one or more systems or performing... Figure 1 One or more activities described.

[0109] At 304, example method 300 may include determining multiple subsequences of a context sequence by one or more computing devices. In some instances, the subsequences may be context block 106, include context block 106, or be included by context block 106. In some instances, determining multiple subsequences of a context sequence may include splitting the context at least partially based on a subsequence length threshold. In some instances, determining multiple subsequences of a context sequence may include splitting the context at least partially based on one or more delimiters. In some instances, at least one of the one or more delimiters may include one or more numeric, alphanumeric, text, or whitespace characters. In some instances, at least one of the one or more delimiters may include one or more document boundaries. In some instances, determining multiple subsequences of a context sequence may include using a machine learning-based model. In some instances, example method 300 may include: at 304, using one or more systems or performing about Figure 1 One or more activities described.

[0110] At 306, example method 300 may include determining, by one or more computing devices, a first machine learning sequence processing model, multiple corresponding semantic embeddings associated with multiple subsequences of a context sequence. In some instances, the corresponding semantic embeddings may be semantic block embedding 110, include semantic block embedding 110, or be included by semantic block embedding 110. In some instances, the first machine learning model may be configured to receive a sequence including multiple lexical units as input; and generate a single semantic embedding based on at least two different lexical units among the multiple lexical units. In some instances, the first machine learning model may be configured to generate a semantic embedding for each input lexical unit it receives. In this example method, determining the semantic embeddings associated with a subsequence of a context sequence may include combining multiple semantic embeddings generated by the first machine learning model based on the subsequence. In some instances, example method 300 may include, at 306, using one or more systems or performing... Figure 1 One or more activities described.

[0111] At 308, example method 300 may include determining a plurality of semantic clusters by one or more computing devices, wherein each of the plurality of semantic clusters includes one or more semantic embeddings of the plurality of semantic embeddings. In some instances, the semantic clusters may be semantic cluster 114, include semantic cluster 114, or be included by semantic cluster 114. In some instances, determining the plurality of semantic clusters may include hierarchical clustering. In some instances, example method 300 may include: at 308, using one or more systems or performing about Figure 1 Or one or more activities as described in Figure 2.

[0112] At 310, example method 300 may include selecting one or more subsequences of a plurality of subsequences of a context sequence by one or more computing devices, at least in part, based on the plurality of semantic clusters. In some instances, the selected subsequence of the context sequence may be a selected context 120, include the selected context 120, or be included by the selected context 120. In some instances, selecting one or more subsequences may include selection based on one or more priorities. In some instances, selection based on one or more priorities may include prioritizing clusters associated with a larger number of semantic embeddings. In some instances, selection based on one or more priorities may include prioritizing semantic embeddings near the center of a semantic cluster. In some instances, the center of a semantic cluster may be determined by averaging the plurality of semantic embeddings of the semantic cluster. In some instances, one or more priorities may include priorities determined based on user input. In some instances, selection based on one or more priorities may include prioritizing semantic embeddings near semantic embeddings associated with user input. In some instances, selection based on one or more priorities may include prioritizing semantic embeddings from clusters having centroids near semantic embeddings associated with user input. In some instances, at least one of the plurality of semantic clusters may include a plurality of subclusters. In some instances, selection based on one or more priorities may include prioritizing the selection of at least one subsequence from each of two or more subclusters in the plurality of subclusters over the selection of a second subsequence from any of the plurality of subclusters. In some instances, selection based on one or more priorities may include prioritizing semantic embeddings near the centroid of subclusters in the plurality of subclusters. In some instances, selection based on one or more priorities may include prioritizing the selection of at least one subsequence from each of the plurality of semantic clusters over the selection of a second subsequence from any of the plurality of semantic clusters. In some instances, example method 300 may include, at 310, using one or more systems or performing... Figure 1 One or more activities described.

[0113] For example, an example selection algorithm can begin with the largest semantic cluster among multiple semantic clusters, selecting the subsequence whose semantic embedding is closest to the center of the largest semantic cluster. Next, the example selection algorithm can select from the second largest cluster, then from the third largest cluster, and so on, until: (a) the context window size constraint is satisfied, or (b) at least one corresponding subsequence has been selected from each of the multiple semantic clusters. If there is remaining space in the context window after each semantic cluster has been represented, a second cluster can be selected from the largest cluster. In some instances, the second cluster may have an embedding that is second closest to the center of the largest semantic cluster, or an embedding that is closest to the center of the hierarchical subcluster of the largest semantic cluster. After each corresponding semantic cluster has been represented in the context window, the corresponding number of additional subsequences added from the corresponding cluster (e.g., the largest cluster, the second largest cluster, etc.) can be based on the size of the corresponding cluster (e.g., roughly proportional to the size of the corresponding cluster).

[0114] To give another example, the example selection algorithm can begin with a semantic cluster having a center that is closest to the semantic embedding of the user input. The example algorithm can then select, for example, (a) the subsequence closest to the cluster center, or (b) the subsequence closest to the semantic embedding of the user input. Next, the example algorithm can select from the cluster that is second closest to the semantic embedding of the user input, and so on, until all clusters have been represented. After each corresponding semantic cluster or subcluster has been represented in the context window, the corresponding number of additional subsequences added from the corresponding cluster or subcluster (e.g., the cluster closest to the user input, the second closest cluster, etc.) can be based on the distance between the center of the corresponding cluster and the semantic embedding of the user input (e.g., inversely proportional to it).

[0115] At 312, example method 300 may include adding data associated with the one or more subsequences to a context window of a second machine learning-based sequence processing model by one or more computing devices. In some instances, the second machine learning-based sequence processing model may be a sequence generation model 116, include a sequence generation model 116, or be included by a sequence generation model 116. In some instances, the second machine learning model may be a first machine learning model, or a model other than the first machine learning model. In some instances, the data associated with the one or more subsequences may include at least one subsequence of the one or more subsequences. In some instances, the data associated with the one or more subsequences may include metadata associated with at least one subsequence of the one or more subsequences. In some instances, the data associated with the one or more subsequences may include a machine-generated summary of at least one subsequence of the one or more subsequences. In some instances, the data associated with the one or more subsequences may include one or more lexical units generated by one or more computing devices based at least in part on semantic embeddings associated with at least one subsequence of the one or more subsequences. In some instances, the data associated with the one or more subsequences may include one or more corresponding semantic embeddings of a plurality of corresponding semantic embeddings. In some instances, the data associated with one or more subsequences may include one or more lexical units generated by one or more computing devices based at least in part on geometric transformations of semantic embeddings associated with at least one of the one or more subsequences. In some instances, example method 300 may include: at 312, using one or more systems or performing... Figure 1 One or more activities described.

[0116] In some instances, example method 300 may include, at 312, adding a hint to the context window of the second machine learning model. In some instances, the hint may include one or more instructions or explanations.

[0117] In some instances, the first machine learning model may be characterized by a first number of parameters; the second machine learning model may be characterized by a second number of parameters; and the second number may be at least five times the first number. In some instances, the second number may be at least ten times the first number. In some instances, the second number may be at least thirty times the first number.

[0118] At 314, example method 300 may include generating an output sequence by one or more computing devices using a second machine learning-based sequence processing model, at least partially based on the plurality of semantic clusters. In some instances, the output sequence may be, include, or be comprised of the generated output 122. In some instances, the output sequence may be generated at least partially based on a context window. In some instances, example method 300 may include, at 314, using one or more systems or performing operations related to... Figure 1 One or more activities described.

[0119] Figure 4 A flowchart depicts a method 400 for training one or more machine learning-based models according to aspects of this disclosure. For example, an example machine learning-based model may include a lightweight sequence embedding model 108 or a sequence generation model 116.

[0120] One or more portions of Example Method 400 may be implemented by a computing system (such as a computing system described, for example, with reference to other diagrams) including one or more computing devices. Each corresponding portion of Example Method 400 may be executed by any one (or any combination of) of the one or more computing devices. Furthermore, one or more portions of Example Method 400 may be implemented on the hardware components of the apparatus described herein, for example, to train one or more systems or models. Figure 4 For illustrative and discussion purposes, elements executed in a specific order are depicted. Those skilled in the art will understand using the disclosure provided herein that elements of any of the methods discussed herein can be adapted, rearranged, extended, omitted, combined, or modified in various ways without departing from the scope of this disclosure. Figure 4 The description is for illustrative purposes only and with reference to elements / terms described with respect to other systems and figures, and is not intended to be limiting. One or more portions of example method 400 may be performed additionally or alternatively by other systems.

[0121] At 402, example method 400 may include obtaining training instances. The training dataset may include multiple training instances partitioned across multiple datasets (e.g., training datasets, validation datasets, or test datasets). Training instances may be labeled or unlabeled. Although referred to as “training” instances in example method 400, it should be understood that runtime inference may also form training instances when training (e.g., online training / learning) a model using an evaluation of the model’s performance on that runtime instance. Example data types for training instances and various tasks associated with them are described throughout this disclosure.

[0122] At 404, example method 400 may include using one or more machine learning models to process training instances to generate output. This output may be obtained directly from the one or more machine learning models, or it may be a downstream result of a processing operation chain that includes the output of the one or more machine learning models.

[0123] At 406, example method 400 may include receiving an evaluation signal associated with the output. The evaluation signal can be obtained using a loss function. Various losses can be used, such as mean squared error, likelihood loss, cross-entropy loss, hinge loss, contrastive loss, or various other loss functions. The evaluation signal can be computed using known baseline ground truth labels (e.g., supervised learning), predicted or estimated labels (e.g., semi-supervised or self-supervised learning), or without labels (e.g., unsupervised learning). The evaluation signal can be a reward (e.g., for reinforcement learning). The reward can be computed using a machine learning-based reward model configured to generate a reward based on the received output. The reward can also be computed using feedback data describing human feedback to the output.

[0124] At 408, example method 400 may include using an evaluation signal to update a machine learning model. For example, in some embodiments, various training or learning techniques, such as backpropagation, may be used to learn the values ​​of the parameters of the machine learning model. For example, the evaluation signal may be backpropagated from the output (or another source of the evaluation signal) through the machine learning model to update one or more parameters of the model (e.g., based on the gradient of the evaluation signal relative to the parameter values). For example, a system containing one or more machine learning models may be trained in an end-to-end manner. Gradient descent techniques may be used to iteratively update the parameters over multiple training iterations. In some implementations, performing error backpropagation may include performing truncated backpropagation over time. Example method 400 may include implementing various generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization ability of the trained model.

[0125] In some implementations, example method 400 can be implemented to train a machine learning model from an initial state to a fully trained state (e.g., when the model exhibits a desired performance profile, such as based on accuracy, precision, recall, etc.).

[0126] In some implementations, example method 400 can be implemented for specific stages of the training process. For example, in some implementations, example method 400 can be implemented for pre-training a machine learning model. Pre-training can include, for example, large-scale training on potentially noisy data to achieve a broad performance level base across multiple tasks / data types. In some implementations, example method 400 can be implemented for fine-tuning a machine learning model. Fine-tuning can include, for example, smaller-scale training on higher-quality (e.g., labeled, curated, etc.) data. Fine-tuning can affect all or part of the parameters of the machine learning model. For example, parts of the machine learning model can be “frozen” for certain training stages. For example, parameters associated with the embedding space can be “frozen” during fine-tuning (e.g., to preserve information learned from a broader domain than that present in the fine-tuned dataset). Example fine-tuning methods include reinforcement learning. Reinforcement learning can be based on user feedback on model performance during use.

[0127] Example of a machine learning model

[0128] Figure 5 This is a block diagram of an example processing flow for using a machine learning-engineered model 1 to process input 2 to generate output 3.

[0129] The machine learning model 1 can be or includes one or more machine learning models or model components. Example machine learning models can include neural networks (e.g., deep neural networks). Example machine learning models can include non-linear or linear models. Instead of or in addition to neural networks, example machine learning models can use other architectures. Example machine learning models can include decision tree-based models, support vector machines, hidden Markov models, Bayesian networks, linear regression models, k-means clustering models, etc.

[0130] Example neural networks can include feedforward neural networks, recurrent neural networks (RNNs) (including long short-term memory (LSTM) based RNNs), convolutional neural networks (CNNs), diffusion models, generative adversarial networks, or other forms of neural networks. Example neural networks can also be deep neural networks. Some example machine learning models may utilize attention mechanisms, such as self-attention. For example, some example machine learning models may include multi-head self-attention models.

[0131] The machine learning-based model 1 may include one or more instances of the same model configured to operate on data from input 2. The machine learning-based model 1 may also include a collection of different models that can collaboratively interact to process data from input 2. For example, the machine learning-based model 1 may employ a hybrid expert architecture. See, for example, Zhou et al., Mixture-of-Experts with Expert Choice Routing, arXiv:2202.09368v2 (October 14, 2022).

[0132] Input 2 can generally include or otherwise represent various types of data. Input 2 can include one type or many different types of data. Output 3 can be data of the same type or a different type compared to the data in Input 2. Output 3 can include one type or many different types of data.

[0133] Example data types for input 2 or output 3 include natural language text data, software code data (e.g., source code, object code, machine code, or any other form of computer-readable instruction or programming language), machine code data (e.g., binary code, assembly code, or other forms of machine-readable instruction that can be executed directly by a computer's central processing unit), assembly code data (e.g., a low-level programming language that uses a symbolic representation of machine code instructions to program the processing unit), genetic data or other chemical or biochemical data, image data, audio data, audiovisual data, tactile data, biometric data, medical data, financial data, statistical data, geographic data, astronomical data, historical data, and generally sensor data (e.g., digital or analog values, such as voltage or other absolute or relative level measurements from real or artificial inputs, such as from audio sensors, light sensors, displacement sensors, etc.). Data can be raw or processed and can be in any format or mode.

[0134] In multimodal input 2 or output 3, example combinations of data types include image data and audio data, image data and natural language data, natural language data and software code data, image data and biometric data, sensor data and medical data, etc. It should be understood that any combination of data types in input 2 or output 3 can exist.

[0135] Example input 2 may include one or more data types, such as the example data types noted above. Example output 3 may include one or more data types, such as the example data types noted above. The data type of input 2 may be the same as or different from the data type of output 3. It should be understood that the example data types noted above are provided for illustrative purposes only. The data types contemplated within the scope of this disclosure are not limited to those examples noted above.

[0136] Example of a sequence processing model using machine learning

[0137] Figure 6 This is a block diagram illustrating an example implementation of a machine learning-based model configured to process information sequences. For example, an example implementation of machine learning model 1 could include a machine learning sequence processing model 4. The example system could pass input 2 to sequence processing model 4. Sequence processing model 4 could include one or more machine learning components. Sequence processing model 4 could process the data from input 2 to obtain input sequence 5. Input sequence 5 could include one or more input elements 5-1, 5-2, ..., 5-M, etc., obtained from input 2. Sequence processing model 4 could use a prediction layer 6 to process input sequence 5 to generate output sequence 7. Output sequence 7 could include one or more output elements 7-1, 7-2, ..., 7-N, etc., generated based on input sequence 5. The system could generate output 3 based on output sequence 7.

[0138] Sequence processing models 4 may include one or more machine learning-based model components configured to ingest, generate, or otherwise infer sequences of information. For example, some example sequence processing models in the text domain are referred to as “large language models” or LLMs. See, for example, the PaLM 2 technology report, Google, https: / / ai.google / static / documents / palm2techreport.pdf (nd). Other example sequence processing models can operate in other domains, such as the image domain (see, for example, Dosovitskiy et al., An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale), arXiv:2010.11929v2 (June 3, 2021); the audio domain (see, for example, Agostinelli et al., MusicLM: Generating Music From Text), arXiv:2301.11325v1 (January 26, 2023); and the biochemical domain (see, for example, Jumper et al., Highly accurate protein structure prediction with AlphaFold), 596 Nature 583 (August 26, 2021). Sequence processing model 4 can process one or more types of data simultaneously. Sequence processing model 4 can include relatively large models (e.g., more parameters, computationally intensive, etc.), relatively small models (e.g., fewer parameters, computationally lightweight, etc.) or both.

[0139] Generally, sequence processing model 4 can use data from input 2 to obtain input sequence 5. For example, input sequence 5 may include a representation of the data from input 2 in a format understood by sequence processing model 4. One or more machine learning components of sequence processing model 4 may ingest data from input 2, parse the data into fragments compatible with the processing architecture of sequence processing model 4 (e.g., via “word segmentation”), and project the fragments into the input space associated with prediction layer 6 (e.g., via “embedding”).

[0140] Sequence processing model 4 can ingest data from input 2 and parse the data into a sequence of elements to obtain input sequence 5. For example, a portion of the input data from input 2 can be decomposed into segments, which together represent the content of that portion of the input data. The segments can provide the elements of the sequence.

[0141] In some cases, elements 5-1, 5-2, ..., 5-M can represent building blocks used to capture or express meaningful information in a specific data domain. For example, elements can describe "atomic units" across one or more domains. For instance, for a text input source, elements can correspond to a group of one or more words or sub-word components (such as a set of one or more characters).

[0142] For example, elements 5-1, 5-2, ..., 5-M can represent tokens obtained using a word segmenter. For instance, a word segmenter can process a given portion of the input source and output a series of tokens representing that portion of the input source (e.g., corresponding to the input elements 5-1, 5-2, ..., 5-M). Various word segmentation methods can be used. For example, byte-pair encoding (BPE) can be used to segment the text input source. For example, see Kudo et al., SentencePiece: A simple and language-independent subword tokenizer and detokenizer for Neural Text Processing, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (SystemDemonstrations), pp. 66–71 (October 31–November 4, 2018), https: / / aclanthology.org / D18-2012.pdf. Lexicalization of image-based input sources can be achieved by extracting and serializing fragments from images.

[0143] Generally speaking, any data type can be serialized and processed into the input sequence 5. It should be understood that... Figure 6 The elements 5-1, 5-2, ..., 5-M described in the text can be lexical units or their embedded representations.

[0144] Prediction layer 6 can predict one or more output elements 7-1, 7-2, ..., 7-N based on the input elements. Prediction layer 6 may include one or more machine learning-based model architectures, such as one or more learned parameter layers, which manipulate and transform the input to extract higher-order meanings and relationships between the input elements 5-1, 5-2, ..., 5-M. In this way, for example, example prediction layer 6 can predict new output elements based on the context provided by the input sequence 5.

[0145] Prediction layer 6 can evaluate the associations between portions of the input sequence 5 and specific output elements. These associations can inform predictions about the likelihood that a particular output follows the input context. For example, consider the text snippet “The carpenter's toolbox was small and heavy. It was full of ___. ” Example prediction layer 6 can identify that “It” refers back to “toolbox” by determining the relationships between the corresponding embeddings. Example prediction layer 6 can also associate “it” with attributes of the toolbox, such as “small” and “heavy”. Based on these associations, for example, prediction layer 6 can assign a higher probability to the word “nails” than to the word “sawdust”.

[0146] The transformer is an example architecture that can be used in prediction layer 4. See, for example, Vaswani et al., Attention Is All You Need, arXiv:1706.03762v7 (August 2, 2023). The transformer is an example of a machine learning-based model architecture that uses an attention mechanism to compute associations between items within a context window. The context window can include a sequence containing an input sequence 5 and potentially one or more output elements 7-1, 7-2, ..., 7-N. The transformer block can include one or more attention layers and one or more post-attention layers (e.g., feedforward layers, such as multilayer perceptrons).

[0147] In addition to or instead of transformer-based architectures, prediction layer 6 can include other machine learning-based model architectures. For example, recurrent neural networks (RNNs) and long short-term memory (LSTM) models, as well as convolutional neural networks (CNNs), can also be used. In general, prediction layer 6 can leverage various artificial neural networks that can understand or generate sequences of information.

[0148] The output sequence 7 may include or otherwise represent the same or different data type as the input sequence 5. For example, the input sequence 5 may represent text data, and the output sequence 7 may represent text data. The input sequence 5 may represent image, audio, or audiovisual data, and the output sequence 7 may represent text data (e.g., describing image, audio, or audiovisual data). It should be understood that the prediction layer 6 and any other gap model components of the sequence processing model 4 can be configured to receive multiple data types from the input sequence 5 and output multiple data types in the output sequence 7.

[0149] Output sequence 7 can have various relationships with input sequence 5. Output sequence 7 can be a continuation of input sequence 5. Output sequence 7 can be a supplement to input sequence 5. Output sequence 7 can translate, transform, expand, or otherwise modify input sequence 5. Output sequence 7 can answer, evaluate, acknowledge, or otherwise respond to input sequence 5. Output sequence 7 can implement instructions provided via input sequence 5 (or describe instructions for implementing those instructions).

[0150] Output sequence 7 can be generated autoregressively. For example, for some applications, the output of one or more prediction layers 6 can be passed through one or more output layers (e.g., softmax layers) to obtain a probability distribution of an output vocabulary (e.g., a text or symbol vocabulary) conditioned on the set of input elements in the context window. In this way, output sequence 7 can be generated autoregressively, for example, by sampling possible next output elements, adding that element to the context window, regenerating the probability distribution based on the updated context window, and sampling possible next output elements, etc.

[0151] Output sequence 7 can also be generated non-autoregressively. For example, multiple output elements of output sequence 7 can be predicted together without explicit order conditions. See, for example, Saharia et al., Non-Autoregressive Machine Translation with Latent Alignments, arXiv:2004.07437v3 (November 16, 2020).

[0152] Output sequence 7 may include one or more parts or elements. In the example content generation configuration, output sequence 7 may include multiple elements corresponding to multiple parts of the generated output sequence (e.g., text sentences, values ​​of discrete waveforms, computer code, etc.). In the example classification configuration, output sequence 7 may include a single element associated with the classification output. For example, the output "vocabulary" may include a set of classes to which the input sequence will be classified. For example, a vision transformer block may pass latent state information to a multilayer perceptron, which outputs possible class values ​​associated with the input image.

[0153] Figure 7 This is a block diagram of an example technique for populating example input sequence 8. Input sequence 8 may include various functional elements forming part of the model infrastructure, such as element 8-0 obtained from task indicator 9, which signals to any model processing input sequence 8 that a specific task is being performed (e.g., helping to adapt the model's performance to that specific task). Input sequence 8 may include various data elements from different data modalities. For example, input modality 10-1 may include one data modality. Data-to-sequence model 11-1 may process the data from input modality 10-1 to project the data into a format compatible with input sequence 8 (e.g., determining one or more vectors of dimensions based on the dimensions of input sequence 8) to obtain elements 8-1, 8-2, 8-3. Another input modality 10-2 may include a different data modality. Data-to-sequence model 11-2 may project the data from input modality 10-2 into a format compatible with input sequence 8 to obtain elements 8-4, 8-5, 8-6. Another input modality 10-3 may include yet another different data modality. The data to sequence model 11-3 can project the data from the input mode 10-3 into a format compatible with the input sequence 8 to obtain elements 8-7, 8-8, and 8-9.

[0154] Input sequence 8 may be the same as or different from input sequence 5. Input sequence 8 may be a multimodal input sequence containing elements representing data from different modalities using a common dimensional representation. For example, the embedding space may have P dimensions. Input sequence 8 may be configured to contain multiple elements with P dimensions. In this way, for example, the example implementation can facilitate information extraction and reasoning across different data modalities by projecting data onto elements in the same embedding space to compare, combine, or otherwise compute between them.

[0155] For example, elements 8-0, ..., 8-9 can indicate specific locations within a multidimensional embedding space. Some elements can map to a discrete set of locations in the embedding space. For example, elements corresponding to discrete members in a predefined lexicon can map to discrete locations in the embedding space associated with those lexicons. Other elements can be continuously distributed across the embedding space. For example, some data types can be decomposed into continuously defined parts (e.g., image patches), which can be described using continuously distributed locations within the embedding space.

[0156] In some implementations, the expressive power of an embedding space may not be limited to the meaning associated with any particular set of lexical units or other building blocks. For example, a continuous embedding space can encode a series of higher-order information. A single piece of information (e.g., a lexical unit) can be mapped to a specific point in the space: for example, the lexical unit of the word "dog" can be projected to an embedding value that points to a specific location in the embedding space associated with dog-related information. Similarly, an image patch of a dog on grass can be projected into the embedding space. In some implementations, the projection of the dog image can be similar to the projection of the word "dog," and also similar to the projection of the word "grass," but potentially different from both. In some implementations, the projection of the image patch may not be perfectly aligned with any single projection of a single word. In some implementations, the projection of the image patch can be aligned with a combination of the projections of the words "dog" and "grass." In this way, for example, a higher-order embedding space can encode information that is independent of the data modality expressing the information.

[0157] Task indicator 9 may include a model or model component configured to identify the ongoing task and inject the input value represented by element 8-0 into input sequence 8, which signals which task is in progress. For example, the input value may be provided as a data type associated with an input modality and projected along with that modality (e.g., the input value may be a text task label embedded along with other text data in the input; the input value may be a pixel-based representation of the task embedded along with other image data in the input; etc.). The input value may be provided as a data type different from or at least independent of other inputs. For example, the input value represented by element 8-0 may be learned within a continuous embedding space.

[0158] Input modes 10⁻¹, 10⁻², and 10⁻³ can be associated with a variety of different data types (e.g., as described above with respect to input 2 and output 3).

[0159] Data-to-sequence models 11-1, 11-2, and 11-3 may be the same as or different from each other. Data-to-sequence models 11-1, 11-2, and 11-3 may be adapted to each corresponding input modality 10-1, 10-2, and 10-3. For example, a text data-to-sequence model may subdivide a portion of the input text and project the subdivisions onto elements in input sequence 8 (e.g., elements 8-1, 8-2, 8-3, etc.). An image data-to-sequence model may subdivide the input image and project these subdivisions onto elements in input sequence 8 (e.g., elements 8-4, 8-5, 8-6, etc.). An arbitrary data type data-to-sequence model may subdivide the input of that arbitrary data type and project these subdivisions onto elements in input sequence 8 (e.g., elements 8-7, 8-8, 8-9, etc.).

[0160] Data-to-sequence models 11-1, 11-2, and 11-3 can form part of the machine learning-based sequence processing model 4. Data-to-sequence models 11-1, 11-2, and 11-3 can be jointly trained with the machine learning-based sequence processing model 4 or trained independently of it. Data-to-sequence models 11-1, 11-2, and 11-3 can be trained end-to-end with the machine learning-based sequence processing model 4.

[0161] Example of a machine learning-based model development platform

[0162] Figure 8 This is a block diagram of the example model development platform 12, which facilitates the creation, adaptation, and refinement of example machine learning models (e.g., machine learning model 1, sequence processing model 4, etc.). The model development platform 12 can provide several different toolkits that developer systems can use to develop new or adapted machine learning models.

[0163] The model development platform 12 can provide one or more model libraries 13 containing building blocks for new models. Model libraries 13 can include one or more pre-trained base models 13-1, which can provide a backbone of processing power across a variety of tasks. Model libraries 13 can include one or more pre-trained expert models 13-2, which can focus on performance in a specific domain. Model libraries 13 can include various model primitives 13-3, which can provide low-level architectures or components (optionally pre-trained) that can be assembled in various arrangements as needed.

[0164] The model development platform 12 can receive selections of various model components 14. The model development platform 12 can transfer the selected model components 14 to the workbench 15, which combines the selected model components 14 into the development model 16.

[0165] Workbench 15 can facilitate further refinement and adaptation of the development model 16 by utilizing multiple different toolkits integrated with the model development platform 12. For example, workbench 15 can facilitate the use of model alignment toolkit 17 to align the development model 16 with expected performance profiles for various tasks.

[0166] The model alignment toolkit 17 can provide a variety of tools for enabling the development model 16 to generate outputs aligned with desired behavioral characteristics. Alignment can include increasing the accuracy, precision, recall, etc., of the model output. Alignment can include enforcing output styles, patterns, or other preferred characteristics of the model output. Alignment can be general or domain-specific. For example, the pre-trained base model 13-1 can start from an initial performance level across multiple domains. Alignment of the pre-trained base model 13-1 can include improving performance in a specific information or task domain (e.g., even at the expense of performance in another information or task domain).

[0167] The model alignment toolkit 17 can integrate one or more datasets 17-1 used to align development models 16. Selected datasets 17-1 may include labeled or unlabeled training data. Datasets 17-1 can be obtained from public domain datasets. Datasets 17-1 can also be obtained from private datasets associated with one or more developer systems used to align custom machine learning models tailored for private use cases.

[0168] The pre-training pipeline 17-2 may include a machine learning-based model training workflow configured to update the development model 16 on a large-scale, potentially noisy dataset. For example, pre-training may utilize unsupervised learning techniques (e.g., denoising, etc.) to process a large number of training instances to update model parameters from an initial state and achieve the desired baseline performance. The pre-training pipeline 17-2 may utilize the unlabeled dataset from dataset 17-1 for pre-training. Workbench 15 may implement the pre-training pipeline 17-2 to pre-train the development model 16.

[0169] The fine-tuning pipeline 17-3 may include a machine learning-based model training workflow configured to refine the model parameters of the development model 16 using higher-quality data. The fine-tuning pipeline 17-3 can update the development model 16 through supervised training using a labeled dataset from dataset 17-1. The fine-tuning pipeline 17-3 can also update the development model 16 through reinforcement learning using reward signals from user feedback. Workbench 15 enables the fine-tuning pipeline 17-3 to fine-tune the development model 16.

[0170] Hint library 17-4 may include a set of inputs configured to induce behavior aligned with desired performance criteria. Hint library 17-4 may include few-shot hints (e.g., providing examples of the desired model output for appending a header to the input of the desired runtime query), thought chain hints (e.g., providing step-by-step reasoning within an example to facilitate comprehensive reasoning by the model), and so on.

[0171] Sample hints can be retrieved from the available repository of hint library 17-4. Sample hints can be facilitated by one or more developer systems using workbench 15.

[0172] In some implementations, a pre-trained or fine-tuned model can achieve satisfactory performance even when there are no paradigms in the input. For example, zero-shot hints can include inputs lacking paradigms. Zero-shot hints can be within the domain of the training dataset or outside the training domain.

[0173] Hint library 17-4 may include one or more hint engineering tools. Hint engineering tools can provide a workflow for retrieving or learning optimized hint values. Hint engineering tools can facilitate the direct learning of hint values ​​(e.g., input element values) based on one or more training iterations. Workbench 15 can implement the hint engineering tools in development model 16.

[0174] Hint library 17-4 may include a pipeline for hint generation. For example, input can be generated using development model 16 itself or other machine learning-based models. In this way, for example, a first model can process information about the task and output input for a second model to process in order to perform the steps of the task. The second model may be the same as or different from the first model. Workbench 15 can implement the hint generation pipeline within development model 16.

[0175] Hint library 17-4 may include a pipeline for context injection. For example, if additional context is provided for performing a specific task, the performance of development model 16 on that task can be improved. Hint library 17-4 may include software components configured to identify desired context, retrieve context from external sources (e.g., databases, sensors, etc.), and add the context to input hints. Workbench 15 can implement the context injection pipeline in development model 16.

[0176] Although the various training examples described herein regarding model development platform 12 involve "pre-training" and "fine-tuning," it should be understood that the model alignment toolkit 17 can generally support a wide variety of training techniques suitable for training a wide range of machine learning models. Example training techniques may correspond to the example training method 400 described above.

[0177] The model development platform 12 may include a model plug-in toolkit 18. The model plug-in toolkit 18 may include a variety of tools configured to enhance the functionality of the machine learning model by integrating it with other systems, devices, and software components. For example, the machine learning model may use tools to improve performance quality where appropriate. For instance, deterministic tasks may be offloaded to dedicated tools instead of performing tasks probabilistically when the risk of error increases. For example, instead of autoregressively predicting solutions to a system of equations, the machine learning model may identify the tools invoked to obtain solutions and pass the system of equations to the appropriate tool. This tool may be a conventional equation solver that operates deterministically to solve the system of equations. The tool's output may be returned in response to the original query. In this way, tool usage can allow some example models to focus on the strengths of the machine learning model—e.g., understanding intent in unstructured requests for a task—while enhancing model performance by offloading certain tasks to more focused tools to mechanically apply deterministic algorithms to well-defined problems.

[0178] The model plugin toolkit 18 may include a validation tool 18-1. The validation tool 18-1 may include tools that can parse and validate the output of the machine learning model. The validation tool 18-1 may include engineered heuristics that establish certain thresholds applied to the model output. For example, the validation tool 18-1 may base the output of the machine learning model on a structured data source (e.g., to mitigate "illusion").

[0179] The model plugin toolkit 18 may include a toolkit 18-2 for implementing one or more tools, which may include scripts or other executable code that can be executed with the development model 16. The toolkit 18-2 may include one or more inputs configured to cause the machine learning-based model to implement the tools (e.g., few-shot hints that induce the model to output tool calls with correct syntax). For example, the toolkit 18-2 may include fine-tuned training data for training the model to use the tools.

[0180] The model plugin toolkit 18 may include interfaces for calling external application programming interfaces (APIs) 18-3. For example, attached to or replacing the direct implementation of tool calls or tool code using development model 16, development model 16 may be aligned with output instructions that initiate API calls to send or retrieve data via external systems.

[0181] The model plugin toolkit 18 can be integrated with the hint library 17-4 to create a catalog of available tools for use with the development model 16. For example, the model can receive a catalog of available tools in its input, and the model can generate output that selects a tool from the available tools and initiates a tool call for using that tool.

[0182] Model development platform 12 may include a suite of computational optimization tools 19 for optimizing the computational performance of development model 16. For example, tools for model compression 19-1 may allow development model 16 to be reduced in size while maintaining the desired performance level. For example, model compression 19-1 may include quantization workflows, weight pruning, and sparsification techniques. Tools for hardware acceleration 19-2 may facilitate the configuration of model storage and execution formats for optimal operation on different hardware resources. For example, hardware acceleration 19-2 may include tools for optimally sharding the model for distributed processing across multiple processing units to increase bandwidth, reduce uniform memory requirements, etc. Tools for distillation 19-3 may provide tools for training a lighter model based on knowledge encoded in development model 16. For example, development model 16 may be a large, high-performance machine learning model optimized using model development platform 12. To obtain a lightweight model for operation in resource-constrained environments, the smaller model may be a “student model” that learns from and imitates development model 16 as the “teacher model.” In this way, for example, the investment in learning and developing the parameters and configuration of model 16 can be efficiently transferred to a smaller model for more efficient inference.

[0183] Workbench 15 may implement one or more of the toolkits implemented in model development platform 12, or may not implement any toolkits. Workbench 15 may output output model 20 based on development model 16. Output model 20 may be a deployment version of development model 16. Output model 20 may be a development or training checkpoint of development model 16. Output model 20 may be a distilled version, compressed version, or otherwise optimized version of development model 16.

[0184] Figure 9 This is a block diagram of an example training process for training a machine learning development model 16. One or more parts of the example training process can be implemented by a computing system (such as, for example, the computing system described with reference to other diagrams) including one or more computing devices. Each corresponding part of the example training process can be performed by any one (or any combination of) of the one or more computing devices. Furthermore, one or more parts of the example training process can be implemented on the hardware components of the apparatus described herein, for example, to train one or more systems or models. Figure 9 For illustrative and discussion purposes, elements performed in a particular order are depicted. Those skilled in the art will understand using the disclosure provided herein that elements of any of the methods discussed herein can be adapted, rearranged, extended, omitted, combined, or modified in various ways without departing from the scope of this disclosure. Figure 9 The descriptions of elements / terms are for illustrative purposes only and are not intended to be limiting. One or more parts of the example training process may be additionally or alternatively performed by other systems.

[0185] Initially, development model 16 can be kept in its initial state as initialization model 21. Development model 16 can be initialized using weight values. The initial weight values ​​can be randomized or based on an initialization scheme. The initial weight values ​​can be based on previous pre-training for the same or different models.

[0186] The initialization model 21 can undergo pre-training in the pre-training phase 22. The pre-training phase 22 can be implemented using one or more pre-training pipelines 17-2 on data from dataset 17-1. For example, if the initialization model 21 has already been pre-trained (e.g., the development model 16 contains, is, or is based on a pre-trained base model or expert model), pre-training can be omitted.

[0187] The pre-trained model 23 can then be a new version of the development model 16, which can remain as the development model 16 or be a new development model. If the development model 16 has already been pre-trained, the pre-trained model 23 can be in its initial state. The pre-trained model 23 can undergo fine-tuning in the fine-tuning phase 24. The fine-tuning phase 24 can be implemented using one or more fine-tuning pipelines 17-3 on data from dataset 17-1. For example, fine-tuning can be omitted if the pre-trained model has satisfactory performance, if the model has already been fine-tuned, or if other tuning methods are preferred.

[0188] The fine-tuned model 29 can then be a new version of the development model 16, which can remain as the development model 16 or a new development model. If the development model 16 has already been fine-tuned, the fine-tuned model 29 can be in its initial state. The fine-tuned model 29 can undergo refinement 26 using user feedback. For example, refinement 26 using user feedback can optionally include reinforcement learning based on human feedback from human users of the fine-tuned model 25. Since reinforcement learning can take the form of fine-tuning, it should be understood that the fine-tuning phase 24 can include a phase for refinement 26 using user feedback. Refinement 26 using user feedback can produce a refined model 27. The refined model 27 can be output to the downstream system 28 for deployment or further development.

[0189] In some implementations, computational optimization operations can be applied before, during, or after each stage. For example, initializing model 21 may undergo computational optimization 29-1 (e.g., using computational optimization toolkit 19) before pre-training stage 22. Pre-trained model 23 may undergo computational optimization 29-2 (e.g., using computational optimization toolkit 19) before fine-tuning stage 24. Fine-tuned model 25 may undergo computational optimization 29-3 (e.g., using computational optimization toolkit 19) before refinement 26 utilizing user feedback. Refined model 27 may undergo computational optimization 29-4 (e.g., using computational optimization toolkit 19) before outputting to downstream system 28. Computational optimizations 29-1, ..., 29-4 may all be the same, all be different, or include at least some different optimization techniques.

[0190] Example of a machine learning-based model inference system

[0191] Figure 10This is a block diagram of an inference system used to operate one or more machine learning-processed models 1 to perform inference (e.g., for training, for deployment, etc.). Model host 31 can receive machine learning-processed models 1. Model host 31 can host one or more model instances 31-1, which can be one or more instances of one or more models. Model host 31 can use available computing resources 31-2 associated with model host 31 to host model instances 31-1.

[0192] Model host 31 can perform inference on behalf of one or more clients 32. Client 32 can transmit input request 33 to model host 31. Using input request 33, model host 31 can obtain input 2 to feed into machine learning model 1. Machine learning model 1 can process input 2 to generate output 3. Using output 3, model host 31 can return output payload 34 in response to input request 33 from client 32. Output payload 34 can include or be based on output 3.

[0193] Model host 31 can utilize various other resources and tools to enhance the inference task. For example, model host 31 can communicate with tool interface 35 to facilitate the use of tools by model instance 31-1. Tool interface 35 may include local or remote APIs. Tool interface 35 may include integrated scripts or other software functions. Model host 31 can use online learning interface 36 to facilitate continuous improvement of the machine learning-based model 1. For example, online learning interface 36 can be used in reinforcement learning loops to retrieve user feedback on inference served by model host 31. Model host 31 can access runtime data source 37 to enhance input 2 using additional contextual information. For example, runtime data source 37 may include knowledge graph 37-1 that facilitates structured information retrieval for information associated with input request 33 (e.g., search engine service). Runtime data source 37 may include public or private, external or local database 37-2 that can store information associated with input request 33 for enhancing input 2. The runtime data source 37 may include account data 37-3, which can be retrieved in association with the user account corresponding to the client 32 to customize the behavior of the model host 31 accordingly.

[0194] The model host 31 may be implemented by one or more computing devices or systems. The client 2 may be implemented by one or more computing devices or systems, which may include computing devices or systems shared with the model host 31.

[0195] For example, model host 31 can operate on a server system that provides machine learning services (e.g., via a local area network or wide area network) to client devices operating client 32. The client device can be an end-user device used by an individual. The client device can also be a server system that operates client 32 to provide various functionalities as services to downstream end-user devices.

[0196] In some implementations, model host 31 may operate on the same device or system as client 32. Model host 31 may be a machine learning service that runs on the device to provide machine learning capabilities to one or more applications operating on the client device, which may include the application implementing client 32. Model host 31 and client 32 may be part of the same application. For example, model host 31 may be a subroutine or method implemented as part of the application, and client 32 may be another subroutine or method that uses model host 31 to perform inference functionality within the application. It should be understood that model host 31 and client 32 may have various different configurations.

[0197] Model instance 31-1 may include one or more machine learning models that can be used to perform inference. Model instance 31-1 may include weights or other model components stored in persistent storage, temporary caches, or loaded into memory. Model instance 31-1 may include multiple instances of the same model (e.g., for parallel execution of more requests on the same model). Model instance 31-1 may include instances of different models. Model instance 31-1 may include cached intermediate states of active or inactive models, which are used to accelerate inference for those models. For example, an inference session with a particular model can generate a significant amount of computational results that can be reused for future inference runs (e.g., using a KV cache for a transformer-based model). These computational results can be stored in association with the inference session, allowing for more efficient execution when the session resumes.

[0198] Computing resource 31-2 may include one or more processors (central processing unit, graphics processing unit, tensor processing unit, machine learning accelerator, etc.) connected to one or more memory devices. Computing resource 31-2 may include a dynamic pool of available resources shared with other processes. Computing resource 31-2 may include a memory device large enough to fit an entire model instance into a single memory instance. Computing resource 31-2 may also share model instances across multiple memory devices (e.g., using data parallelization or tensor parallelization). Doing so can increase parallelization or execute large models using multiple memory devices that, individually, may not be able to fit the entire model into memory.

[0199] Input request 33 may include data for input 2. Model host 31 can process input request 33 to obtain input 2. Input 2 can be obtained directly from input request 33 or retrieved using input request 33. Input request 33 can be submitted to model host 31 via API.

[0200] Model host 31 can perform inference on multiple batches of input requests 33 in parallel. For example, model instance 31-1 can be configured with an input structure having batch dimensions. Individual inputs 2 can be distributed across batch dimensions (e.g., rows of an array). Individual inputs 2 can include completely different contexts. Individual inputs 2 can be multiple inference steps for the same task. Individual inputs 2 can be interleaved in the input structure, such that any given inference loop can operate on different parts of the corresponding inputs 2. In this way, for example, model host 31 can perform inference on batches in parallel, such that output 3 can also contain batch dimensions and return the inference results of batches of input 2 in parallel. In this way, for example, multiple batches of input requests 33 can be processed in parallel to achieve higher throughput of output payload 34.

[0201] The output payload 34 may include or be based on the output 3 from the machine learning model 1. The model host 31 may process the output 3 to obtain the output payload 34. This may include linking multiple rounds of inference (e.g., iteratively, recursively, across the same model or different models) to obtain the final output of the task to be returned in the output payload 34. The output payload 34 may be transferred to the client 32 via an API.

[0202] Online learning interface 36 can facilitate reinforcement learning of machine learning-based Model 1. Online learning interface 36 can facilitate reinforcement learning with human feedback (RLHF). Online learning interface 36 can facilitate federated learning of machine learning-based Model 1.

[0203] Model host 31 can execute machine learning-enabled model 1 to perform inference for various tasks using various types of data. For example, various inputs 2 and outputs 3 can be used for various different tasks. In some implementations, input 2 may be or otherwise represent image data. Machine learning-enabled model 1 can process image data to generate outputs. As an example, machine learning-enabled model 1 can process image data to generate image recognition outputs (e.g., image data identification, latent embedding of image data, encoded representation of image data, hashing of image data, etc.). As another example, machine learning-enabled model 1 can process image data to generate image segmentation outputs. As another example, machine learning-enabled model 1 can process image data to generate image classification outputs. As another example, machine learning-enabled model 1 can process image data to generate image data modification outputs (e.g., image data alterations, etc.). As another example, machine learning-enabled model 1 can process image data to generate encoded image data outputs (e.g., encoded and / or compressed representations of image data, etc.). As another example, machine learning-enabled model 1 can process image data to generate expanded image data outputs. As another example, Model 1, which is processed by machine learning, can process image data to generate predictive output.

[0204] In some implementations, the task is a computer vision task. In some cases, the input 2 includes pixel data from one or more images, and the task is an image processing task. For example, the image processing task could be image classification, where the output is a set of scores, each score corresponding to a different object class and representing the probability that one or more images depict an object belonging to that object class. The image processing task could be object detection, where the image processing output identifies one or more regions in one or more images, and for each region, identifies the probability that the region depicts an object of interest. As another example, the image processing task could be image segmentation, where the image processing output defines a corresponding probability for each class in a predetermined set of classes for each pixel in one or more images. For example, the class set could be foreground and background. As another example, the class set could be object classes. As another example, the image processing task could be depth estimation, where the image processing output defines a corresponding depth value for each pixel in one or more images. As another example, the image processing task could be motion estimation, where the network input includes multiple images, and the image processing output defines the motion of the scene depicted at each pixel in one of the input images between the images in the network input.

[0205] In some implementations, input 2 may be or otherwise represent natural language data. The machine learning-enabled model 1 can process the natural language data to generate output. As an example, the machine learning-enabled model 1 can process natural language data to generate a language-encoded output. As another example, the machine learning-enabled model 1 can process natural language data to generate a latent text embedding output. As another example, the machine learning-enabled model 1 can process natural language data to generate a translation output. As another example, the machine learning-enabled model 1 can process natural language data to generate a classification output. As another example, the machine learning-enabled model 1 can process natural language data to generate a text segmentation output. As another example, the machine learning-enabled model 1 can process natural language data to generate a semantic intent output. As another example, the machine learning-enabled model 1 can process natural language data to generate an expanded text or natural language output (e.g., text or natural language data of higher quality than the input text or natural language). As another example, the machine learning-enabled model 1 can process natural language data to generate a predicted output (e.g., one or more predicted subsequent portions of the natural language content).

[0206] In some implementations, input 2 can be or otherwise represent speech data (e.g., data describing spoken natural language, such as audio data, text data, etc.). The machine learning model 1 can process the speech data to generate output. As an example, the machine learning model 1 can process speech data to generate speech recognition output. As another example, the machine learning model 1 can process speech data to generate speech translation output. As another example, the machine learning model 1 can process speech data to generate latent embedding output. As another example, the machine learning model 1 can process speech data to generate encoded speech output (e.g., encoded and / or compressed representations of speech data, etc.). As another example, the machine learning model 1 can process speech data to generate amplified speech output (e.g., speech data of higher quality than the input speech data, etc.). As another example, the machine learning model 1 can process speech data to generate text representation output (e.g., a text representation of the input speech data, etc.). As another example, the machine learning model 1 can process speech data to generate predictive output.

[0207] In some implementations, input 2 can be or otherwise represent latent encoded data (e.g., a latent space representation of the input). The machine learning-processed model 1 can process the latent encoded data to generate an output. As an example, the machine learning-processed model 1 can process the latent encoded data to generate an identification output. As another example, the machine learning-processed model 1 can process the latent encoded data to generate a reconstruction output. As another example, the machine learning-processed model 1 can process the latent encoded data to generate a search output. As another example, the machine learning-processed model 1 can process the latent encoded data to generate a re-clustering output. As yet another example, the machine learning-processed model 1 can process the latent encoded data to generate a prediction output.

[0208] In some implementations, input 2 may be or otherwise represent statistical data. Statistical data may be, represent, or otherwise include data calculated and / or computed from another data source. The machine learning model 1 can process the statistical data to generate output. As an example, the machine learning model 1 can process the statistical data to generate an identification output. As another example, the machine learning model 1 can process the statistical data to generate a prediction output. As another example, the machine learning model 1 can process the statistical data to generate a classification output. As another example, the machine learning model 1 can process the statistical data to generate a segmentation output. As another example, the machine learning model 1 can process the statistical data to generate a visualization output. As another example, the machine learning model 1 can process the statistical data to generate a diagnostic output.

[0209] In some implementations, input 2 can be or otherwise represent sensor data. The machine learning-based model 1 can process the sensor data to generate output. As an example, the machine learning-based model 1 can process sensor data to generate identification output. As another example, the machine learning-based model 1 can process sensor data to generate prediction output. As another example, the machine learning-based model 1 can process sensor data to generate classification output. As another example, the machine learning-based model 1 can process sensor data to generate segmentation output. As another example, the machine learning-based model 1 can process sensor data to generate visualization output. As another example, the machine learning-based model 1 can process sensor data to generate diagnostic output. As another example, the machine learning-based model 1 can process sensor data to generate detection output.

[0210] In some implementations, the machine learning-based Model 1 can be configured to perform a task that includes encoding input data to achieve reliable and / or efficient transmission or storage (and / or corresponding decoding). For example, the task could be an audio compression task. The input could include audio data, and the output could include compressed audio data. In another example, the input includes visual data (e.g., one or more images or videos), the output includes compressed visual data, and the task is a visual data compression task. In another example, the task could include generating embeddings for input data (e.g., input audio or visual data). In some cases, the input includes audio data representing spoken utterances, and the task is a speech recognition task. The output could include text output mapped to spoken utterances. In some cases, the task includes encrypting or decrypting the input data. In some cases, the task includes microprocessor performance tasks such as branch prediction or memory address translation.

[0211] In some implementations, the task is a generative task, and the machine learning-based model 1 can be configured to output content generated from input 2. For example, input 2 can be, or otherwise represent, data of one or more modalities that encode the context used to generate additional content.

[0212] In some implementations, the task can be a text completion task. A machine learning-based model 1 can be configured to process input 2, representing text data, and generate output 3, which represents additional text data to complete the text sequence of input 2. For example, the machine learning-based model 1 can be configured to generate output 3 to complete a sentence, paragraph, or section of text following a portion of the text represented by input 2.

[0213] In some implementations, the task can be an instruction-following task. A machine learning model 1 can be configured to process input 2 representing instructions for performing a function and generate output 3 that advances towards a goal satisfying the instruction function (e.g., at least one step of a multi-step process for performing the function). Output 3 can represent data of the same or different modality as input 2. For example, input 2 can represent text data (e.g., natural language instructions for a task to be performed), and machine learning model 1 can process input 2 to generate output 3 representing text data in response to the instructions (e.g., a natural language response, a programming language response, a machine language response, etc.). Input 2 can represent image data (e.g., image-based instructions for a task to be performed, optionally accompanied by text instructions), and machine learning model 1 can process input 2 to generate output 3 representing text data in response to the instructions (e.g., a natural language response, a programming language response, a machine language response, etc.). One or more outputs 3 can be generated iteratively or recursively to process and complete the steps leading to the completion of the requested function in a sequential manner. For example, the initial output can be executed by an external system or processed by a machine learning model 1 to complete the initial steps of the function. Multiple steps can be performed, in which the final output is obtained in response to the initial instructions.

[0214] In some implementations, the task can be a question-answering task. A machine learning model 1 can be configured to process input 2 representing a question to be answered and generate output 3 that advances towards the goal of returning an answer to the question (e.g., at least one step in a multi-step process for performing the function). Output 3 can represent data of the same or different modality as input 2. For example, input 2 can represent text data (e.g., natural language instructions for a task to be performed), and machine learning model 1 can process input 2 to generate output 3 representing text data in response to the question (e.g., a natural language response, a programming language response, a machine language response, etc.). Input 2 can represent image data (e.g., image-based instructions for a task to be performed, optionally accompanied by text instructions), and machine learning model 1 can process input 2 to generate output 3 representing text data in response to the question (e.g., a natural language response, a programming language response, a machine language response, etc.). One or more outputs 3 can be generated iteratively or recursively to process and complete the steps toward answering the question sequentially. For example, the initial output can be executed by an external system or processed by a machine learning model 1 to complete the initial steps to obtain an answer to the question (e.g., querying a database, performing calculations, executing scripts, etc.). Multiple steps can be performed to obtain the final output in response to the question.

[0215] In some implementations, the task can be an image generation task. A machine learning-based model 1 can be configured to process input 2, which represents context regarding a desired portion of the image content. Context can include text data, image data, audio data, etc. The machine learning-based model 1 can be configured to generate output 3, which represents image data depicting the image in context. For example, the machine learning-based model 1 can be configured to generate pixel data of an image. The values ​​of the channels associated with pixels in the pixel data can be selected based on context (e.g., based on probabilities determined according to the context).

[0216] In some implementations, the task can be an audio generation task. A machine learning model 1 can be configured to process input 2, which represents context regarding a desired portion of the audio content. Context can include text data, image data, audio data, etc. The machine learning model 1 can be configured to generate output 3, which represents context-dependent audio data. For example, the machine learning model 1 can be configured to generate waveform data in the form of an image (e.g., a spectrogram). The values ​​of channels associated with pixels in the image can be selected based on the context. The machine learning model 1 can also be configured to generate waveform data in the form of a sequence of discrete samples of a continuous waveform. The values ​​of the sequence can be selected based on the context (e.g., based on probabilities determined according to the context).

[0217] In some implementations, the task can be a data generation task. The machine learning-based model 1 can be configured to process input 2, which represents context regarding a desired portion of the data (e.g., data from various data domains, such as sensor data, image data, multimodal data, statistical data, etc.). For example, the desired data can be synthetic data used to train other machine learning models. The context can include any data type. The machine learning-based model 1 can be configured to generate output 3, which represents data aligned with the desired data. For example, the machine learning-based model 1 can be configured to generate data values ​​to populate a dataset. The values ​​of data objects can be selected based on context (e.g., based on probabilities determined according to the context).

[0218] Example computing systems and devices

[0219] Figure 11This is a block diagram of an example networked computing system capable of implementing aspects of the exemplary implementations of this disclosure. The system may include multiple computing devices and systems communicatively coupled via network 49. Example computing device 50 is described as an example of a computing device capable of implementing any aspect of this disclosure (e.g., implementing model host 31, client 32, or both). Example server computing system 60 is described as an example of a server computing system capable of implementing any aspect of this disclosure (e.g., implementing model host 31, client 32, or both). Computing device 50 and server computing system 60 may interact collaboratively (e.g., via network 49) to implement any aspect of this disclosure (e.g., implementing model host 31, client 32, or both). Model development platform system 70 is an example system that can host or provide a model development platform 12 for developing machine learning models. Third-party system 80 is an example system that any of computing device 50, server computing system 60, or model development platform system 70 can interact with in performing various aspects of this disclosure (e.g., using third-party tools, accessing third-party databases, or other resources, etc.).

[0220] Network 49 can be any type of communication network, such as a local area network (e.g., intranet), a wide area network (e.g., the Internet), or some combination thereof, and can include any number of wired or wireless links. Generally, communication via network 49 can use a wide variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), or protection schemes (e.g., VPN, Secure HTTP, SSL) via any type of wired or wireless connection. Network 49 can also be implemented via a system bus. For example, Figure 11 One or more devices or systems may be located in the same place as, contained in, or otherwise integrated into one or more other devices or systems.

[0221] Computing device 50 can be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop computer), a mobile computing device (e.g., a smartphone or tablet computer), a game console or controller, a wearable computing device, an embedded computing device, a server computing device, a virtual machine operating on a host device, or any other type of computing device. Computing device 50 can be a client computing device. Computing device 50 can be an end-user computing device. Computing device 50 can be a computing device that provides services to an end user (who may use another computing device to interact with computing device 50).

[0222] Computing device 50 may include one or more processors 51 and memory 52. ​​Processor 51 may be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and may be a single processor or multiple processors operatively connected. Memory 52 may include one or more non-transitory computer-readable storage media, such as HBM, RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. Memory 52 may store data 53 and instructions 54, which may be executed by processor 51 to cause computing device 50 to perform operations. These operations may implement any or more features described herein. These operations may implement the exemplary methods and techniques described herein.

[0223] The computing device 50 may also include one or more input components for receiving user input. For example, the user input component may be a touch-sensitive component (e.g., a touch-sensitive display or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or stylus). The touch-sensitive component can be used to implement a virtual keyboard. Other example user input components include a microphone, camera, LiDAR, physical keyboard or other buttons, or other components through which the user can provide input.

[0224] The computing device 50 may store or include one or more machine learning models 55. The machine learning models 55 may include one or more machine learning models 1, such as sequence processing model 4. The machine learning models 55 may include one or more model instances 31-1. The machine learning models 55 may be received from a server computing system 60, a model development platform system 70, a third-party system 80 (e.g., an application distribution platform), or developed locally on the computing device 50. The machine learning models 55 may be loaded into memory 52 and used by the processor 51 or otherwise implemented. The computing device 50 may implement multiple parallel instances of the machine learning models 55.

[0225] Server computing system 60 may include one or more processors 61 and memory 62. Processor 61 may be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and may be a single processor or multiple processors operatively connected. Memory 62 may include one or more non-transitory computer-readable storage media, such as HBM, RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. Memory 62 may store data 63 and instructions 64, which can be executed by processor 61 to cause server computing system 60 to perform operations. These operations may implement any or more features described herein. These operations may implement the exemplary methods and techniques described herein.

[0226] In some implementations, server computing system 60 includes one or more server computing devices or is otherwise implemented by one or more server computing devices. In instances where server computing system 60 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0227] Server computing system 60 may store or otherwise include one or more machine learning models 65. Machine learning models 65 may be the same as or different from machine learning models 55. Machine learning models 65 may include one or more machine learning models 1, such as sequence processing model 4. Machine learning models 65 may include one or more model instances 31-1. Machine learning models 65 may be received from computing device 50, model development platform system 70, third-party system 80, or developed locally on server computing system 60. Machine learning models 65 may be loaded into memory 62 and used by processor 61 or otherwise implemented. Server computing system 60 may implement multiple parallel instances of machine learning models 65.

[0228] In the example configuration, the machine learning model 65 may be included in or otherwise stored and implemented by the server computing system 60 to establish a client-server relationship with the computing device 50 for service model inference. For example, the server computing system 60 may implement the model host 31 on behalf of client 32 on the computing device 50. For example, the machine learning model 65 may be implemented by the server computing system 60 as part of a web service (e.g., a remote machine learning model hosting service, such as an online interface for performing machine learning model operations on the server computing system 60 over a network). For example, the server computing system 60 may communicate with the computing device 50 via a local intranet or internet connection. For example, the computing device 50 may be a workstation or endpoint communicating with the server computing system 60, where the implementation of the machine learning model 65 is managed by the server computing system 60 to remotely perform inference (e.g., for runtime or training operations), and where output is returned (e.g., projected, streamed, etc.) to the computing device 50. The machine learning model 65 can work collaboratively or interoperably with the machine learning model 55 on the computing device 50 to perform various tasks.

[0229] The model development platform system 70 may include one or more processors 71 and memory 72. Processor 71 may be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and may be a single processor or multiple processors operatively connected. Memory 72 may include one or more non-transitory computer-readable storage media, such as HBM, RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. Memory 72 may store data 73 and instructions 74, which can be executed by processor 71 to cause the model development platform system 70 to perform operations. These operations may implement any one or more features described herein. These operations may implement the example methods and techniques described herein. Example operations include the functionality described herein with respect to model development platform 12. This functionality and other functionalities may be implemented by developer tools 75.

[0230] Third-party system 80 may include one or more processors 81 and memory 82. Processor 81 may be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and may be a single processor or multiple processors operatively connected. Memory 82 may include one or more non-transitory computer-readable storage media, such as HBM, RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. Memory 82 may store data 83 and instructions 84, which may be executed by processor 81 to cause third-party system 80 to perform operations. These operations may implement any one or more features described herein. These operations may implement the example methods and techniques described herein. Example operations include the functionality described herein with respect to tools and other external resources (e.g., third-party resource 85) invoked when training machine learning models 1, 4, 16, 20, 55, 65, etc., or performing inference using machine learning models.

[0231] Figure 11 An example arrangement of a computing system that can be used to implement the present disclosure is shown. Other computing system configurations may also be used. For example, in some implementations, one or both of computing system 50 or server computing system 60 may implement all or part of the operation of model development platform system 70. For example, computing system 50 or server computing system 60 may implement developer tool 75 (or extensions thereof) to develop, update / train, or refine machine learning models 1, 4, 16, 20, 55, 65, etc., using one or more techniques described herein with respect to model alignment toolkit 17. In this way, for example, computing system 50 or server computing system 60 may develop, update / train, or refine machine learning models based on local datasets (e.g., for model personalization / customization, as permitted by user data preference selection).

[0232] Figure 12 This is a block diagram of an example computing device 98 implemented according to an example embodiment of the present disclosure. The computing device 98 may be a user computing device or a server computing device (e.g., computing device 50, server computing system 60, etc.). The computing device 98 may implement model host 31. For example, the computing device 98 may include multiple applications (e.g., applications 1 to N). Each application may contain its own machine learning library and machine learning-enabled models. For example, each application may include machine learning-enabled models. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc. Figure 12As shown, each application can communicate with multiple other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, or additional components. In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is application-specific.

[0233] Figure 13 This is a block diagram of an example computing device 99 implemented according to an example embodiment of the present disclosure. Computing device 99 may be the same as or different from computing device 98. Computing device 99 may be a user computing device or a server computing device (e.g., computing device 50, server computing system 60, etc.). Computing device 98 may implement model host 31. For example, computing device 99 may include multiple applications (e.g., applications 1 to N). Each application may communicate with a central intelligence layer. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc. In some implementations, each application may use an API (e.g., a common API across all applications) to communicate with the central intelligence layer (and the model stored therein).

[0234] The central intelligence layer can include multiple machine learning-engineered models. For example, such as Figure 13 As shown, a corresponding machine learning model can be provided for each application, and this corresponding machine learning model is managed by a central intelligence layer. In other implementations, two or more applications can share a single machine learning model. For example, in some implementations, the central intelligence layer can provide a single model for all applications. In some implementations, the central intelligence layer is included within or otherwise implemented by the operating system of the computing device 99.

[0235] The central intelligence layer can communicate with the central device data layer. The central device data layer can be a centralized data storage repository for computing device 99. For example... Figure 13 As shown, the central device data layer can communicate with multiple other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0236] Additional Publication

[0237] This paper discusses technologies related to servers, databases, software applications, and other computer-based systems, as well as the actions taken and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a wide range of possible configurations, combinations, and partitions of tasks and functionality among and within components. For example, the processes discussed herein can be implemented using a single device or component, or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0238] While the subject matter has been described in detail with respect to various specific example embodiments, each example is provided by way of illustration and not limitation. Modifications, alterations, and equivalents to such embodiments will be readily apparent to those skilled in the art upon understanding the foregoing. Therefore, this disclosure does not exclude such modifications, alterations, or additions to the subject matter that will be readily understood by those of ordinary skill in the art. For example, features shown or described as part of one embodiment may be used with another embodiment to produce yet another embodiment. Therefore, it is intended that this disclosure cover such modifications, alterations, and equivalents.

[0239] The aspects of this disclosure have been described with reference to their illustrative embodiments. Any and all features of the appended claims can be combined or rearranged in any possible manner, including combinations of claims not expressly listed together, because the illustrative claims dependencies listed herein should not be construed as limiting the scope of possible combinations of the features disclosed herein. Therefore, the scope of this disclosure is illustrative rather than limiting, and this disclosure does not exclude such modifications, alterations, or additions to the subject matter that will be readily understood by one of ordinary skill in the art. Furthermore, terms are described herein using lists of illustrative elements connected by conjunctions such as “and,” “or,” “but,” etc. It should be understood that such conjunctions are provided for illustrative purposes only. For example, a sequence of terms and other items connected by a specific conjunction such as “or” may refer to “and / or,” “at least one of,” “any combination of,” etc., of the illustrative elements listed therein. Terms such as “based on” should be understood as “at least partially based on.”

[0240] The term "capable" should be understood as referring to the possibility of a feature in various implementations, rather than specifying a capability that must exist in every implementation. For example, the phrase "X is capable of Y" should be understood as indicating that in various implementations, X may be configured to perform Y, rather than indicating that X must always be capable of performing Y in every instance. It should be understood that in various implementations, X may not be able to perform Y and is still within the scope of this disclosure.

[0241] The term "may" should be understood as referring to the possibility of a feature in various implementations, rather than specifying a capability that must exist in every implementation. For example, the phrase "X may perform Y" should be understood as indicating that in various implementations, X may be configured to perform Y, rather than indicating that X must always be able to perform Y in every instance. It should be understood that in various implementations, X may not be able to perform Y and is still within the scope of this disclosure.

Claims

1. A computer-implemented method for semantic clustering of sequence data to achieve an arbitrarily long context window for a machine learning-based sequence processing model, the method comprising: The context sequence is obtained from one or more computing devices; The one or more computing devices determine multiple subsequences of the context sequence; The one or more computing devices use a first machine learning-based sequence processing model to determine multiple corresponding semantic embeddings that are associated with the multiple subsequences of the context sequence, respectively; The one or more computing devices determine a plurality of semantic clusters, wherein each of the plurality of semantic clusters includes one or more semantic embeddings among the plurality of semantic embeddings; as well as The output sequence is generated by the one or more computing devices using at least one of the first machine learning-based sequence processing model and the second machine learning-based sequence processing model, based at least in part on the plurality of semantic clusters.

2. The method according to any of the preceding claims, comprising: The one or more computing devices select one or more subsequences of the plurality of subsequences of the context sequence, at least in part based on the plurality of semantic clusters; as well as The one or more computing devices add data associated with the one or more sub-sequences to the context window of the second machine learning-based sequence processing model; The output sequence is generated at least in part based on the context window.

3. The method of any of the preceding claims, wherein the data associated with the one or more subsequences includes at least one of the one or more subsequences.

4. The method of any of the preceding claims, wherein the data associated with the one or more subsequences comprises sequence data generated by the one or more computing devices based at least in part on a semantic embedding associated with at least one of the one or more subsequences.

5. The method of any of the preceding claims, wherein the data associated with the one or more subsequences includes one or more corresponding semantic embeddings of the plurality of corresponding semantic embeddings.

6. The method of any preceding claim, wherein the data associated with one or more subsequences comprises sequence data generated by the one or more computing devices based at least in part on a geometric transformation of a semantic embedding associated with at least one of the one or more subsequences.

7. The method of any of the preceding claims, wherein selecting the one or more subsequences comprises selecting based on one or more priorities.

8. The method of any of the preceding claims, wherein selecting based on the one or more priorities includes giving preference to clusters associated with a larger number of semantic embeddings.

9. The method of any of the preceding claims, wherein selecting based on the one or more priorities includes giving preference to semantic embeddings near the center of the semantic cluster.

10. The method of any of the preceding claims, wherein the one or more priorities include priorities determined based on user input.

11. The method of any of the preceding claims, wherein selecting based on the one or more priorities includes making selecting at least one subsequence from each of the plurality of semantic clusters a priority over selecting a second subsequence from any of the plurality of semantic clusters.

12. The method as claimed in any of the preceding claims, wherein: The first machine learning model is characterized by a first number of parameters; The second machine learning model is characterized by a second number of parameters; and The second quantity is at least 30 times the first quantity.

13. The method of any of the preceding claims, wherein determining the plurality of semantic clusters includes hierarchical clustering.

14. The method as claimed in any claim, wherein at least one of the plurality of semantic clusters comprises a plurality of subclusters; and selection based on the one or more priorities includes giving preference to semantic embeddings near the centroid of the subclusters among the plurality of subclusters.

15. The method of any of the preceding claims, wherein determining the plurality of subsequences of the context sequence comprises splitting the context at least in part based on a subsequence length threshold.

16. The method of any of the preceding claims, wherein determining the plurality of subsequences of the context sequence comprises splitting the context at least in part based on one or more delimiters or sequence boundaries.

17. The method of any of the preceding claims, wherein determining the plurality of subsequences of the context sequence comprises using a machine learning-based model.

18. The method of any of the preceding claims, further comprising adding a prompt to the context window of the second machine learning-processed sequence model, wherein the prompt includes one or more instructions or explanations.

19. A computing system comprising one or more processors and one or more non-transitory computer-readable media, the one or more non-transitory computer-readable media storing instructions executable by the one or more processors to cause the computing system to perform operations, the operations including: Obtain the context sequence; Determine multiple subsequences of the context sequence; A first machine learning-based sequence processing model is used to determine multiple corresponding semantic embeddings that are associated with the multiple subsequences of the context sequence, respectively. A plurality of semantic clusters are defined, wherein each of the plurality of semantic clusters includes one or more semantic embeddings from the plurality of semantic embeddings; as well as The output sequence is generated using at least one of the first machine learning-based sequence processing model and the second machine learning-based sequence processing model, based at least in part on the plurality of semantic clusters.

20. One or more non-transitory computer-readable media storing instructions executable by a computing system to perform operations, the operations including: Obtain the context sequence; Determine multiple subsequences of the context sequence; A first machine learning-based sequence processing model is used to determine multiple corresponding semantic embeddings that are associated with the multiple subsequences of the context sequence, respectively. A plurality of semantic clusters are defined, wherein each of the plurality of semantic clusters includes one or more semantic embeddings from the plurality of semantic embeddings; as well as The output sequence is generated using at least one of the first machine learning-based sequence processing model and the second machine learning-based sequence processing model, based at least in part on the plurality of semantic clusters.

Images