Automata-based constraints for model decoding

Abstract

Provided are systems and methods for enhancing the output accuracy and consistency of sequence processing models, particularly when generating text that must conform to a specific formal language. Systems and methods can integrate finite-state or push-down automata with the decoding process to ensure that the generated sequences adhere to the desired syntax or grammar rules. A system can generate or otherwise leverage a finite-state or push-down automaton that encodes constraints associated with a particular formal language or syntax. At each decoding iteration of a sequence processing model, the decoding system can use the finite-state or push-down automaton to evaluate the token validity for each of a number of possible output tokens. The output of the sequence processing model can then be limited to or otherwise guided towards tokens that are indicated as valid by the finite-state or push-down automaton.

Description

AUTOMATA-BASED CONSTRAINTS FOR MODEL DECODING PRIORITY CLAIM

[0001] The present application is based on and claims priority to United States Provisional Application Number 63 / 575,135 having a filing date of April 5, 2024. Application claims priority to and the benefit of each of such applications and incorporates all such applications herein by reference in their entirety. FIELD

[0002] The present disclosure relates generally to machine learning processes and machine-learned devices and systems. More particularly, the present disclosure relates to automate-based constraints for syntax-constrained decoding in sequence processing models such as large language models. BACKGROUND

[0003] In the field of natural language processing (NLP), sequence processing models, such as language models (LMs) (e.g., so-called “large language models”), have become increasingly capable of generating complex text sequences that mimic human-like language patterns. These models are trained on vast corpora of text data and have numerous applications, including automated code generation, data formatting, and the creation of application programming interface (API) calls. However, a significant technical problem arises when these models are expected to generate outputs that adhere to strict formal languages or grammars, which is a requirement in many technical domains.

[0004] Formal languages, characterized by precise syntax and structural rules, are fundamental in computer science applications, such as programming languages, structured data formats (e.g., JSON, XML), and command-line interfaces. The challenge is that while sequence processing models can be prompted or fine-tuned to improve their adherence to such formal languages, there is no inherent mechanism within the models to guarantee conformance to the syntax or grammar rules. This limitation presents a technical problem, as non-conforming outputs can lead to parsing errors, execution failures, or other downstream processing issues that directly affect the functionality and reliability of computer systems.

[0005] Furthermore, the tokenization process, which is an integral part of how sequence processing models interpret and generate text, introduces additional complications.Tokenization involves breaking down input text into smaller units, or tokens, which the model uses for processing. The tokenization schemes, often based on subword units, are typically not aligned with the boundaries of formal language tokens. This misalignment can result in fragmented or merged tokens that do not correspond to valid lexical units in the target formal language. For example, a language model may tokenize a function call in a programming language in a way that disrupts the syntactic structure, thereby creating invalid code.

[0006] Another technical problem is the resource-intensive nature of tuning or training sequence processing models to produce valid formal language outputs, especially for less common or highly specialized languages and formats. The tuning process requires significant computational resources and domain expertise, making it impractical for many applications. SUMMARY

[0007] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or can be learned from the description, or can be learned through practice of the embodiments.

[0008] One example aspect of the present disclosure is directed to a computer- implemented method for improved decoding from sequence processing models, the method comprising: for each of one or more decoding iterations: processing, by the computing system, a current sequence of tokens with a machine-learned sequence processing model to generate a plurality of scores respectively for a plurality of tokens included in a token vocabulary associated with the machine-learned sequence processing model, wherein the current sequence of tokens comprises tokens from the plurality of tokens; evaluating, by the computing system, a finite-state or push-down automaton based on a current sequence of tokens to determine a respective validity status for each of the plurality of tokens, wherein the finite-state or push-down automaton comprises a plurality of states and a plurality of edges that define token-based-transitions between the plurality of states; selecting, by the computing system, one of the plurality of tokens as an output token for the current decoding iteration based at least in part on the plurality of scores and the respective validity status for each of the plurality of tokens.

[0009] Another example aspect of the present disclosure is directed to a computer system comprising: one or more processors; and one or more non-transitory computer- readable media that collectively store: a machine-learned sequence processing modelconfigured to process a current sequence of tokens to generate a plurality of scores respectively for a plurality of tokens included in a token vocabulary associated with the machine-learned sequence processing model, wherein the current sequence of tokens comprises tokens from the plurality of tokens; a finite-state or push-down automaton configured to evaluate a respective validity status for each of the plurality of tokens based on the current sequence of tokens, wherein the finite-state or push-down automaton comprises a plurality of states and a plurality of edges that define token-based-transitions between the plurality of states; and computer-executable decoding logic configured to select one of the plurality of tokens as an output token for the current decoding iteration based at least in part on the plurality of scores and the respective validity status for each of the plurality of tokens.

[0010] Other aspects of the present disclosure are directed to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices.

[0011] These and other features, aspects, and advantages of various embodiments of the present disclosure will become better understood with reference to the following description and appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the present disclosure and, together with the description, serve to explain the related principles. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 illustrates an example of three distinct finite-state automata (FSAs) representing various regular expression patterns according to example embodiments of the present disclosure.

[0013] Figure 2 illustrates an example of a non-deterministic finite-state automaton (FSA) derived from a regular expression and its subsequent determinization according to example embodiments of the present disclosure.

[0014] Figure 3 illustrates an example of finite-state transducers (FSTs) that convert specific input strings into predetermined output strings according to example embodiments of the present disclosure.

[0015] Figure 4 illustrates an example of a token vocabulary and its associated detokenizing finite-state transducer (FST) according to example embodiments of the present disclosure.

[0016] Figure 5 illustrates an example of a character-based finite-state automaton (FSA) reflecting a regular expression and its composition with a detokenizing FST according to example embodiments of the present disclosure.

[0017] Figure 6 illustrates an example of the composition process between a vocab finite-state transducer (FST) and a regex finite-state automaton (FSA) to form a combined automaton according to example embodiments of the present disclosure.

[0018] Figure 7 illustrates an example of the decoding flow within a token processing and validation system according to example embodiments of the present disclosure.

[0019] Figures 8, 9, 10, 11, 12, and 13A-H illustrate diagrams of example extensions according to example embodiments of the present disclosure.

[0020] Figure 14 illustrates an example of a grammar for the canonical context-free language and its corresponding push-down automaton (PDA) according to example embodiments of the present disclosure.

[0021] Figure 15 illustrates an example of a simple token vocabulary, a detokenizing finite-state transducer (FST) constructed from it, and the FST’s composition with a PDA according to example embodiments of the present disclosure.

[0022] Figure 16 is a flow chart diagram illustrating an example method for training a machine-learned model according to example implementations of aspects of the present disclosure;

[0023] Figure 17 is a block diagram of an example processing flow for using machine-learned model(s) to process input(s) to generate output(s) according to example implementations of aspects of the present disclosure;

[0024] Figure 18 is a block diagram of an example sequence processing model according to example implementations of aspects of the present disclosure;

[0025] Figure 19 is a block diagram of an example technique for populating an example input sequence for processing by a sequence processing model according to example implementations of aspects of the present disclosure;

[0026] Figure 20 is a block diagram of an example model development platform according to example implementations of aspects of the present disclosure;

[0027] Figure 21 is a block diagram of an example training workflow for training a machine-learned model according to example implementations of aspects of the present disclosure;

[0028] Figure 22 is a block diagram of an inference system for operating one or more machine-learned model(s) to perform inference according to example implementations of aspects of the present disclosure;

[0029] Figure 23 is a block diagram of an example networked computing system according to example implementations of aspects of the present disclosure;

[0030] Figure 24 is a block diagram of an example computing device according to example implementations of aspects of the present disclosure; and

[0031] Figure 25 is a block diagram of an example computing device according to example implementations of aspects of the present disclosure. DETAILED DESCRIPTION

[0032] Example aspects of the present disclosure are directed to systems and methods for enhancing the output accuracy and consistency of sequence processing models, particularly when generating text that must conform to a specific formal language. Specifically, the proposed systems and methods can integrate finite-state or push-down automata with the decoding process to ensure that the generated sequences adhere to the desired syntax or grammar rules. In particular, example implementations of the present disclosure can generate or otherwise leverage a finite-state or push-down automaton that encodes constraints associated with a particular formal language or syntax. At each decoding iteration of a sequence processing model, the decoding system can use the finite-state or push-down automaton to evaluate the token validity for each of a number of possible output tokens. The output of the sequence processing model can then be limited to or otherwise guided towards tokens that are indicated as valid by the finite-state or push-down automaton, thereby ensuring that the output of the model complies with the constraints of the formal language. This technology can be applied to various sequence processing models, such as large language models used for generating code snippets, structured data, or API calls.

[0033] More particularly, each decoding iteration of a sequence processing model can include generating, by the sequence processing model, scores for a number of potential output tokens. For instance, in a language model generating programming code, the scores could be associated with different syntactical elements such as variables, functions, or control structures. The scores generally reflect the likelihood of each token being the correct next element in the sequence. For example, the scores may be logit scores that correspond to the likelihood of each token being the correct next element in the sequence.

[0034] According to an aspect of the present disclosure, a finite-state automaton (FSA) can be added as a core component of the decoding process. An FSA is a computational model comprising a finite number of states and transitions between those states based on input symbols. In this context, the FSA can be used to define the permissible sequences oftokens that conform to a particular syntax, such as the structure of a JSON object or the grammar of a SQL query.

[0035] The proposed FSA is particularly advantageous for its simplicity and efficiency. Being generally equivalent to or otherwise encoding regular expressions, FSAs are a powerful tool for defining constraints on text. They are well-suited for enforcing syntax rules that do not require memory of previous states beyond the current token, such as those found in many structured data formats and simpler programming languages.

[0036] Some example implementations of the present disclosure can alternatively leverage a push-down automata (PDA). PDAs are similar to FSAs but are equipped with a stack, allowing them to handle context-free languages, which can represent more intricate patterns such as nested parentheses or recursive grammatical rules, making them suitable for parsing programming languages or nested data formats. For example, PDAs can be used for more complex language structures that go beyond the capabilities of regular language.

[0037] Thus, for scenarios requiring a more robust solution, the present disclosure’s push-down automaton can be used as an alternative to the finite-state automaton. PDAs can handle the additional complexity of context-free languages, which include most programming languages and other structured text formats that involve nested or recursive patterns.

[0038] According to an aspect of the present disclosure, the automata (e.g., FSA or PDA) used in the present disclosure can be specifically designed to work at the token level. This means that they are constructed to understand and process the tokens used by the sequence processing model, which may represent words, subwords, or other textual units, depending on the model’s tokenization scheme. The use of a token-level automaton has numerous benefits as it pairs much more elegantly with the output space of the sequence processing model.

[0039] In particular, in some implementations, a token-level automaton can be constructed by the composition of a token-level finite state transducer (FSTs) with a character-level automaton, for example whether they be FSAs or PDAs. FSTs are automata that can output a different string than the one they receive as input, effectively transforming sequences. By composing an FST that represents the model’s tokenization with a character- level automaton representing the desired constraints, the resulting automaton can directly process the tokens output by the model.

[0040] In some implementations, the constraints that the automata enforce can be based on regular expressions (e.g., in the case of FSAs) and / or deterministic grammars (e.g., in the case of PDAs). Regular expressions are patterns that describe sets of strings, anddeterministic grammars are rules that define the structure of syntactically correct sentences. These constraints could be as simple as a regular expression (also referred to as a “regex”) for email validation or as complex as a grammar for a programming language.

[0041] Another aspect of the present disclosure is directed to techniques for selecting tokens based on their validity as determined by the automaton. For example, after processing the current sequence of tokens through the model, each token’s validity can be evaluated by the automaton. Tokens that lead to valid continuations in the automaton are considered for selection, while those that do not are penalized or masked, thereby guiding the model to generate syntactically correct sequences.

[0042] According to another aspect, one of the challenges addressed by some implementations present disclosure is the handling of wildcards in regular expressions when composing them with the token vocabulary. Wildcards, such as the dot character in regex, can match almost any character and thus can lead to an explosion in the size of the resulting automaton. Some example implementations of the present disclosure can mitigate this by introducing pre-computed “terminal” masks that efficiently represent sets of valid tokens.

[0043] The present disclosure includes several examples of terminal labels for wildcard matching, such as QUOTED_TEXT for matching quoted strings with escape sequences, UNQUOTED_TEXT for matching YAML-style non-quoted strings, and IMAGE for matching images in multi-modal language models. These terminals facilitate the construction of constraints that can handle common patterns in text generation tasks.

[0044] According to another aspect, to enhance the expressiveness and efficiency of the constraints, some example implementations of the present disclosure can include various syntactic extensions. These extensions, such as SUBSTRING_OF, DELIMITED_LIST, and DELIMITED_SUBSEQUENCE_OF, provide shorthand methods for matching complex patterns that would otherwise require verbose and computationally expensive regular expressions.

[0045] According to another example aspect, some example implementations of the present disclosure can implement a novel table-based representation of the finite-state automaton (FSA) that is specifically designed to be evaluated within a computation graph (e.g., using JAX operations). This approach can enable the direct execution of syntactic constraints on hardware accelerators (e.g., Tensor Processing Units (TPUs)) by converting the operations into XLA (Accelerated Linear Algebra) instructions. By doing so, the proposed techniques overcome the limitations of prior methods that incur significantcommunication overhead between the central processing unit (CPU) and TPU, a bottleneck that can considerably slow down the decoding process.

[0046] In conclusion, the present disclosure offers a framework for ensuring that the output of sequence processing models conforms to specified syntactic constraints. By leveraging automata theory and integrating it with token-level processing, the technology provides a powerful tool for a wide range of applications where formal language output is required from machine learning models.

[0047] The systems and methods of the present disclosure provide a number of technical effects and benefits. As one example, the present disclosure offers a technical solution to a recognized technical problem in the field of natural language processing: the generation of syntactically incorrect sequences by sequence processing models. By implementing finite-state or push-down automata to guide the decoding process, the proposed techniques ensure that the generated text sequences strictly adhere to the rules of formal languages. This is not merely a data processing convenience but a technical requirement for the correct operation of software systems, such as compilers, interpreters, and data parsers, which demand syntactically precise inputs to function properly. The technical character of the solution is evident in its ability to transform the inherently probabilistic outputs of language models into deterministic, rule-conforming sequences, thereby addressing a critical technical issue in automated text generation.

[0048] As another example, a technical problem inherent in the application of sequence processing models is the misalignment of tokenization with the formal syntax of target languages, a consequence of the models’ reliance on subword units. The proposed techniques provide a technical solution to this problem by employing automata theory to map token sequences onto the formal structure of the desired language. This technical measure effectively rectifies tokenization misalignments, ensuring that the tokens produced by the model correspond to valid lexical units within the formal language framework. The benefit of this technical solution is twofold: it enhances the functional accuracy of text generation and reduces the incidence of parsing errors in subsequent processing stages, thereby improving the overall reliability and technical robustness of systems that utilize the generated text.

[0049] As yet another example, the present disclosure introduces a technical solution to the computationally demanding task of applying syntax constraints to language models, particularly in the context of retraining or fine-tuning models for specialized or less common formal languages. By leveraging pre-computed automata that encode the syntax rules as constraints, the proposed techniques enable the application of these constraints in a resource-efficient manner. This technical solution enables the deployment of language models in environments with limited computational resources, such as embedded systems or mobile devices, where the overhead of model retraining is technically and / or economically infeasible.

[0050] As yet another example, to address the technical challenge of managing the vast vocabulary size associated with sequence processing models, some example implementations of the present disclosure can employ a specialized state table structure that eschews the traditional dense matrix format, which becomes impractical due to the prohibitive size of the vocabulary. Instead, the state table can be constructed to leverage the inherent sparsity of the constraint FSTs prior to the expansion of terminal arcs into token arcs. By maintaining this sparsity, the proposed techniques circumvent the impracticalities of both dense and general-purpose sparse array representations. The resulting “bursty” nature of the FSA—characterized by states with highly variable numbers of outbound arcs—is adeptly handled by the disclosed method. This can be achieved by deferring the expansion of terminal arcs until required during in-graph computation. This technical solution not only preserves the compactness and efficiency of the FSA representation but also aligns with the computational paradigms of certain hardware accelerators (e.g., TPUs), thereby providing a substantial technical advantage in the field of natural language processing. Example Finite-State Constraints

[0051] This section provides some background and then elaborates on example contributions provided herein.

[0052] Finite-state automata (FSAs)

[0053] A finite-state automaton ^^^^ is a tuple (^^^^,^^^^, ^^^^,^^^^,^^^^) where ^^^^ is a set of inputsymbols, ^^^^ is a finite set of states, ^^^^ ∈ ^^^^ and ^^^^ ⊆ ^^^^ are initial and final states, and ^^^^ ⊆^^^^ × ^^^^^^^^ × ^^^^, where ^^^^^^^^ = ^^^^ ∪ {^^^^}, is a set of edges. Each edge ^^^^ ∈ ^^^^ is a tuple(^^^^^^^^, ^^^^^^^^, ^^^^^^^^) where ^^^^^^^^ and ^^^^^^^^ are its source and target states and ^^^^^^^^ is its input label, or ^^^^ ifnone.

[0054] As simplified examples for illustration, Figure 1 illustrates FSAs that accept ab (left), odd numbers of “a”s (center), and runs of “a”s or “b”s (right). States are depicted as circles, with the start state in bold and final states doubled. Edges are depicted as directed arcs, labeled with the relevant input symbol, or ^^^^ if none.

[0055] An FSA ^^^^ ^^^^ ∈ ^^^^∗ if there exist ^^^^1, … , ^^^^^^^^ ∈ ^^^^ such that ^^^^ =… , note that ^^^^ > |^^^^| iff ^^^^^^^^^^^^ = ^^^^.

[0056] To express the functional behavior of FSAs, this description overloads notation and defines ^^^^(^^^^) as a predicate that is true if ^^^^ accepts ^^^^. For any FSA ^^^^, define itslanguage ^^^^^^^^ = {^^^^ ∈ ^^^^∗: ^^^^(^^^^)} as the set of strings it accepts. More generally, the regularlanguages can be defined as {^^^^^^^^ ∶ ^^^^ ^^^^^^^^ ^^^^^^^^ ^^^^^^^^^^^^}.

[0057] Conveniently, the regular languages can also be defined by regular expressions, which are equivalent to FSAs. This description uses the mathematical definition of regular expressions. Many tools extend regular expressions with non-regular features like backreferences—at the cost of exponential runtime Common tools like UNIX grep compileregular expressions into FSAs with ^^^^ = ^^^^^^^^^^^^^^^^^^^^^^^^^^^^, which are then used as predicates on text.

[0058] An FSA is deterministic if ∀^^^^ ∈ ^^^^, ^^^^^^^^ ≠ ^^^^ and ∀^^^^ ∈ ^^^^,^^^^ ∈^^^^, |{^^^^ ∈ ^^^^ ∶ ^^^^^^^^ = ^^^^ ∧ ^^^^^^^^ = ^^^^}| ≤ 1. Intuitively, outbound edges have unique inputs soexecuting the FSA is trivial: track the current state and traverse the edge matching the next input. Surprisingly, non-deterministic and deterministic FSAs are equivalent. In particular,given an arbitrary FSA ^^^^ one can build a deterministic FSA ^^^^’ such that ^^^^^^^^ = ^^^^^^^^’

[0059] Figure 2 illustrates an example of this process. In particular, the FSA constructed from the regular expression / a+|ab / is initially nondeterministic (shown on left), but can be determinized (shown on right).

[0060] Finite-state transducers (FSTs)

[0061] A finite-state transducer is an FSA that generates output. Formally, an FST ^^^^is a tuple (^^^^,∆,^^^^, ^^^^,^^^^,^^^^) where ^^^^,^^^^, ^^^^, and ^^^^ are as defined for FSAs, ∆ is a set of outputand ^^^^ ⊆ ^^^^ × ^^^^^^^^ × ∆^^^^ × ^^^^ is a set of edges. Each edge ^^^^ ∈ ^^^^ is a tuple, ^^^^^^^^ , ^^^^^^^^ , ^^^^^^^^� where ^^^^^^^^, ^^^^^^^^, and ^^^^^^^^ are as defined for FSAs and ^^^^ is its output label, or ^^^^ ifnone.

[0062] As simplified examples for illustration, Figure 3 illustrates FSTs that transduce ab into x (left), odd numbers of “a”s into xoxo...x (center), and runs of “a”s or “b”sinto bracketed versions of themselves (right). Edge labels are ^^^^^^^^: ^^^^^^^^.

[0063] An FST ^^^^ transduces ^^^^ ∈ ^^^^∗ into ^^^^ ∈ ∆∗ if there exist ^^^^ , … , ^^^^^^^^ ∈ ^^^^that ^^^^… , … , ^^^^ ^^^^−1 SimilarFSAs, we write ^^^^ = ^^^^(^^^^) if ^^^^ transduces ^^^^ to ^^^^. ^^^^(^^^^) can be a set because a non-deterministic FST may have multiple paths transducing the same input to different outputs.However, in some implementations, the FSTs satisfy |^^^^(^^^^)| = 1 so we simply write ^^^^ =^^^^(^^^^).

[0064] Importantly, the output of one FST can be fed as the input of another FST orFSA. Given FSTs ^^^^ and ^^^^ where ∆ = ^^^^ , we can c ’1 2 1 2 ompose them into a new FST ^^^^ =^^^^2 ◦ ^^^^1 where ^^^^’ = ^^^^1, ∆’= ∆2, and ^^^^ ’(^^^^) = ^^^^2�^^^1^ (^^^^)�. Similarly, given an FST ^^^1^ andFSA ^^^^ ’ ’2 where ∆1= ^^^^2, we can compose them into a new FSA ^^^^ = ^^^^2 ◦ ^^^^1 where ^^^^ =^^^^1 and ^^^^’(^^^^) = ^^^^2�^^^^1(^^^^)�.

[0065] Detokenization as transduction

[0066] One contribution of the present disclosure is a reformulation of detokenization (i.e., the process of converting token sequences back into text) as an FST. Given a vocabulary of tokens ^^^^, some example implementations can construct a detokenizing FST ^^^^^^^^as follows: 1) Let = ^^^^, and let ∆^^^^= ^^^^ ∈ ^^^^, 1 ≤ ^^^^ ≤ |^^^^|} be the characters of tokens^^^^. 2) Add a single root state ^^^^^^^^, with ^^^^^^^^ = ^^^^^^^^ and ^^^^^^^^ = {^^^^^^^^}.3) For each token ^^^^ ∈ ^^^^:a. Let ^^^^ = |^^^^| and build a chain of ^^^^ − 1 states and edges starting from ^^^^^^^^,where ^^^^^^^^^^^^ = ^^^^ and ^^^^^^^^^^^^ = ^^^^^^^^, reusing existing edges to form a structureisomorphic to a prefix trie. b. Add another edge ^^^^^^^^ that loops back to ^^^^^^^^ with ^^^^^^^^^^^^ = ^^^^^^^^ = ^^^^^^^^.

[0067] ^^^^^^^^ has one cycle for each ^^^^ ∈ ^^^^ that accepts ^^^^ and generates its characters.Thus, for any token sequence ^^^^ ∈ ^^^^∗,^^^^^^^^(^^^^) is precisely the detokenized characters of ^^^^.

[0068] As a simplified example of this process, Figure 4 illustrates a simple vocabulary of tokens (left), and a detokenizing FST that transduces sequences of those tokens into sequences of characters (right).

[0069] Adapting regular expressions to tokens

[0070] Another contribution of the present disclosure is a generic method for adapting any FSA from characters to tokens. Specifically, given a token vocabulary ^^^^ and an FSA ^^^^that accepts character sequences, ^^^^^^^^ = ^^^^ ◦ ^^^^^^^^ accepts essentially the same language as ^^^^,but in token form. More precisely, for each token sequence ^^^^ ∈ ^^^^^^^^’, the detokenization of ^^^^is in ^^^^^^^^.

[0071] Note that the converse does not hold: ^^^^ ∈ ^^^^^^^^ has a counterpart in ^^^^^^^^’ only if^^^^ can be segmented into tokens from ^^^^. For example, suppose ^^^^ accepts any number in hexadecimal format, but the tokens in ^^^^ only cover the digits 0-9. Nevertheless, to the extentthat ^^^^^^^^can be tokenized by ^^^^, strings in ^^^^^^^^are represented in ^^^^^^^^’. Indeed, due to tokenization ambiguity, there may be multiple token sequences in ^^^^^^^^’ that are equivalent to the same string in ^^^^^^^^.

[0072] To provide a simplified example, Figure 5 illustrates the character-based FSA equivalent to / (foo)+d / (left) and its composition with the detokenizing FST from Figure 4 (right). Note that the same text can have many tokenizations (e.g., foo vs f oo), and tokens are allowed to cross sub-expression boundaries (e.g., food merges the last repeat of / (foo)+ / with / d / ).

[0073] One example method for constraining a sequence processing model to a regular language is as follows: Before decoding: 1) Convert the model vocabulary ^^^^ into a detokenizing FST ^^^^^^^^. 2) Define a regular expression ^^^^ that matches the desired output. 3) Compile ^^^^ into an FSA ^^^^^^^^that accepts characters. 4) Adapt ^^^^^^^^ into an FSA ^^^^^^^^◦^^^^ = ^^^^^^^^ ◦ ^^^^^^^^ that accepts tokens.5) Determinize ^^^^^^^^◦^^^^for efficient execution.

[0074] Note that ^^^^^^^^◦^^^^is a closed-form solution: it expresses ^^^^ using all relevant tokens from ^^^^ and can be executed independently from both. For example, when decoding,initialize ^^^^ = ^^^^^^^^◦^^^^ and then on each decoding step:1) Read the valid next tokens off the outbound edges: {^^^^^^^^: ∃^^^^ ∈ ^^^^, ^^^^^^^^ = ^^^^}.2) If ^^^^ ∈ ^^^^^^^^◦^^^^, then also treat the end token < / ^^^^ > as valid.3) Penalize the decoding logits of invalid tokens. 4) After the model selects a token, update ^^^^ by traversing the matching edge.

[0075] The required operations are indexing, slicing, and basic arithmetic, which are efficient and simple enough to execute directly on an accelerator with minimal latency overhead. One pleasing aspect of this solution is how neatly it separates concerns amongst the two halves: ^^^^^^^^is vocabulary-specific and, while large, can easily be pre-computed for each model; ^^^^^^^^is vocabulary-agnostic, easily specified, and portable across different models.

[0076] This clean decomposition is only possible because FST-FSA composition provides a fast, automatic, and general method for joining the two halves.

[0077] Referring now to Figure 6, a schematic representation is provided to illustrate the composition of a vocab finite state transducer (FST) 102 and a regex finite state automaton (FSA) 104 to form a composed vocab+regex FSA 106. The vocab FST 102 serves as a linguistic processor that converts a sequence of tokens into a corresponding text stringbased on a predefined vocabulary. The tokens input to the vocab FST 102 represent discrete units of text, such as words or phrases, that are recognized by the system’s vocabulary.

[0078] The regex FSA 104, on the other hand, functions as a pattern recognition module that evaluates text strings against specified regular expression patterns. The regular expressions define criteria for text string matching, allowing for the identification and extraction of information that conforms to the patterns.

[0079] When the vocab FST 102 and regex FSA 104 are composed, the resulting composed vocab+regex FSA 106 integrates the capabilities of both components. The composition process involves the intersection of the vocab FST 102 and regex FSA 104 state machines, thereby creating a new FSA that can process tokens through the vocabulary constraints and simultaneously apply regular expression patterns to the generated text. This integrated FSA enables more sophisticated text analysis by leveraging both vocabulary-based processing and regular expression pattern matching in a single operation. Specifically, the vocab FST 102 would first convert the tokenized input into text, which would then be scrutinized by the regex FSA 104 to locate and extract entities that match the defined regular expression patterns.

[0080] Referring now to Figure 7, a schematic diagram is presented illustrating the operational flow of the token processing and validation system within the disclosed invention. The system commences with a sequence of tokens 122, which are fed into the sequence processing model 124. The sequence processing model 124 is configured to analyze the sequence of tokens 122 and generate corresponding token scores 126, which reflect likelihood of each token in the vocabulary being the next token in the sequence. In some implementations, the plurality of scores 126 generated by the sequence processing model 124 are in the form of logit scores. These logit scores quantify the probability of each token being the subsequent addition to the sequence 122.

[0081] In parallel with the sequence processing model 124, a finite-state automaton 128 operates to maintain a current state 130 based on the sequence processing. The finite- state automaton 128 is a computational model that transitions between different states based on the input tokens and its predefined state transition rules. The current state 130 represents the current operational status of the finite-state automaton 128, which is influenced by the sequence of tokens 122 processed thus far.

[0082] The finite-state automaton 128 operates to output validity statuses 132, which are indicative of the legitimacy or acceptability of each of the tokens in the vocabulary being available to be output as the next token in the sequence. These validity statuses 132 are usedby decoding logic 134 to understand which tokens are permissible and should be considered in generating an output token 116.

[0083] In particular, in some implementations, the evaluation of token validity by finite-state automaton 128 can include a procedure where the current state 130 of the finite- state automaton 128 is determined. Then, tokens associated with outbound edges from the current state 130 are assigned a positive validity status, indicating their permissibility in the sequence. Conversely, tokens that do not correspond to any outbound edge are assigned a negative validity status, marking them as invalid for the continuation of the sequence. This evaluation ensures the generated sequences are both contextually and syntactically coherent.

[0084] Next, the token scores 126 and the validity statuses 132 are input into the decoding logic 134. The decoding logic 134 interprets the token scores 126 and determines the most probable output token 116 from the given scores. The decoding logic 134 may utilize various algorithms, such as maximum likelihood estimation or neural network-based decoding, to accurately predict the output token 116. The decoding logic 134 may also be configured to handle multiple candidate tokens and select the output token 116 based on a scoring system, probability distribution, or other decision-making criteria.

[0085] In some implementations, the selection of an output token by decoding logic 134 is a two-step process that involves the modification of the scores 126 based on the validity statuses 132 determined by the finite-state automaton 128. Initially, the decoding logic 134 adjusts the scores 126 for the tokens, creating a modified set of scores that reflect the validity of each token. Subsequently, the decoding logic 134 selects the output token 116 from this modified score set, ensuring that the chosen token is both statistically likely and syntactically valid within the constraints of the formal language.

[0086] In some implementations, when the decoding logic 134 modifies the scores 126, it penalizes the scores associated with tokens deemed invalid by the automaton’s evaluation. This penalization effectively lowers the probability of selecting syntactically incorrect tokens, thereby steering the model towards generating sequences that are in compliance with the defined syntax rules.

[0087] Alternatively, the decoding logic 134 may mask the scores 126 of tokens with a negative validity status, effectively removing these tokens from consideration in the current decoding iteration. Masking more directly impacts the model’s token selection process by excluding invalid tokens from the set of potential outputs. This ensures that the generated sequences are not only probable but also adhere strictly to the formal language constraints.

[0088] The output token 116 is then appended to the sequence, as indicated by the “Append” label, which signifies the expansion of the token sequence with the newly determined token. The system may also include an “Update” feedback loop, wherein the output token 116 influences the subsequent current state 130 of the finite-state automaton 128, allowing for dynamic state transitions within the finite-state automaton 128 to account for the updated sequence of tokens 122.

[0089] It should be appreciated that the described system can be adapted to various applications, such as natural language processing, code compilation, or any domain requiring sequence analysis and token validation. Alternative embodiments may include additional components or different configurations of the decoding logic 134 and finite-state automaton 128 to cater to specific use cases or performance requirements.

[0090] Example Extensions

[0091] The present disclosure also provides a set of regular expression extensions, written as specially-named capturing groups, that greatly increase the efficiency and expressiveness of the system. Some illustrative examples are described below.

[0092] Wildcard matching:

[0093] One challenge with applying regular expressions to model outputs is with “wildcard” matches like / . / or / [^0-9] / that match nearly any character. After composition with ^^^^^^^^, some states in the resulting FSA will have almost |^^^^| outbound edges, making itexpensive to use—for example, |^^^^| > 100,000 for some commonly-used large languagemodels.

[0094] Example implementations of the present disclosure can mitigate this issue by defining terminal labels, which are token IDs disjoint from ^^^^ that map to pre-computed masks of valid tokens. For example, the specially-named capturing group / (?P<PARAGRAPH TOKEN>) / parses into a terminal edge whose mask indicates newline- free tokens. This is useful for structuring free text: for example, / Summary:(\n\* (?P<PARAGRAPH TOKEN>)+){3,5} / would match the heading “Summary:” followed by three to five bullets. When penalizing logits, the terminal masks can be applied en masse. When updating state, if the selected token matches a terminal mask¸ the system can traverse that terminal edge.

[0095] Note that a normal token edge and terminal edge can both match the sampled token, leading to ambiguity about which to traverse. Similarly, two terminal edges can match the sampled token. Some example implementations can address this with a simple heuristic:prefer normal token edges when available, and otherwise follow a semi-arbitrary precedence ordering over terminal labels. This heuristic works well because terminals are generally used for wildcard matches, and in practice wildcards are typically abutted by disjoint delimiting expressions. For example, in the bulleted list constraint above, the / (?P<PARAGRAPH TOKEN>)+ / expression is bounded by a newline.

[0096] One additional example extension is the QUOTED_TEXT label, which can operate to match quoted strings with specific escape sequences. This label can be represented as a regular expression that matches “-delimited strings, handling backslash escapes such as \”, \n, and \\. The label can utilize pre-computed masks for tokens that start or end with quotes or contain no quotes at all, ensuring that the selected token can be unambiguously processed by the FSA. Some implementations of QUOTED_TEXT can be designed to respect the hard token boundaries imposed by the FSA while providing the necessary flexibility to account for complex quoting patterns in text. This is achieved by employing terminal labels and alternate paths that accommodate different tokenization scenarios, thus enhancing the model’s ability to generate syntactically precise and contextually appropriate strings within the specified constraints.

[0097] Thus, for example, a specially-named empty group / (?P<QUOTED_TEXT>) / matches a “-delimited string and is equivalent to the following regex:

[0098] / ” *(?:[^\s”\\]|\\[“n\\])(?: |[^\s”\\]|\\[“n\\])*” /

[0099] Specifically, it matches a quoted string that:

[0100] Only uses the following backslash escapes: \”, \n, and \\.

[0101] Does not end in an unescaped backslash.

[0102] Contains no unescaped quotes.

[0103] Contains no whitespace except spaces.

[0104] Contains at least one non-space.

[0105] Note that in some implementations the regex above cannot be used directly, since the character classes match too broadly-as described above, such character classes are forbidden in some implementations.

[0106] While QUOTED_TEXT forbids empty and whitespace-only strings, they can be included using normal regex syntax:

[0107] / (?P<QUOTED_TEXT>)|”“ /

[0108] / (?P<QUOTED_TEXT>)|” *” /

[0109] To give some sense for how terminal expressions work, a simplified version that behaves like the regex / ”[^”]*” / is now described. This version would expand to the following pseudo-regex:

[0110] / [[:both:]]|(“|[[:start:]])[[:none:]]*(“|[[:end:]]) /

[0111] where the [[:foo:]] terminal labels match the following sets of tokens:

[0112] [[:start:]]: Tokens that start with “.

[0113] [[:end:]]: Tokens that end with “

[0114] [[:none:]]: Tokens with no “ on either side.

[0115] [[:both:]]: Tokens with “ on both sides.

[0116] Note: The pseudo-regex is called such because it explicitly matches tokens, via the terminal labels, rather than only matching characters.

[0117] Masks for these four terminal labels can be pre-computed from the vocabulary and applied whenever the current FSA state has an outbound arc with one of these terminal labels. The sets above are disjoint from each other and from “ and space tokens, so once the decoder selects a token we can unambiguously infer which arc to traverse in the FSA.

[0118] As explained earlier, terminal labels reduce the flexibility of the FSA by establishing “hard” token boundaries. The pseudo-regex mitigates this by introducing alternate paths that use the “ character.

[0119] As one example, Figure 8 illustrates an example FSA implementation of the QUOTED_TEXT extension. In Figure 8, the ALL_CAPS arcs are terminal labels with the following meanings:

[0120] TOKEN_QUOTE_QUOTE_QUOTE: Tokens starting with “ and ending with an unescaped “.

[0121] TOKEN_QUOTE_QUOTE_SLASH: Tokens starting with “ and ending with an unescaped \.

[0122] TOKEN_QUOTE_QUOTE_OTHER: Tokens starting with “ and ending with something else.

[0123] TOKEN_QUOTE_ESCAPE_QUOTE: Tokens starting with an escapable character, like n, and ending with an unescaped “.

[0124] TOKEN_QUOTE_ESCAPE_SLASH: Tokens starting with an escapable character, like n, and ending with an unescaped \.

[0125] TOKEN_QUOTE_ESCAPE_OTHER: Tokens starting with an escapable character, like n, and ending with something else.

[0126] TOKEN_QUOTE_NORMAL_QUOTE: Tokens starting with a non-”, and ending with an unescaped “.

[0127] TOKEN_QUOTE_NORMAL_SLASH: Tokens starting with a non-”, and ending with an unescaped \.

[0128] TOKEN_QUOTE_NORMAL_OTHER: Tokens starting with a non-”, and ending with something else.

[0129] Another example extension is the UNQUOTED_TEXT label, which can operate to match YAML-style non-quoted strings. This label is particularly adept at recognizing key-value separators, denoted by a colon followed by a space, and list nesting indicators, such as a hyphen followed by a space. The terminal can be implemented using a pseudo-regex approach that provides flexibility around token boundaries, ensuring accurate parsing even when tokens do not align perfectly with YAML’s syntactical elements. The UNQUOTED_TEXT label can avoid errors by only matching tokens that start with a space and do not begin with YAML special characters, thereby ensuring that the FSA can process YAML content accurately. This extension employs terminal labels to match tokens based on their position and content, allowing for the generation of structurally and contextually correct YAML and JSON outputs, which is vital for the technical integrity of systems relying on these data formats.

[0130] Thus, the specially-named empty group / (?P<UNQUOTED_TEXT>) / matches a YAML-style non-quoted string. Since there are no “s to mark the extent of the string content, YAML control sequences will be interpreted when they appear in a non-quoted string. In particular,

[0131] A : followed by a space is interpreted as a key-value separator:

[0132] # As YAML.

[0133] Avengers: Endgame

[0134] # As JSON.

[0135] {“Avengers”: “Endgame”}

[0136] At the end of a line, a : indicates a key-value with a null value:

[0137] # As YAML.

[0138] Proof of Theorem 1:

[0139] # As JSON.

[0140] {“Proof of Theorem 1”: null}

[0141] At the start of a line, a - followed by a space indicates list nesting:

[0142] # As YAML.

[0143] - 4 - 2 = - 6

[0144] # As JSON.

[0145] [“4 - 2 = - 6”]

[0146] These control sequences will cause errors if they appear in an unquoted string. To avoid such errors, this terminal behaves equivalently to the following regex:

[0147] / +(?:[^][{}\s”‘:-]|[:-]:*[^\s:])(?: |:*[^\s:])* /

[0148] Specifically, it matches a non-quoted string with the following properties:

[0149] It begins with at least one space. This allows the implementation to provide some flexibility around token boundaries, and is not restrictive since the normal use case (as an array element or as the value for some key) already requires a preceding space.

[0150] Its first non-space is not “ or ‘. Use QUOTED_TEXT for that.

[0151] Its first non-space is not {, or a few similar characters like [.

[0152] If its first non-space is -, or a few similar characters like ?, then it must be followed by a non-space.

[0153] A : is always followed by a non-space.

[0154] Like QUOTED_TEXT, this terminal is implemented as a pseudo-regex that uses terminal labels to match large sets of tokens, and the pseudo-regex provides multiple paths to “soften” token boundaries.

[0155] Figure 9 illustrates an example FSA implementations of the UNQUOTED_TEXT extension. In Figure 9, the ALL_CAPS arcs are terminal labels with the following meanings:

[0156] TOKEN_UNQUOTE_SPACE_COLON: Tokens starting with space and ending with :.

[0157] TOKEN_UNQUOTE_SPACE_NOCOLON: Tokens starting with space and ending with non-:.

[0158] TOKEN_UNQUOTE_NOSPACE_COLON: Tokens starting with non-space and ending with :.

[0159] TOKEN_UNQUOTE_NOSPACE_NOCOLON: Tokens starting with non- space and ending with non-:.

[0160] TOKEN_UNQUOTE_START_SPACE_COLON: Tokens starting with space and ending with : that are also non-blank and exclude YAML special characters.

[0161] TOKEN_UNQUOTE_START_SPACE_NOCOLON: Tokens starting with space and ending with non-: that are also non-blank and exclude YAML special characters.

[0162] TOKEN_UNQUOTE_START_NOSPACE_COLON: Tokens starting with non-space and ending with : that are also non-blank and exclude YAML special characters.

[0163] TOKEN_UNQUOTE_START_NOSPACE_NOCOLON: Tokens starting with non-space and ending with non-: that are also non-blank and exclude YAML special characters.

[0164] Another example extension is the IMAGE label. The IMAGE label can operate to match sequences of quantized image tokens. It can be defined by a specially- named empty group within the finite-state automaton (FSA) and can be represented by a pseudo-regex that delineates the start, content, and end of an image sequence. The terminal labels [[:start:]], [[:content:]], and [[:end:]] can be used to match the respective special start- of-image token, any quantized image token, and the special end-of-image token. This mechanism simplifies the process of distinguishing between text and image tokens, ensuring that multi-modal sequence processing models can accurately generate and incorporate image data in their outputs. Thus, the IMAGE label can seamlessly integrate visual content within text sequences, enhancing the functionality and applicability of multi-modal systems.

[0165] Another example extension is the TEXT_TOKEN label, which matches individual text tokens within a sequence processing model’s vocabulary, as defined in a specially-named empty group within the finite-state automaton (FSA). This label can isolate single text tokens, excluding tokens specific to non-textual content such as images in multi- modal models. The versatility of the TEXT_TOKEN label is further enhanced by its compatibility with quantifiers, allowing it to match sequences of text tokens, which can range from empty to multiple tokens in length. In multi-modal models, the related ANY_TOKEN terminal expands this functionality to include both text and image tokens, whereas in text- only models, TEXT_TOKEN serves as the standard matcher for any token in the vocabulary. This extension ensures that the FSA can accurately process and generate sequences that are syntactically coherent within both text-only and multi-modal contexts, thereby addressing the technical need for precise token identification and selection in complex language modeling applications.

[0166] For example, a specially-named empty group / (?P<ANY_TOKEN>) / matches a single token of any kind (except control tokens like < / s> or <pad>). Note that this can be used with quantifiers, as in:

[0167] / (?P<ANY_TOKEN>)+ /

[0168] The above is not the same as / .+ / , which matches any non-empty character (not token) sequence and is therefore allowed to start or end matching in the middle of a token. Moreover, / .+ / is forbidden as explained above.

[0169] Another example extension (introduced above) is the PARAGRAPH_TOKEN. The PARAGRAPH_TOKEN extension is a specialized feature within the finite-state automaton framework that matches single tokens without line breaks, enabling precise control over text generation at the paragraph level. This terminal distinguishes itself from the TEXT_TOKEN by specifically excluding tokens that contain line breaks, thereby facilitating the generation of text where paragraph integrity is maintained. The use of quantifiers with the PARAGRAPH_TOKEN allows for the definition of patterns that can match sequences of tokens forming complete paragraphs. For instance, by using the pattern / (?P<PARAGRAPH_TOKEN>)+\n(?P<PARAGRAPH_TOKEN>)+\n(?P<PARAGRAPH_T OKEN>)+ / , the system can decode text that consists of exactly three paragraphs, each separated by a newline character. This capability is beneficial for applications that require structured text outputs, such as document generation or formatting tasks where paragraph demarcation is a technical requirement.

[0170] Another example extension is the TEXT_UNTIL extension. The TEXT_UNTIL extension can be used to manage context transitions within sequence processing models, addressing the need to switch between free text and structured text based on specific textual markers. This is particularly useful in scenarios where a language model is required to generate a combination of unconstrained text and code or other structured data, delineated by markers such as triple-backtick fences. The TEXT_UNTIL terminal is implemented as a regular expression that matches any sequence up to but not including a defined stop phrase, ensuring that the finite-state automaton (FSA) processes the sequence correctly up to the point of transition. This technical feature allows for precise control over the model’s output and enables complex text generation tasks that involve varying levels of structure and constraints. The ability to embed TEXT_UNTIL within other regular expression constructs further enhances its utility, providing a means to implement limited forms of orchestration and control flow within the generated text sequences.

[0171] As an example, in some use cases, one needs to “switch” between different behavior depending on some textual marker. For example, in a tool-use scenario we might want the LM to generate a mix of free text and constrained structures, like so:

[0172] Free text…

[0173] my_tool(arg1=123, arg2=“foo”, ...)

[0174] where the triple-` code fence marks a transition from free text to code. One might be tempted to write a constraint regex like so:

[0175] / (?P<TEXT_TOKEN>)*\n\n``TOOL_CALL\nmy_tool\(arg1=[0-9]+, ...\)\n`` /

[0176] Unfortunately, the initial / (?P<TEXT_TOKEN>)* / can match anything, including a fenced code block, allowing the LM to generate code as free text and ignore the intended code constraints. Ideally, the initial wildcard would match anything except the TOOL_CALL code fence, so it forces the FST into the latter half of the regex.

[0177] This issue is not peculiar to constraint regexes, the same problem arises in:

[0178] / .*\n\n``TOOL_CALL\n...\n``\n\n.* /

[0179] For some common use cases, the terminals have been designed so these transitions are naturally forced. For example, PARAGRAPH_TOKEN can be “fenced” by newlines, and TEXT_TOKEN and IMAGE use disjoint tokens so they fence each other.

[0180] The TEXT_UNTIL terminal allows one to define forcing transitions based on custom text markers, with syntax like the following:

[0181] / (?P<TEXT_UNTIL>\nstop phrase\n) /

[0182] This behaves similarly to:

[0183] / (?P<TEXT_TOKEN>)*\nstop phrase\n /

[0184] except that the / (?P<TEXT_TOKEN>)* / stops matching right before the “\nstop phrase\n” appears in full.

[0185] Since TEXT_UNTIL has *-style repetition, it allows the LM to immediately generate the stop phrase. If you want +-style repetition, just prepend a PARAGRAPH_TOKEN:

[0186] / (?P<PARAGRAPH_TOKEN>)(?P<TEXT_UNTIL>\nstop phrase) /

[0187] Using TEXT_UNTIL, we can properly constrain the code-fencing example:

[0188] / (?P<TEXT_UNTIL>\n\n``TOOL_CALL\n)my_tool\(arg1=[0-9]+, ...\)\n`` /

[0189] Note that the TEXT_UNTIL terminal is a regular expression like any other, and can be composed using normal regex operators. For example, a TEXT_UNTIL can be embedded inside a DELIMITED_SUBSEQUENCE_OF, or one can take a union between TEXT_UNTIL and some other matcher, or between two TEXT_UNTILs, to allow “alternate exits” from a run of free-text. This allows some limited forms of orchestration and control flow to be implemented in a regular expression.

[0190] Syntactic sugar:

[0191] Sometimes, a constraint is inefficient to define as a regular expression. For example, suppose it is an objective to reduce hallucination by forcing the model to generate a substring of some reference ^^^^, such as a user utterance. Unfortunately, the regular expressionfor substrings of ^^^^ is ^^^^(|^^^^|2), even though the corresponding FSA is ^^^^(|^^^^|)—this can beviewed as a defect of the specification language. Therefore, some example implementations can define various syntactic extensions, examples of which are as follows.

[0192] One example syntactic extension is the SUBSTRING_OF extension. For example, the expression / (?P<SUBSTRING OF>abc) / can be used to match any substring of “abc”, resolving the defect by providing a ^^^^(|^^^^|) specification syntax. For instance, the expression / (?P<SUBSTRING_OF>foo bar baz) / would match substrings such as “f”, “baz”, and “o bar b”. This extension can be implemented as a finite-state automaton (FSA) network with a series of states representing the sequence of characters in the text, and transitions that allow for the matching of substrings starting and ending at any point within the text.

[0193] For example, a specially-named group / (?P<SUBSTRING_OF>...) / matches any non-empty substring of the enclosed text. For example, / (?P<SUBSTRING_OF>foo bar baz) / would match “f”, “baz”, and “o bar b”. If no text is provided, as in / (?P<SUBSTRING_OF>) / , then it matches nothing.

[0194] Unlike many of the terminals above, this is a character pattern, not a token pattern. This terminal may be useful for, e.g., mitigating hallucinations by forcing a text field to extract from the user query.

[0195] The basic idea is to build a FSA network that consists of the following:

[0196] A chain of states encoding the text from which the substring is drawn.

[0197] Arcs from the start state to each point along the chain, allowing the substring to start anywhere.

[0198] Arcs from each point along to the final state, allowing the substring to end anywhere.

[0199] Figure 10 illustrates an example FSA network for substrings of “foobar”.

[0200] Another example syntactic extension is the DELIMITED_SUBSEQUENCE_OF extension, which can be used to address the problem of matching non-empty subsequences within a sequence of delimited items. This is beneficial for parsing and generating structured data with sequence processing models. This specially- named group functions by capturing subsequences from a list of items separated by aspecified delimiter, such as in the pattern / (?P<DELIMITED_SUBSEQUENCE_OF>([+- ])(a)(b+)(c[de])) / , which can match various combinations of the provided items with the delimiters intact. The extension ensures technical precision in the matching process by allowing the inclusion of required items in the subsequence, which must always be present, thus enabling complex data structure generation that adheres to specified formats. By constructing a finite-state automaton (FSA) network that encodes the sequence and utilizes arcs to represent possible subsequences, including those with required items, the DELIMITED_SUBSEQUENCE_OF extension is able to efficiently and accurately handle delimited data within the constraints of regular expressions, which is of particular importance for applications that involve data serialization and deserialization, such as APIs and data interchange formats.

[0201] Thus, for example, a specially-named group / (?P<DELIMITED_SUBSEQUENCE_OF>...) / matches any non-empty delimited sub- sequence of the enclosed arguments. For example, / (?P<DELIMITED_SUBSEQUENCE_OF>([+-])(a)(b+)(c[de])) / would match “a”, “b”, “cd”, “a+bb”, “a-ce”, “bbb+cd”, “a-bbbb+ce”, and so on.

[0202] In some implementations, each argument can be wrapped in a capturing group to prevent the RE2 parser from rearranging it. More concretely, the first argument matches the delimiter and the rest match the list from which a subsequence is drawn. The matched subsequence has a delimiter in between each pair of items, and is never empty---unless the original list was empty.

[0203] If any of the arguments is wrapped in a capturing group named REQUIRED, then it will always be included in the matched subsequence. For example, if we modified the example above to: / (?P<DELIMITED_SUBSEQUENCE_OF>([+-])(a)(?P<REQUIRED>b+)(c[de])) / then it would no longer match “a”, “cd”, or “a-ce”.

[0204] The basic idea is to build a FSA network that consists of the following:

[0205] A chain of states encoding the full item-delimiter-item-delimiter-... sequence.

[0206] Arcs from the start state to each point along the chain, allowing the subsequence to start anywhere.

[0207] Arcs skipping individual (delimiter, item) pairs.

[0208] As an example, Figure 11 illustrates an example FSA network for comma- delimited subsequences of the items “a”, “b”, and “c”.

[0209] To handle required items, some example implementations simply remove some of the epsilon arcs. Specifically: Remove all epsilon arcs from the start state to states after the first required item; and Remove all epsilon arcs that skip the (delimiter, item) pair for required items. Figure 12 provides an example.

[0210] Another example syntactic extension is the DELIMITED_LIST extension, which can be used to match non-empty lists with specified delimiters. This extension can use a specially-named group to match sequences where a delimiter separates list items, and supports bounded repetition to control the minimum and maximum occurrences of these items. For example, the expression / (?P<DELIMITED_LIST>([+-])(a+)) / can match lists like “a+aa” or “aaa-aa+a”, capturing the structure of lists as found in programming languages, including the syntax of Python tuples which may require a trailing delimiter. This extension is beneficial for generating syntactically accurate and delimited sequences, such as arrays or parameter lists, which are essential for the technical operation of software systems that process such lists. The DELIMITED_LIST terminal can be implemented as a cycle within the FSA, providing a technical mechanism to ensure the correct formation of lists according to the specified pattern.

[0211] Thus, for example, a specially-named group / (?P<DELIMITED_LIST>...) / matches a non-empty delimited list specified by its two arguments. For example, / (?P<DELIMITED_LIST>([+-])(a+)) / would match “a”, “a+aa”, “aaa-aa+a”, “a-aaa+aa-aaaa”, and so on.

[0212] In some implementations, each argument can be wrapped in a capturing group to prevent the RE2 parser from rearranging it. More concretely, the first argument matches the delimiter and the second matches an item in the list. The matched list has a delimiter in between each pair of items, and is never empty.

[0213] This is similar to DELIMITED_SUBSEQUENCE_OF, but this matches a homogeneous list of the same item pattern, while that matches (parts of) an explicitly enumerated list of heterogeneous item patterns.

[0214] Bounded repetition can be supported by appending an optional min and max to the terminal name, with behavior similar to / a{m,n} / :

[0215] # Matches at least two “a”s.

[0216] / (?P<DELIMITED_LIST_2>(,)(a)) /

[0217] # Matches one to three “a”s.

[0218] / (?P<DELIMITED_LIST_1_3>(,)(a)) /

[0219] In some implementations, since this terminal always matches a non-empty list, the min and max, when specified, must be at least 1.

[0220] Python tuples have unusual syntax, in that a single-item tuple requires a trailing delimiter; e.g., (1,). This use case can be supported by appending an optional _T to the terminal name, which forces a trailing delimiter on single-item lists:

[0221] # Matches “a,”, “a,a”, etc.

[0222] / (?P<DELIMITED_LIST_T>(,)(a)) /

[0223] # Matches “a,a”, “a,a,a”, etc.

[0224] / (?P<DELIMITED_LIST_T_2>(,)(a)) /

[0225] # Matches “a,”, “a,a”, and “a,a,a”.

[0226] / (?P<DELIMITED_LIST_T_1_3>(,)(a)) /

[0227] The basic network structure is a cycle, with the forward path matching a list item, and the backward path matching a delimiter. Figure 13A provides a simple example with item “a” and delimiter “,”:

[0228] With bounded repetition, if only the minimum is specified, then that number of (delimiter,item) pairs must be duplicated before entering the loop. Figures 13B and 13C provide examples.

[0229] With both minimum and maximum specified, then a chain of items and delimiters is unrolled up to the maximum, and states are marked final to allow ending when the required number of items is matched. Figures 13D and 13E provide examples.

[0230] To handle tuple-like behavior, there are three cases:

[0231] 1) If there is no bounded minimum or maximum, then unroll to at least two copies of the item and adjust the final state to ensure that a comma appears. Figure 13F provides an example.

[0232] 2) If there are bounds and the minimum is more than 1, then the network is identical to the non-tuple-like network, since tuple-like behavior only affects singleton lists.

[0233] 3) If the bounds are that the minimum and maximum are both 1, then add a trailing delimiter. Figure 13G provides an example.

[0234] 4) For all other bounded cases, the network structure is identical to the non- tuple-like network except the first final state is adjusted to include the delimiter. Figure 13H provides an example.

[0235] Besides improving usability, these extensions mitigate the explosion in regular expression size that can occur in complex applications like JSON constraints. In some cases,growth in the regular expression reflects growth in the resulting FSA, but it has been found that they diverge in some important practical use cases. Push-Down Constraints

[0236] This section briefly presents another set of contributions, describing how the proposed systems and methods can easily be adapted to grammar-based constraints using push-down automata. The section begins with some background on PDAs and then describes example contributions.

[0237] Push-down automata (PDAs)

[0238] A push-down automaton can be viewed as an FSA equipped with a stack.Formally, a PDA ^^^^ is a tuple (^^^^,^^^^, ^^^^,^^^^, ^^^^,^^^^,^^^^) where ^^^^,^^^^, ^^^^, and ^^^^ are as defined for FSAs,^^^^ is a set of stack symbols, ^^^^ ∈ ^^^^ is a unique initial stack symbol, and ^^^^ ⊆^^^^ × ^^^^ × ^^^^∗ × ^^^ ∗^^^^ ^ × ^^^^ is a set of edges. Each edge ^^^^ ∈ ^^^^ is a , , , ,where ^^^^^^^^, ^^^^^^^^, and ^^^^^^^^are as defined for FSAs, and ^^^^↑and ^^^^↓are popped and pushed stack symbols.

[0239] As simplified examples, Figure 14 illustrates a grammar for the canonical context-free language ^^^^^^^^^^^^^^^^(left), and an equivalent PDA (right). Edge labels have ^^^^^^^^on top, then ^^^^↑marked with ) and ^^^^↓marked with (. The stack symbols are Earley-style dotted rules denoting “return addresses”. The initial stack symbol ^^^^ is written as a dot marked with square brackets [ and ].

[0240] ^^^^ ^^^^ ∈ ^^^^∗ if there exist ^^^^ , … , ^^^^ ^^^^ ^^^^1 ^^^^^^^^ ∈ ^^^^ such that ^^^^ = ^^^^1 … ^^^^^^^^ , ^^^^1 =^^^^, ^^^^^^^^ ^^^^↑ ↓^^^^ ∈ ^^^^, ^^^^^^^^ = for ^^^^ > 1, and ^^^^ and ^^^^ form a coherent of stack edits.Formal let ^^^^^^^^ be the stack before ^^^^ where ^^^^1 = ^^^^,^^^^^^^^ =^^^^1^^^^:^^^^^^^^^^^^^^↓^^, then ^^^^^^↑^^= ^^^^^^^^^^^^^^^^+1:^^^^^^^^must hold. In other words, at each step ^^^^^^↑^^must match the suffix of ^^^^^^^^, and the system can update the stack by popping ^^^^^^↑^^off and pushing ^^^^^^↓^^on. As above, we write ^^^^(^^^^) to indicate that ^^^^ accepts ^^^^.

[0241] Unlike FSAs, non-deterministic and deterministic PDAs are not equivalent: the former accept the context-free languages, which are also defined by context-free grammars, while the latter accept a strict subset, the deterministic context- free languages. This subset nevertheless includes all regular languages and the LL(k) and LR(k) languages, covering the syntax of most programming languages.

[0242] As a result of this non-equivalence, there is no general algorithm to determinize an arbitrary PDA, as we had for FSAs. Instead, as LL(k) and LR(k) parsers do,our system detects non-deterministic grammars and signals an error, leaving it to the author of the grammar to rewrite the grammar in a deterministic manner.

[0243] Adapting Grammars to Tokens

[0244] Although PDAs are more expressive than FSAs, they behave similarly in many ways and, importantly, FSTs and PDAs are composable. Formally, given an FST ^^^^1and PDA ^^^^ where ∆ = ^^^^ , t ’2 1 2 hey can be composed into a new PDA ^^^^ = ^^^^2 ◦ ^^^^1 where^^^^’ = and ^^^^’(^^^^) = ^^^^2(^^^^1(^^^^)). As with FSAs, composition with the detokenizing FST ^^^^^^^^adapts a character-based PDA into one that accepts tokens in ^^^^.

[0245] As simplified examples, Figure 15 illustrates a simple vocabulary of tokens (left), the detokenizing FST built from it (center), and its composition with the PDA from Figure 14 (right). Note that edges are allowed to cross the boundaries of terminals and non- terminals, allowing them to push or pop multiple stack symbols. For legibility, the stack symbols have been simplified into single brackets that are interleaved with the characters of the token.This approach enables the methods described above with reference to FSAs to be adapted to PDAs with minimal modification: 1) Convert the model vocabulary ^^^^ into a detokenizing FST ^^^^^^^^. 2) Define the constraints as a deterministic grammar ^^^^. 3) Compile ^^^^ into a character-based PDA ^^^^^^^^. 4) Adapt ^^^^^^^^ into a PDA ^^^^^^^^◦^^^^ = ^^^^^^^^ ◦ ^^^^^^^^ that accepts tokens.

[0246] When decoding, the handling system can apply ^^^^^^^^◦^^^^similarly to ^^^^^^^^◦^^^^, except that the system can maintain a stack ^^^^ alongside ^^^^, filter edges by compatibility between ^^^^↑and ^^^^, and update ^^^^ according to ^^^^↑and ^^^^↓.

[0247] The benefits of formulating detokenization as an FST can be emphasized as follows. By representing the entire task in terms of automata, the proposed approach easily generalizes from regular expressions to grammars while preserving many desirable features. For example, just as FST-FSA composition enables tokens to cross sub-expression boundaries in the regular expression, FST-PDA composition enables tokens to cross (non)terminal boundaries in the grammar. Example Model Training

[0248] Figure 16 depicts a flowchart of a method 1600 for training one or more machine-learned models according to aspects of the present disclosure. For instance, an example machine-learned model can include a sequence processing model.

[0249] One or more portion(s) of example method 1600 can be implemented by a computing system that includes one or more computing devices such as, for example, computing systems described with reference to the other figures. Each respective portion of example method 1600 can be performed by any (or any combination) of one or more computing devices. Moreover, one or more portion(s) of example method 1600 can be implemented on the hardware components of the device(s) described herein, for example, to train one or more systems or models. Figure 16 depicts elements performed in a particular order for purposes of illustration and discussion. Those of ordinary skill in the art, using the disclosures provided herein, will understand that the elements of any of the methods discussed herein can be adapted, rearranged, expanded, omitted, combined, or modified in various ways without deviating from the scope of the present disclosure. Figure 16 is described with reference to elements / terms described with respect to other systems and figures for exemplary illustrated purposes and is not meant to be limiting. One or more portions of example method 1600 can be performed additionally, or alternatively, by other systems.

[0250] At 1602, example method 1600 can include obtaining a training instance. A set of training data can include a plurality of training instances divided between multiple datasets (e.g., a training dataset, a validation dataset, or testing dataset). A training instance can be labeled or unlabeled. Although referred to in example method 1600 as a “training” instance, it is to be understood that runtime inferences can form training instances when a model is trained using an evaluation of the model’s performance on that runtime instance (e.g., online training / learning). Example data types for the training instance and various tasks associated therewith are described throughout the present disclosure.

[0251] At 1604, example method 1600 can include processing, using one or more machine-learned models, the training instance to generate an output. The output can be directly obtained from the one or more machine-learned models or can be a downstream result of a chain of processing operations that includes an output of the one or more machine- learned models.

[0252] At 1606, example method 1600 can include receiving an evaluation signal associated with the output. The evaluation signal can be obtained using a loss function. Various determinations of loss can be used, such as mean squared error, likelihood loss, cross entropy loss, hinge loss, contrastive loss, or various other loss functions. The evaluation signal can be computed using known ground-truth labels (e.g., supervised learning), predicted or estimated labels (e.g., semi- or self-supervised learning), or without labels (e.g.,unsupervised learning). The evaluation signal can be a reward (e.g., for reinforcement learning). The reward can be computed using a machine-learned reward model configured to generate rewards based on output(s) received. The reward can be computed using feedback data describing human feedback on the output(s).

[0253] At 1608, example method 1600 can include updating the machine-learned model using the evaluation signal. For example, values for parameters of the machine-learned model(s) can be learned, in some embodiments, using various training or learning techniques, such as, for example, backwards propagation. For example, the evaluation signal can be backpropagated from the output (or another source of the evaluation signal) through the machine-learned model(s) to update one or more parameters of the model(s) (e.g., based on a gradient of the evaluation signal with respect to the parameter value(s)). For example, system(s) containing one or more machine-learned models can be trained in an end-to-end manner. Gradient descent techniques can be used to iteratively update the parameters over a number of training iterations. In some implementations, performing backwards propagation of errors can include performing truncated backpropagation through time. Example method 1600 can include implementing a number of generalization techniques (e.g., weight decays, dropouts, etc.) to improve the generalization capability of the models being trained.

[0254] In some implementations, example method 1600 can be implemented for training a machine-learned model from an initialized state to a fully trained state (e.g., when the model exhibits a desired performance profile, such as based on accuracy, precision, recall, etc.).

[0255] In some implementations, example method 1600 can be implemented for particular stages of a training procedure. For instance, in some implementations, example method 1600 can be implemented for pre-training a machine-learned model. Pre-training can include, for instance, large-scale training over potentially noisy data to achieve a broad base of performance levels across a variety of tasks / data types. In some implementations, example method 1600 can be implemented for fine-tuning a machine-learned model. Fine-tuning can include, for instance, smaller-scale training on higher-quality (e.g., labeled, curated, etc.) data. Fine-tuning can affect all or a portion of the parameters of a machine-learned model. For example, various portions of the machine-learned model can be “frozen” for certain training stages. For example, parameters associated with an embedding space can be “frozen” during fine-tuning (e.g., to retain information learned from a broader domain(s) than present in the fine-tuning dataset(s)). An example fine-tuning approach includes reinforcementlearning. Reinforcement learning can be based on user feedback on model performance during use. Example Machine-Learned Models

[0256] Figure 17 is a block diagram of an example processing flow for using machine-learned model(s) 1 to process input(s) 2 to generate output(s) 3.

[0257] Machine-learned model(s) 1 can be or include one or multiple machine- learned models or model components. Example machine-learned models can include neural networks (e.g., deep neural networks). Example machine-learned models can include non- linear models or linear models. Example machine-learned models can use other architectures in lieu of or in addition to neural networks. Example machine-learned models can include decision tree based models, support vector machines, hidden Markov models, Bayesian networks, linear regression models, k-means clustering models, etc.

[0258] Example neural networks can include feed-forward neural networks, recurrent neural networks (RNNs), including long short-term memory (LSTM) based recurrent neural networks, convolutional neural networks (CNNs), diffusion models, generative-adversarial networks, or other forms of neural networks. Example neural networks can be deep neural networks. Some example machine-learned models can leverage an attention mechanism such as self-attention. For example, some example machine-learned models can include multi- headed self-attention models.

[0259] Machine-learned model(s) 1 can include a single or multiple instances of the same model configured to operate on data from input(s) 2. Machine-learned model(s) 1 can include an ensemble of different models that can cooperatively interact to process data from input(s) 2. For example, machine-learned model(s) 1 can employ a mixture-of-experts structure. See, e.g., Zhou et al., Mixture-of-Experts with Expert Choice Routing, ARXIV:2202.09368v2 (Oct.14, 2022).

[0260] Input(s) 2 can generally include or otherwise represent various types of data. Input(s) 2 can include one type or many different types of data. Output(s) 3 can be data of the same type(s) or of different types of data as compared to input(s) 2. Output(s) 3 can include one type or many different types of data.

[0261] Example data types for input(s) 2 or output(s) 3 include natural language text data, software code data (e.g., source code, object code, machine code, or any other form of computer-readable instructions or programming languages), machine code data (e.g., binary code, assembly code, or other forms of machine-readable instructions that can be executeddirectly by a computer’s central processing unit), assembly code data (e.g., low-level programming languages that use symbolic representations of machine code instructions to program a processing unit), genetic data or other chemical or biochemical data, image data, audio data, audiovisual data, haptic data, biometric data, medical data, financial data, statistical data, geographical data, astronomical data, historical data, sensor data generally (e.g., digital or analog values, such as voltage or other absolute or relative level measurement values from a real or artificial input, such as from an audio sensor, light sensor, displacement sensor, etc.), and the like. Data can be raw or processed and can be in any format or schema.

[0262] In multimodal inputs 2 or outputs 3, example combinations of data types include image data and audio data, image data and natural language data, natural language data and software code data, image data and biometric data, sensor data and medical data, etc. It is to be understood that any combination of data types in an input 2 or an output 3 can be present.

[0263] An example input 2 can include one or multiple data types, such as the example data types noted above. An example output 3 can include one or multiple data types, such as the example data types noted above. The data type(s) of input 2 can be the same as or different from the data type(s) of output 3. It is to be understood that the example data types noted above are provided for illustrative purposes only. Data types contemplated within the scope of the present disclosure are not limited to those examples noted above. Example Machine-Learned Sequence Processing Models

[0264] Figure 18 is a block diagram of an example implementation of an example machine-learned model configured to process sequences of information. For instance, an example implementation of machine-learned model(s) 1 can include machine-learned sequence processing model(s) 4. An example system can pass input(s) 2 to sequence processing model(s) 4. Sequence processing model(s) 4 can include one or more machine- learned components. Sequence processing model(s) 4 can process the data from input(s) 2 to obtain an input sequence 5. Input sequence 5 can include one or more input elements 5-1, 5- 2, ... , 5-M, etc. obtained from input(s) 2. Sequence processing model 4 can process input sequence 5 using prediction layer(s) 6 to generate an output sequence 7. Output sequence 7 can include one or more output elements 7-1, 7-2, ... , 7-N, etc. generated based on input sequence 5. The system can generate output(s) 3 based on output sequence 7.

[0265] Sequence processing model(s) 4 can include one or multiple machine-learned model components configured to ingest, generate, or otherwise reason over sequences ofinformation. For example, some example sequence processing models in the text domain are referred to as “Large Language Models,” or LLMs. See, e.g., PaLM 2 Technical Report, GOOGLE, https: / / ai.google / static / documents / palm2techreport.pdf (n.d.). Other example sequence processing models can operate in other domains, such as image domains, see, e.g., Dosovitskiy et al., An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, ARXIV:2010.11929v2 (Jun.3, 2021), audio domains, see, e.g., Agostinelli et al., MusicLM: Generating Music From Text, ARXIV:2301.11325v1 (Jan.26, 2023), biochemical domains, see, e.g., Jumper et al., Highly accurate protein structure prediction with AlphaFold, 596 Nature 583 (Aug.26, 2021), by way of example. Sequence processing model(s) 4 can process one or multiple types of data simultaneously. Sequence processing model(s) 4 can include relatively large models (e.g., more parameters, computationally expensive, etc.), relatively small models (e.g., fewer parameters, computationally lightweight, etc.), or both.

[0266] In general, sequence processing model(s) 4 can obtain input sequence 5 using data from input(s) 2. For instance, input sequence 5 can include a representation of data from input(s) 2 in a format understood by sequence processing model(s) 4. One or more machine- learned components of sequence processing model(s) 4 can ingest the data from input(s) 2, parse the data into pieces compatible with the processing architectures of sequence processing model(s) 4 (e.g., via “tokenization”), and project the pieces into an input space associated with prediction layer(s) 6 (e.g., via “embedding”).

[0267] Sequence processing model(s) 4 can ingest the data from input(s) 2 and parse the data into a sequence of elements to obtain input sequence 5. For example, a portion of input data from input(s) 2 can be broken down into pieces that collectively represent the content of the portion of the input data. The pieces can provide the elements of the sequence.

[0268] Elements 5-1, 5-2, ... , 5-M can represent, in some cases, building blocks for capturing or expressing meaningful information in a particular data domain. For instance, the elements can describe “atomic units” across one or more domains. For example, for textual input source(s), the elements can correspond to groups of one or more words or sub-word components, such as sets of one or more characters.

[0269] For example, elements 5-1, 5-2, ... , 5-M can represent tokens obtained using a tokenizer. For instance, a tokenizer can process a given portion of an input source and output a series of tokens (e.g., corresponding to input elements 5-1, 5-2, ... , 5-M) that represent the portion of the input source. Various approaches to tokenization can be used. For instance, textual input source(s) can be tokenized using a byte-pair encoding (BPE) technique. See, e.g., Kudo et al., SentencePiece: A simple and language independent subwordtokenizer and detokenizer for Neural Text Processing, PROCEEDINGS OF THE 2018 CONFERENCE ON EMPIRICAL METHODS IN NATURAL LANGUAGE PROCESSING (System Demonstrations), pages 66–71 (October 31–November 4, 2018), https: / / aclanthology.org / D18-2012.pdf. Image-based input source(s) can be tokenized by extracting and serializing patches from an image.

[0270] In general, arbitrary data types can be serialized and processed into input sequence 5. It is to be understood that element(s) 5-1, 5-2, ... , 5-M depicted in Figure 18 can be the tokens or can be the embedded representations thereof.

[0271] Prediction layer(s) 6 can predict one or more output elements 7-1, 7-2, ... , 7- N based on the input elements. Prediction layer(s) 6 can include one or more machine-learned model architectures, such as one or more layers of learned parameters that manipulate and transform the input(s) to extract higher-order meaning from, and relationships between, input element(s) 5-1, 5-2, ... , 5-M. In this manner, for instance, example prediction layer(s) 6 can predict new output element(s) in view of the context provided by input sequence 5.

[0272] Prediction layer(s) 6 can evaluate associations between portions of input sequence 5 and a particular output element. These associations can inform a prediction of the likelihood that a particular output follows the input context. For example, consider the textual snippet, “The carpenter’s toolbox was small and heavy. It was full of ___.” Example prediction layer(s) 6 can identify that “It” refers back to “toolbox” by determining a relationship between the respective embeddings. Example prediction layer(s) 6 can also link “It” to the attributes of the toolbox, such as “small” and “heavy.” Based on these associations, prediction layer(s) 6 can, for instance, assign a higher probability to the word “nails” than to the word “sawdust.”

[0273] A transformer is an example architecture that can be used in prediction layer(s) 4. See, e.g., Vaswani et al., Attention Is All You Need, ARXIV:1706.03762v7 (Aug.2, 2023). A transformer is an example of a machine-learned model architecture that uses an attention mechanism to compute associations between items within a context window. The context window can include a sequence that contains input sequence 5 and potentially one or more output element(s) 7-1, 7-2, ... , 7-N. A transformer block can include one or more attention layer(s) and one or more post-attention layer(s) (e.g., feedforward layer(s), such as a multi-layer perceptron).

[0274] Prediction layer(s) 6 can include other machine-learned model architectures in addition to or in lieu of transformer-based architectures. For example, recurrent neural networks (RNNs) and long short-term memory (LSTM) models can also be used, as well asconvolutional neural networks (CNNs). In general, prediction layer(s) 6 can leverage various kinds of artificial neural networks that can understand or generate sequences of information.

[0275] Output sequence 7 can include or otherwise represent the same or different data types as input sequence 5. For instance, input sequence 5 can represent textual data, and output sequence 7 can represent textual data. Input sequence 5 can represent image, audio, or audiovisual data, and output sequence 7 can represent textual data (e.g., describing the image, audio, or audiovisual data). It is to be understood that prediction layer(s) 6, and any other interstitial model components of sequence processing model(s) 4, can be configured to receive a variety of data types in input sequence(s) 5 and output a variety of data types in output sequence(s) 7.

[0276] Output sequence 7 can have various relationships to input sequence 5. Output sequence 7 can be a continuation of input sequence 5. Output sequence 7 can be complementary to input sequence 5. Output sequence 7 can translate, transform, augment, or otherwise modify input sequence 5. Output sequence 7 can answer, evaluate, confirm, or otherwise respond to input sequence 5. Output sequence 7 can implement (or describe instructions for implementing) an instruction provided via input sequence 5.

[0277] Output sequence 7 can be generated autoregressively. For instance, for some applications, an output of one or more prediction layer(s) 6 can be passed through one or more output layers (e.g., softmax layer) to obtain a probability distribution over an output vocabulary (e.g., a textual or symbolic vocabulary) conditioned on a set of input elements in a context window. In this manner, for instance, output sequence 7 can be autoregressively generated by sampling a likely next output element, adding that element to the context window, and re-generating the probability distribution based on the updated context window, and sampling a likely next output element, and so forth.

[0278] Output sequence 7 can also be generated non-autoregressively. For instance, multiple output elements of output sequence 7 can be predicted together without explicit sequential conditioning on each other. See, e.g., Saharia et al., Non-Autoregressive Machine Translation with Latent Alignments, ARXIV:2004.07437v3 (Nov.16, 2020).

[0279] Output sequence 7 can include one or multiple portions or elements. In an example content generation configuration, output sequence 7 can include multiple elements corresponding to multiple portions of a generated output sequence (e.g., a textual sentence, values of a discretized waveform, computer code, etc.). In an example classification configuration, output sequence 7 can include a single element associated with a classification output. For instance, an output “vocabulary” can include a set of classes into which an inputsequence is to be classified. For instance, a vision transformer block can pass latent state information to a multilayer perceptron that outputs a likely class value associated with an input image.

[0280] Figure 19 is a block diagram of an example technique for populating an example input sequence 8. Input sequence 8 can include various functional elements that form part of the model infrastructure, such as an element 8-0 obtained from a task indicator 9 that signals to any model(s) that process input sequence 8 that a particular task is being performed (e.g., to help adapt a performance of the model(s) to that particular task). Input sequence 8 can include various data elements from different data modalities. For instance, an input modality 10-1 can include one modality of data. A data-to-sequence model 11-1 can process data from input modality 10-1 to project the data into a format compatible with input sequence 8 (e.g., one or more vectors dimensioned according to the dimensions of input sequence 8) to obtain elements 8-1, 8-2, 8-3. Another input modality 10-2 can include a different modality of data. A data-to-sequence model 11-2 can project data from input modality 10-2 into a format compatible with input sequence 8 to obtain elements 8-4, 8-5, 8- 6. Another input modality 10-3 can include yet another different modality of data. A data-to- sequence model 11-3 can project data from input modality 10-3 into a format compatible with input sequence 8 to obtain elements 8-7, 8-8, 8-9.

[0281] Input sequence 8 can be the same as or different from input sequence 5. Input sequence 8 can be a multimodal input sequence that contains elements that represent data from different modalities using a common dimensional representation. For instance, an embedding space can have P dimensions. Input sequence 8 can be configured to contain a plurality of elements that have P dimensions. In this manner, for instance, example implementations can facilitate information extraction and reasoning across diverse data modalities by projecting data into elements in the same embedding space for comparison, combination, or other computations therebetween.

[0282] For example, elements 8-0, ... , 8-9 can indicate particular locations within a multidimensional embedding space. Some elements can map to a set of discrete locations in the embedding space. For instance, elements that correspond to discrete members of a predetermined vocabulary of tokens can map to discrete locations in the embedding space that are associated with those tokens. Other elements can be continuously distributed across the embedding space. For instance, some data types can be broken down into continuously defined portions (e.g., image patches) that can be described using continuously distributed locations within the embedding space.

[0283] In some implementations, the expressive power of the embedding space may not be limited to meanings associated with any particular set of tokens or other building blocks. For example, a continuous embedding space can encode a spectrum of high-order information. An individual piece of information (e.g., a token) can map to a particular point in that space: for instance, a token for the word “dog” can be projected to an embedded value that points to a particular location in the embedding space associated with canine-related information. Similarly, an image patch of an image of a dog on grass can also be projected into the embedding space. In some implementations, the projection of the image of the dog can be similar to the projection of the word “dog” while also having similarity to a projection of the word “grass,” while potentially being different from both. In some implementations, the projection of the image patch may not exactly align with any single projection of a single word. In some implementations, the projection of the image patch can align with a combination of the projections of the words “dog” and “grass.” In this manner, for instance, a high-order embedding space can encode information that can be independent of data modalities in which the information is expressed.

[0284] Task indicator 9 can include a model or model component configured to identify a task being performed and inject, into input sequence 8, an input value represented by element 8-0 that signals which task is being performed. For instance, the input value can be provided as a data type associated with an input modality and projected along with that input modality (e.g., the input value can be a textual task label that is embedded along with other textual data in the input; the input value can be a pixel-based representation of a task that is embedded along with other image data in the input; etc.). The input value can be provided as a data type that differs from or is at least independent from other input(s). For instance, the input value represented by element 8-0 can be a learned within a continuous embedding space.

[0285] Input modalities 10-1, 10-2, and 10-3 can be associated with various different data types (e.g., as described above with respect to input(s) 2 and output(s) 3).

[0286] Data-to-sequence models 11-1, 11-2, and 11-3 can be the same or different from each other. Data-to-sequence models 11-1, 11-2, and 11-3 can be adapted to each respective input modality 10-1, 10-2, and 10-3. For example, a textual data-to-sequence model can subdivide a portion of input text and project the subdivisions into element(s) in input sequence 8 (e.g., elements 8-1, 8-2, 8-3, etc.). An image data-to-sequence model can subdivide an input image and project the subdivisions into element(s) in input sequence 8 (e.g., elements 8-4, 8-5, 8-6, etc.). An arbitrary datatype data-to-sequence model cansubdivide an input of that arbitrary datatype and project the subdivisions into element(s) in input sequence 8 (e.g., elements 8-7, 8-8, 8-9, etc.).

[0287] Data-to-sequence models 11-1, 11-2, and 11-3 can form part of machine- learned sequence processing model(s) 4. Data-to-sequence models 11-1, 11-2, and 11-3 can be jointly trained with or trained independently from machine-learned sequence processing model(s) 4. Data-to-sequence models 11-1, 11-2, and 11-3 can be trained end-to-end with machine-learned sequence processing model(s) 4. Example Machine-Learned Model Development Platform

[0288] Figure 20 is a block diagram of an example model development platform 12 that can facilitate creation, adaptation, and refinement of example machine-learned models (e.g., machine-learned model(s) 1, sequence processing model(s) 4, etc.). Model development platform 12 can provide a number of different toolkits that developer systems can employ in the development of new or adapted machine-learned models.

[0289] Model development platform 12 can provide one or more model libraries 13 containing building blocks for new models. Model libraries 13 can include one or more pre- trained foundational models 13-1, which can provide a backbone of processing power across various tasks. Model libraries 13 can include one or more pre-trained expert models 13-2, which can be focused on performance in particular domains of expertise. Model libraries 13 can include various model primitives 13-3, which can provide low-level architectures or components (optionally pre-trained), which can be assembled in various arrangements as desired.

[0290] Model development platform 12 can receive selections of various model components 14. Model development platform 12 can pass selected model components 14 to a workbench 15 that combines selected model components 14 into a development model 16.

[0291] Workbench 15 can facilitate further refinement and adaptation of development model 16 by leveraging a number of different toolkits integrated with model development platform 12. For example, workbench 15 can facilitate alignment of the development model 16 with a desired performance profile on various tasks using a model alignment toolkit 17.

[0292] Model alignment toolkit 17 can provide a number of tools for causing development model 16 to generate outputs aligned with desired behavioral characteristics. Alignment can include increasing an accuracy, precision, recall, etc. of model outputs. Alignment can include enforcing output styles, schema, or other preferential characteristics of model outputs. Alignment can be general or domain-specific. For instance, a pre-trainedfoundational model 13-1 can begin with an initial level of performance across multiple domains. Alignment of the pre-trained foundational model 13-1 can include improving a performance in a particular domain of information or tasks (e.g., even at the expense of performance in another domain of information or tasks).

[0293] Model alignment toolkit 17 can integrate one or more dataset(s) 17-1 for aligning development model 16. Curated dataset(s) 17-1 can include labeled or unlabeled training data. Dataset(s) 17-1 can be obtained from public domain datasets. Dataset(s) 17-1 can be obtained from private datasets associated with one or more developer system(s) for the alignment of bespoke machine-learned model(s) customized for private use-cases.

[0294] Pre-training pipelines 17-2 can include a machine-learned model training workflow configured to update development model 16 over large-scale, potentially noisy datasets. For example, pre-training can leverage unsupervised learning techniques (e.g., de- noising, etc.) to process large numbers of training instances to update model parameters from an initialized state and achieve a desired baseline performance. Pre-training pipelines 17-2 can leverage unlabeled datasets in dataset(s) 17-1 to perform pre-training. Workbench 15 can implement a pre-training pipeline 17-2 to pre-train development model 16.

[0295] Fine-tuning pipelines 17-3 can include a machine-learned model training workflow configured to refine the model parameters of development model 16 with higher- quality data. Fine-tuning pipelines 17-3 can update development model 16 by conducting supervised training with labeled dataset(s) in dataset(s) 17-1. Fine-tuning pipelines 17-3 can update development model 16 by conducting reinforcement learning using reward signals from user feedback signals. Workbench 15 can implement a fine-tuning pipeline 17-3 to fine- tune development model 16.

[0296] Prompt libraries 17-4 can include sets of inputs configured to induce behavior aligned with desired performance criteria. Prompt libraries 17-4 can include few-shot prompts (e.g., inputs providing examples of desired model outputs for prepending to a desired runtime query), chain-of-thought prompts (e.g., inputs providing step-by-step reasoning within the exemplars to facilitate thorough reasoning by the model), and the like.

[0297] Example prompts can be retrieved from an available repository of prompt libraries 17-4. Example prompts can be contributed by one or more developer systems using workbench 15.

[0298] In some implementations, pre-trained or fine-tuned models can achieve satisfactory performance without exemplars in the inputs. For instance, zero-shot prompts caninclude inputs that lack exemplars. Zero-shot prompts can be within a domain within a training dataset or outside of the training domain(s).

[0299] Prompt libraries 17-4 can include one or more prompt engineering tools. Prompt engineering tools can provide workflows for retrieving or learning optimized prompt values. Prompt engineering tools can facilitate directly learning prompt values (e.g., input element values) based one or more training iterations. Workbench 15 can implement prompt engineering tools in development model 16.

[0300] Prompt libraries 17-4 can include pipelines for prompt generation. For example, inputs can be generated using development model 16 itself or other machine- learned models. In this manner, for instance, a first model can process information about a task and output a input for a second model to process in order to perform a step of the task. The second model can be the same as or different from the first model. Workbench 15 can implement prompt generation pipelines in development model 16.

[0301] Prompt libraries 17-4 can include pipelines for context injection. For instance, a performance of development model 16 on a particular task can improve if provided with additional context for performing the task. Prompt libraries 17-4 can include software components configured to identify desired context, retrieve the context from an external source (e.g., a database, a sensor, etc.), and add the context to the input prompt. Workbench 15 can implement context injection pipelines in development model 16.

[0302] Although various training examples described herein with respect to model development platform 12 refer to “pre-training” and “fine-tuning,” it is to be understood that model alignment toolkit 17 can generally support a wide variety of training techniques adapted for training a wide variety of machine-learned models. Example training techniques can correspond to the example training method 1600 described above.

[0303] Model development platform 12 can include a model plugin toolkit 18. Model plugin toolkit 18 can include a variety of tools configured for augmenting the functionality of a machine-learned model by integrating the machine-learned model with other systems, devices, and software components. For instance, a machine-learned model can use tools to increase performance quality where appropriate. For instance, deterministic tasks can be offloaded to dedicated tools in lieu of probabilistically performing the task with an increased risk of error. For instance, instead of autoregressively predicting the solution to a system of equations, a machine-learned model can recognize a tool to call for obtaining the solution and pass the system of equations to the appropriate tool. The tool can be a traditional system of equations solver that can operate deterministically to resolve the system of equations. Theoutput of the tool can be returned in response to the original query. In this manner, tool use can allow some example models to focus on the strengths of machine-learned models—e.g., understanding an intent in an unstructured request for a task—while augmenting the performance of the model by offloading certain tasks to a more focused tool for rote application of deterministic algorithms to a well-defined problem.

[0304] Model plugin toolkit 18 can include validation tools 18-1. Validation tools 18- 1 can include tools that can parse and confirm output(s) of a machine-learned model. Validation tools 18-1 can include engineered heuristics that establish certain thresholds applied to model outputs. For example, validation tools 18-1 can ground the outputs of machine-learned models to structured data sources (e.g., to mitigate “hallucinations”).

[0305] Model plugin toolkit 18 can include tooling packages 18-2 for implementing one or more tools that can include scripts or other executable code that can be executed alongside development model 16. Tooling packages 18-2 can include one or more inputs configured to cause machine-learned model(s) to implement the tools (e.g., few-shot prompts that induce a model to output tool calls in the proper syntax, etc.). Tooling packages 18-2 can include, for instance, fine-tuning training data for training a model to use a tool.

[0306] Model plugin toolkit 18 can include interfaces for calling external application programming interfaces (APIs) 18-3. For instance, in addition to or in lieu of implementing tool calls or tool code directly with development model 16, development model 16 can be aligned to output instruction that initiate API calls to send or obtain data via external systems.

[0307] Model plugin toolkit 18 can integrate with prompt libraries 17-4 to build a catalog of available tools for use with development model 16. For instance, a model can receive, in an input, a catalog of available tools, and the model can generate an output that selects a tool from the available tools and initiates a tool call for using the tool.

[0308] Model development platform 12 can include a computational optimization toolkit 19 for optimizing a computational performance of development model 16. For instance, tools for model compression 19-1 can allow development model 16 to be reduced in size while maintaining a desired level of performance. For instance, model compression 19-1 can include quantization workflows, weight pruning and sparsification techniques, etc. Tools for hardware acceleration 19-2 can facilitate the configuration of the model storage and execution formats to operate optimally on different hardware resources. For instance, hardware acceleration 19-2 can include tools for optimally sharding models for distributed processing over multiple processing units for increased bandwidth, lower unified memory requirements, etc. Tools for distillation 19-3 can provide for the training of lighter-weightmodels based on the knowledge encoded in development model 16. For instance, development model 16 can be a highly performant, large machine-learned model optimized using model development platform 12. To obtain a lightweight model for running in resource-constrained environments, a smaller model can be a “student model” that learns to imitate development model 16 as a “teacher model.” In this manner, for instance, the investment in learning the parameters and configurations of development model 16 can be efficiently transferred to a smaller model for more efficient inference.

[0309] Workbench 15 can implement one, multiple, or none of the toolkits implemented in model development platform 12. Workbench 15 can output an output model 20 based on development model 16. Output model 20 can be a deployment version of development model 16. Output model 20 can be a development or training checkpoint of development model 16. Output model 20 can be a distilled, compressed, or otherwise optimized version of development model 16.

[0310] Figure 21 is a block diagram of an example training flow for training a machine-learned development model 16. One or more portion(s) of the example training flow can be implemented by a computing system that includes one or more computing devices such as, for example, computing systems described with reference to the other figures. Each respective portion of the example training flow can be performed by any (or any combination) of one or more computing devices. Moreover, one or more portion(s) of the example training flow can be implemented on the hardware components of the device(s) described herein, for example, to train one or more systems or models. FIG.21 depicts elements performed in a particular order for purposes of illustration and discussion. Those of ordinary skill in the art, using the disclosures provided herein, will understand that the elements of any of the methods discussed herein can be adapted, rearranged, expanded, omitted, combined, or modified in various ways without deviating from the scope of the present disclosure. FIG.21 is described with reference to elements / terms described with respect to other systems and figures for exemplary illustrated purposes and is not meant to be limiting. One or more portions of the example training flow can be performed additionally, or alternatively, by other systems.

[0311] Initially, development model 16 can persist in an initial state as an initialized model 21. Development model 16 can be initialized with weight values. Initial weight values can be random or based on an initialization schema. Initial weight values can be based on prior pre-training for the same or for a different model.

[0312] Initialized model 21 can undergo pre-training in a pre-training stage 22. Pre- training stage 22 can be implemented using one or more pre-training pipelines 17-2 over data from dataset(s) 17-1. Pre-training can be omitted, for example, if initialized model 21 is already pre-trained (e.g., development model 16 contains, is, or is based on a pre-trained foundational model or an expert model).

[0313] Pre-trained model 23 can then be a new version of development model 16, which can persist as development model 16 or as a new development model. Pre-trained model 23 can be the initial state if development model 16 was already pre-trained. Pre-trained model 23 can undergo fine-tuning in a fine-tuning stage 24. Fine-tuning stage 24 can be implemented using one or more fine-tuning pipelines 17-3 over data from dataset(s) 17-1. Fine-tuning can be omitted, for example, if a pre-trained model as satisfactory performance, if the model was already fine-tuned, or if other tuning approaches are preferred.

[0314] Fine-tuned model 29 can then be a new version of development model 16, which can persist as development model 16 or as a new development model. Fine-tuned model 29 can be the initial state if development model 16 was already fine-tuned. Fine-tuned model 29 can undergo refinement with user feedback 26. For instance, refinement with user feedback 26 can include reinforcement learning, optionally based on human feedback from human users of fine-tuned model 25. As reinforcement learning can be a form of fine-tuning, it is to be understood that fine-tuning stage 24 can subsume the stage for refining with user feedback 26. Refinement with user feedback 26 can produce a refined model 27. Refined model 27 can be output to downstream system(s) 28 for deployment or further development.

[0315] In some implementations, computational optimization operations can be applied before, during, or after each stage. For instance, initialized model 21 can undergo computational optimization 29-1 (e.g., using computational optimization toolkit 19) before pre-training stage 22. Pre-trained model 23 can undergo computational optimization 29-2 (e.g., using computational optimization toolkit 19) before fine-tuning stage 24. Fine-tuned model 25 can undergo computational optimization 29-3 (e.g., using computational optimization toolkit 19) before refinement with user feedback 26. Refined model 27 can undergo computational optimization 29-4 (e.g., using computational optimization toolkit 19) before output to downstream system(s) 28. Computational optimization(s) 29-1, ... , 29-4 can all be the same, all be different, or include at least some different optimization techniques.Example Machine-Learned Model Inference System

[0316] Figure 22 is a block diagram of an inference system for operating one or more machine-learned model(s) 1 to perform inference (e.g., for training, for deployment, etc.). A model host 31 can receive machine-learned model(s) 1. Model host 31 can host one or more model instance(s) 31-1, which can be one or multiple instances of one or multiple models. Model host 31 can host model instance(s) 31-1 using available compute resources 31-2 associated with model host 31.

[0317] Model host 31 can perform inference on behalf of one or more client(s) 32. Client(s) 32 can transmit an input request 33 to model host 31. Using input request 33, model host 31 can obtain input(s) 2 for input to machine-learned model(s) 1. Machine-learned model(s) 1 can process input(s) 2 to generate output(s) 3. Using output(s) 3, model host 31 can return an output payload 34 for responding to input request 33 from client(s) 32. Output payload 34 can include or be based on output(s) 3.

[0318] Model host 31 can leverage various other resources and tools to augment the inference task. For instance, model host 31 can communicate with tool interfaces 35 to facilitate tool use by model instance(s) 31-1. Tool interfaces 35 can include local or remote APIs. Tool interfaces 35 can include integrated scripts or other software functionality. Model host 31 can engage online learning interface(s) 36 to facilitate ongoing improvements to machine-learned model(s) 1. For instance, online learning interface(s) 36 can be used within reinforcement learning loops to retrieve user feedback on inferences served by model host 31. Model host 31 can access runtime data source(s) 37 for augmenting input(s) 2 with additional contextual information. For instance, runtime data source(s) 37 can include a knowledge graph 37-1 that facilitates structured information retrieval for information associated with input request(s) 33 (e.g., a search engine service). Runtime data source(s) 37 can include public or private, external or local database(s) 37-2 that can store information associated with input request(s) 33 for augmenting input(s) 2. Runtime data source(s) 37 can include account data 37-3 which can be retrieved in association with a user account corresponding to a client 32 for customizing the behavior of model host 31 accordingly.

[0319] Model host 31 can be implemented by one or multiple computing devices or systems. Client(s) 2 can be implemented by one or multiple computing devices or systems, which can include computing devices or systems shared with model host 31.

[0320] For example, model host 31 can operate on a server system that provides a machine-learning service to client device(s) that operate client(s) 32 (e.g., over a local or wide-area network). Client device(s) can be end-user devices used by individuals. Clientdevice(s) can be server systems that operate client(s) 32 to provide various functionality as a service to downstream end-user devices.

[0321] In some implementations, model host 31 can operate on a same device or system as client(s) 32. Model host 31 can be a machine-learning service that runs on-device to provide machine-learning functionality to one or multiple applications operating on a client device, which can include an application implementing client(s) 32. Model host 31 can be a part of a same application as client(s) 32. For instance, model host 31 can be a subroutine or method implemented by one part of an application, and client(s) 32 can be another subroutine or method that engages model host 31 to perform inference functions within the application. It is to be understood that model host 31 and client(s) 32 can have various different configurations.

[0322] Model instance(s) 31-1 can include one or more machine-learned models that are available for performing inference. Model instance(s) 31-1 can include weights or other model components that are stored on in persistent storage, temporarily cached, or loaded into high-speed memory. Model instance(s) 31-1 can include multiple instance(s) of the same model (e.g., for parallel execution of more requests on the same model). Model instance(s) 31-1 can include instance(s) of different model(s). Model instance(s) 31-1 can include cached intermediate states of active or inactive model(s) used to accelerate inference of those models. For instance, an inference session with a particular model may generate significant amounts of computational results that can be re-used for future inference runs (e.g., using a KV cache for transformer-based models). These computational results can be saved in association with that inference session so that session can be executed more efficiently when resumed.

[0323] Compute resource(s) 31-2 can include one or more processors (central processing units, graphical processing units, tensor processing units, machine-learning accelerators, etc.) connected to one or more memory devices. Compute resource(s) 31-2 can include a dynamic pool of available resources shared with other processes. Compute resource(s) 31-2 can include memory devices large enough to fit an entire model instance in a single memory instance. Compute resource(s) 31-2 can also shard model instance(s) across multiple memory devices (e.g., using data parallelization or tensor parallelization, etc.). This can be done to increase parallelization or to execute a large model using multiple memory devices which individually might not be able to fit the entire model into memory.

[0324] Input request 33 can include data for input(s) 2. Model host 31 can process input request 33 to obtain input(s) 2. Input(s) 2 can be obtained directly from input request 33or can be retrieved using input request 33. Input request 33 can be submitted to model host 31 via an API.

[0325] Model host 31 can perform inference over batches of input requests 33 in parallel. For instance, a model instance 31-1 can be configured with an input structure that has a batch dimension. Separate input(s) 2 can be distributed across the batch dimension (e.g., rows of an array). The separate input(s) 2 can include completely different contexts. The separate input(s) 2 can be multiple inference steps of the same task. The separate input(s) 2 can be staggered in an input structure, such that any given inference cycle can be operating on different portions of the respective input(s) 2. In this manner, for instance, model host 31 can perform inference on the batch in parallel, such that output(s) 3 can also contain the batch dimension and return the inference results for the batched input(s) 2 in parallel. In this manner, for instance, batches of input request(s) 33 can be processed in parallel for higher throughput of output payload(s) 34.

[0326] Output payload 34 can include or be based on output(s) 3 from machine- learned model(s) 1. Model host 31 can process output(s) 3 to obtain output payload 34. This can include chaining multiple rounds of inference (e.g., iteratively, recursively, across the same model(s) or different model(s)) to arrive at a final output for a task to be returned in output payload 34. Output payload 34 can be transmitted to client(s) 32 via an API.

[0327] Online learning interface(s) 36 can facilitate reinforcement learning of machine-learned model(s) 1. Online learning interface(s) 36 can facilitate reinforcement learning with human feedback (RLHF). Online learning interface(s) 36 can facilitate federated learning of machine-learned model(s) 1.

[0328] Model host 31 can execute machine-learned model(s) 1 to perform inference for various tasks using various types of data. For example, various different input(s) 2 and output(s) 3 can be used for various different tasks. In some implementations, input(s) 2 can be or otherwise represent image data. Machine-learned model(s) 1 can process the image data to generate an output. As an example, machine-learned model(s) 1 can process the image data to generate an image recognition output (e.g., a recognition of the image data, a latent embedding of the image data, an encoded representation of the image data, a hash of the image data, etc.). As another example, machine-learned model(s) 1 can process the image data to generate an image segmentation output. As another example, machine-learned model(s) 1 can process the image data to generate an image classification output. As another example, machine-learned model(s) 1 can process the image data to generate an image data modification output (e.g., an alteration of the image data, etc.). As another example, machine-learned model(s) 1 can process the image data to generate an encoded image data output (e.g., an encoded and / or compressed representation of the image data, etc.). As another example, machine-learned model(s) 1 can process the image data to generate an upscaled image data output. As another example, machine-learned model(s) 1 can process the image data to generate a prediction output.

[0329] In some implementations, the task is a computer vision task. In some cases, input(s) 2 includes pixel data for one or more images and the task is an image processing task. For example, the image processing task can be image classification, where the output is a set of scores, each score corresponding to a different object class and representing the likelihood that the one or more images depict an object belonging to the object class. The image processing task may be object detection, where the image processing output identifies one or more regions in the one or more images and, for each region, a likelihood that region depicts an object of interest. As another example, the image processing task can be image segmentation, where the image processing output defines, for each pixel in the one or more images, a respective likelihood for each category in a predetermined set of categories. For example, the set of categories can be foreground and background. As another example, the set of categories can be object classes. As another example, the image processing task can be depth estimation, where the image processing output defines, for each pixel in the one or more images, a respective depth value. As another example, the image processing task can be motion estimation, where the network input includes multiple images, and the image processing output defines, for each pixel of one of the input images, a motion of the scene depicted at the pixel between the images in the network input.

[0330] In some implementations, input(s) 2 can be or otherwise represent natural language data. Machine-learned model(s) 1 can process the natural language data to generate an output. As an example, machine-learned model(s) 1 can process the natural language data to generate a language encoding output. As another example, machine-learned model(s) 1 can process the natural language data to generate a latent text embedding output. As another example, machine-learned model(s) 1 can process the natural language data to generate a translation output. As another example, machine-learned model(s) 1 can process the natural language data to generate a classification output. As another example, machine-learned model(s) 1 can process the natural language data to generate a textual segmentation output. As another example, machine-learned model(s) 1 can process the natural language data to generate a semantic intent output. As another example, machine-learned model(s) 1 can process the natural language data to generate an upscaled text or natural language output(e.g., text or natural language data that is higher quality than the input text or natural language, etc.). As another example, machine-learned model(s) 1 can process the natural language data to generate a prediction output (e.g., one or more predicted next portions of natural language content).

[0331] In some implementations, input(s) 2 can be or otherwise represent speech data (e.g., data describing spoken natural language, such as audio data, textual data, etc.). Machine-learned model(s) 1 can process the speech data to generate an output. As an example, machine-learned model(s) 1 can process the speech data to generate a speech recognition output. As another example, machine-learned model(s) 1 can process the speech data to generate a speech translation output. As another example, machine-learned model(s) 1 can process the speech data to generate a latent embedding output. As another example, machine-learned model(s) 1 can process the speech data to generate an encoded speech output (e.g., an encoded and / or compressed representation of the speech data, etc.). As another example, machine-learned model(s) 1 can process the speech data to generate an upscaled speech output (e.g., speech data that is higher quality than the input speech data, etc.). As another example, machine-learned model(s) 1 can process the speech data to generate a textual representation output (e.g., a textual representation of the input speech data, etc.). As another example, machine-learned model(s) 1 can process the speech data to generate a prediction output.

[0332] In some implementations, input(s) 2 can be or otherwise represent latent encoding data (e.g., a latent space representation of an input, etc.). Machine-learned model(s) 1 can process the latent encoding data to generate an output. As an example, machine- learned model(s) 1 can process the latent encoding data to generate a recognition output. As another example, machine-learned model(s) 1 can process the latent encoding data to generate a reconstruction output. As another example, machine-learned model(s) 1 can process the latent encoding data to generate a search output. As another example, machine- learned model(s) 1 can process the latent encoding data to generate a reclustering output. As another example, machine-learned model(s) 1 can process the latent encoding data to generate a prediction output.

[0333] In some implementations, input(s) 2 can be or otherwise represent statistical data. Statistical data can be, represent, or otherwise include data computed and / or calculated from some other data source. Machine-learned model(s) 1 can process the statistical data to generate an output. As an example, machine-learned model(s) 1 can process the statistical data to generate a recognition output. As another example, machine-learned model(s) 1 canprocess the statistical data to generate a prediction output. As another example, machine- learned model(s) 1 can process the statistical data to generate a classification output. As another example, machine-learned model(s) 1 can process the statistical data to generate a segmentation output. As another example, machine-learned model(s) 1 can process the statistical data to generate a visualization output. As another example, machine-learned model(s) 1 can process the statistical data to generate a diagnostic output.

[0334] In some implementations, input(s) 2 can be or otherwise represent sensor data. Machine-learned model(s) 1 can process the sensor data to generate an output. As an example, machine-learned model(s) 1 can process the sensor data to generate a recognition output. As another example, machine-learned model(s) 1 can process the sensor data to generate a prediction output. As another example, machine-learned model(s) 1 can process the sensor data to generate a classification output. As another example, machine-learned model(s) 1 can process the sensor data to generate a segmentation output. As another example, machine-learned model(s) 1 can process the sensor data to generate a visualization output. As another example, machine-learned model(s) 1 can process the sensor data to generate a diagnostic output. As another example, machine-learned model(s) 1 can process the sensor data to generate a detection output.

[0335] In some implementations, machine-learned model(s) 1 can be configured to perform a task that includes encoding input data for reliable and / or efficient transmission or storage (and / or corresponding decoding). For example, the task may be an audio compression task. The input may include audio data and the output may comprise compressed audio data. In another example, the input includes visual data (e.g. one or more images or videos), the output comprises compressed visual data, and the task is a visual data compression task. In another example, the task may comprise generating an embedding for input data (e.g. input audio or visual data). In some cases, the input includes audio data representing a spoken utterance and the task is a speech recognition task. The output may comprise a text output which is mapped to the spoken utterance. In some cases, the task comprises encrypting or decrypting input data. In some cases, the task comprises a microprocessor performance task, such as branch prediction or memory address translation.

[0336] In some implementations, the task is a generative task, and machine-learned model(s) 1 can be configured to output content generated in view of input(s) 2. For instance, input(s) 2 can be or otherwise represent data of one or more modalities that encodes context for generating additional content.

[0337] In some implementations, the task can be a text completion task. Machine- learned model(s) 1 can be configured to process input(s) 2 that represent textual data and to generate output(s) 3 that represent additional textual data that completes a textual sequence that includes input(s) 2. For instance, machine-learned model(s) 1 can be configured to generate output(s) 3 to complete a sentence, paragraph, or portion of text that follows from a portion of text represented by input(s) 2.

[0338] In some implementations, the task can be an instruction following task. Machine-learned model(s) 1 can be configured to process input(s) 2 that represent instructions to perform a function and to generate output(s) 3 that advance a goal of satisfying the instruction function (e.g., at least a step of a multi-step procedure to perform the function). Output(s) 3 can represent data of the same or of a different modality as input(s) 2. For instance, input(s) 2 can represent textual data (e.g., natural language instructions for a task to be performed) and machine-learned model(s) 1 can process input(s) 2 to generate output(s) 3 that represent textual data responsive to the instructions (e.g., natural language responses, programming language responses, machine language responses, etc.). Input(s) 2 can represent image data (e.g., image-based instructions for a task to be performed, optionally accompanied by textual instructions) and machine-learned model(s) 1 can process input(s) 2 to generate output(s) 3 that represent textual data responsive to the instructions (e.g., natural language responses, programming language responses, machine language responses, etc.). One or more output(s) 3 can be iteratively or recursively generated to sequentially process and accomplish steps toward accomplishing the requested functionality. For instance, an initial output can be executed by an external system or be processed by machine-learned model(s) 1 to complete an initial step of performing a function. Multiple steps can be performed, with a final output being obtained that is responsive to the initial instructions.

[0339] In some implementations, the task can be a question answering task. Machine- learned model(s) 1 can be configured to process input(s) 2 that represent a question to answer and to generate output(s) 3 that advance a goal of returning an answer to the question (e.g., at least a step of a multi-step procedure to perform the function). Output(s) 3 can represent data of the same or of a different modality as input(s) 2. For instance, input(s) 2 can represent textual data (e.g., natural language instructions for a task to be performed) and machine- learned model(s) 1 can process input(s) 2 to generate output(s) 3 that represent textual data responsive to the question (e.g., natural language responses, programming language responses, machine language responses, etc.). Input(s) 2 can represent image data (e.g., image-based instructions for a task to be performed, optionally accompanied by textualinstructions) and machine-learned model(s) 1 can process input(s) 2 to generate output(s) 3 that represent textual data responsive to the question (e.g., natural language responses, programming language responses, machine language responses, etc.). One or more output(s) 3 can be iteratively or recursively generated to sequentially process and accomplish steps toward answering the question. For instance, an initial output can be executed by an external system or be processed by machine-learned model(s) 1 to complete an initial step of obtaining an answer to the question (e.g., querying a database, performing a computation, executing a script, etc.). Multiple steps can be performed, with a final output being obtained that is responsive to the question.

[0340] In some implementations, the task can be an image generation task. Machine- learned model(s) 1 can be configured to process input(s) 2 that represent context regarding a desired portion of image content. The context can include text data, image data, audio data, etc. Machine-learned model(s) 1 can be configured to generate output(s) 3 that represent image data that depicts imagery related to the context. For instance, machine-learned model(s) 1 can be configured to generate pixel data of an image. Values for channel(s) associated with the pixels in the pixel data can be selected based on the context (e.g., based on a probability determined based on the context).

[0341] In some implementations, the task can be an audio generation task. Machine- learned model(s) 1 can be configured to process input(s) 2 that represent context regarding a desired portion of audio content. The context can include text data, image data, audio data, etc. Machine-learned model(s) 1 can be configured to generate output(s) 3 that represent audio data related to the context. For instance, machine-learned model(s) 1 can be configured to generate waveform data in the form of an image (e.g., a spectrogram). Values for channel(s) associated with pixels of the image can be selected based on the context. Machine- learned model(s) 1 can be configured to generate waveform data in the form of a sequence of discrete samples of a continuous waveform. Values of the sequence can be selected based on the context (e.g., based on a probability determined based on the context).

[0342] In some implementations, the task can be a data generation task. Machine- learned model(s) 1 can be configured to process input(s) 2 that represent context regarding a desired portion of data (e.g., data from various data domains, such as sensor data, image data, multimodal data, statistical data, etc.). The desired data can be, for instance, synthetic data for training other machine-learned models. The context can include arbitrary data type(s). Machine-learned model(s) 1 can be configured to generate output(s) 3 that represent data that aligns with the desired data. For instance, machine-learned model(s) 1 can be configured togenerate data values for populating a dataset. Values for the data object(s) can be selected based on the context (e.g., based on a probability determined based on the context). Example Computing Systems and Devices

[0343] Figure 23 is a block diagram of an example networked computing system that can perform aspects of example implementations of the present disclosure. The system can include a number of computing devices and systems that are communicatively coupled over a network 49. An example computing device 50 is described to provide an example of a computing device that can perform any aspect of the present disclosure (e.g., implementing model host 31, client(s) 32, or both). An example server computing system 60 is described as an example of a server computing system that can perform any aspect of the present disclosure (e.g., implementing model host 31, client(s) 32, or both). Computing device 50 and server computing system(s) 60 can cooperatively interact (e.g., over network 49) to perform any aspect of the present disclosure (e.g., implementing model host 31, client(s) 32, or both). Model development platform system 70 is an example system that can host or serve model development platform(s) 12 for development of machine-learned models. Third-party system(s) 80 are example system(s) with which any of computing device 50, server computing system(s) 60, or model development platform system(s) 70 can interact in the performance of various aspects of the present disclosure (e.g., engaging third-party tools, accessing third-party databases or other resources, etc.).

[0344] Network 49 can be any type of communications network, such as a local area network (e.g., intranet), wide area network (e.g., Internet), or some combination thereof and can include any number of wired or wireless links. In general, communication over network 49 can be carried via any type of wired or wireless connection, using a wide variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), or protection schemes (e.g., VPN, secure HTTP, SSL). Network 49 can also be implemented via a system bus. For instance, one or more devices or systems of Figure 23 can be co-located with, contained by, or otherwise integrated into one or more other devices or systems.

[0345] Computing device 50 can be any type of computing device, such as, for example, a personal computing device (e.g., laptop or desktop), a mobile computing device (e.g., smartphone or tablet), a gaming console or controller, a wearable computing device, an embedded computing device, a server computing device, a virtual machine operating on a host device, or any other type of computing device. Computing device 50 can be a clientcomputing device. Computing device 50 can be an end-user computing device. Computing device 50 can be a computing device of a service provided that provides a service to an end user (who may use another computing device to interact with computing device 50).

[0346] Computing device 50 can include one or more processors 51 and a memory 52. Processor(s) 51 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. Memory 52 can include one or more non-transitory computer-readable storage media, such as HBM, RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. Memory 52 can store data 53 and instructions 54 which can be executed by processor(s) 51 to cause computing device 50 to perform operations. The operations can implement any one or multiple features described herein. The operations can implement example methods and techniques described herein.

[0347] Computing device 50 can also include one or more input components that receive user input. For example, a user input component can be a touch-sensitive component (e.g., a touch-sensitive display screen or a touch pad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component can serve to implement a virtual keyboard. Other example user input components include a microphone, camera, LIDAR, a physical keyboard or other buttons, or other means by which a user can provide user input.

[0348] Computing device 50 can store or include one or more machine-learned models 55. Machine-learned models 55 can include one or more machine-learned model(s) 1, such as a sequence processing model 4. Machine-learned models 55 can include one or multiple model instance(s) 31-1. Machine-learned model(s) 55 can be received from server computing system(s) 60, model development platform system 70, third party system(s) 80 (e.g., an application distribution platform), or developed locally on computing device 50. Machine-learned model(s) 55 can be loaded into memory 52 and used or otherwise implemented by processor(s) 51. Computing device 50 can implement multiple parallel instances of machine-learned model(s) 55.

[0349] Server computing system(s) 60 can include one or more processors 61 and a memory 62. Processor(s) 61 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. Memory 62 can include one or more non-transitory computer-readable storage media, such as HBM, RAM, ROM,EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. Memory 62 can store data 63 and instructions 64 which can be executed by processor(s) 61 to cause server computing system(s) 60 to perform operations. The operations can implement any one or multiple features described herein. The operations can implement example methods and techniques described herein.

[0350] In some implementations, server computing system 60 includes or is otherwise implemented by one or multiple server computing devices. In instances in which server computing system 60 includes multiple server computing devices, such server computing devices can operate according to sequential computing architectures, parallel computing architectures, or some combination thereof.

[0351] Server computing system 60 can store or otherwise include one or more machine-learned models 65. Machine-learned model(s) 65 can be the same as or different from machine-learned model(s) 55. Machine-learned models 65 can include one or more machine-learned model(s) 1, such as a sequence processing model 4. Machine-learned models 65 can include one or multiple model instance(s) 31-1. Machine-learned model(s) 65 can be received from computing device 50, model development platform system 70, third party system(s) 80, or developed locally on server computing system(s) 60. Machine-learned model(s) 65 can be loaded into memory 62 and used or otherwise implemented by processor(s) 61. Server computing system(s) 60 can implement multiple parallel instances of machine-learned model(s) 65.

[0352] In an example configuration, machine-learned models 65 can be included in or otherwise stored and implemented by server computing system 60 to establish a client-server relationship with computing device 50 for serving model inferences. For instance, server computing system(s) 60 can implement model host 31 on behalf of client(s) 32 on computing device 50. For instance, machine-learned models 65 can be implemented by server computing system 60 as a portion of a web service (e.g., remote machine-learned model hosting service, such as an online interface for performing machine-learned model operations over a network on server computing system(s) 60). For instance, server computing system(s) 60 can communicate with computing device 50 over a local intranet or internet connection. For instance, computing device 50 can be a workstation or endpoint in communication with server computing system(s) 60, with implementation of machine-learned models 65 being managed by server computing system(s) 60 to remotely perform inference (e.g., for runtime or training operations), with output(s) returned (e.g., cast, streamed, etc.) to computing device50. Machine-learned models 65 can work cooperatively or interoperatively with machine- learned models 55 on computing device 50 to perform various tasks.

[0353] Model development platform system(s) 70 can include one or more processors 71 and a memory 72. Processor(s) 71 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. Memory 72 can include one or more non-transitory computer-readable storage media, such as HBM, RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. Memory 72 can store data 73 and instructions 74 which can be executed by processor(s) 71 to cause model development platform system(s) 70 to perform operations. The operations can implement any one or multiple features described herein. The operations can implement example methods and techniques described herein. Example operations include the functionality described herein with respect to model development platform 12. This and other functionality can be implemented by developer tool(s) 75.

[0354] Third-party system(s) 80 can include one or more processors 81 and a memory 82. Processor(s) 81 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. Memory 82 can include one or more non-transitory computer-readable storage media, such as HBM, RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. Memory 82 can store data 83 and instructions 84 which can be executed by processor(s) 81 to cause third-party system(s) 80 to perform operations. The operations can implement any one or multiple features described herein. The operations can implement example methods and techniques described herein. Example operations include the functionality described herein with respect to tools and other external resources called when training or performing inference with machine-learned model(s) 1, 4, 16, 20, 55, 65, etc. (e.g., third-party resource(s) 85).

[0355] Figure 23illustrates one example arrangement of computing systems that can be used to implement the present disclosure. Other computing system configurations can be used as well. For example, in some implementations, one or both of computing system 50 or server computing system(s) 60 can implement all or a portion of the operations of model development platform system 70. For example, computing system 50 or server computing system(s) 60 can implement developer tool(s) 75 (or extensions thereof) to develop, update / train, or refine machine-learned models 1, 4, 16, 20, 55, 65, etc. using one or moretechniques described herein with respect to model alignment toolkit 17. In this manner, for instance, computing system 50 or server computing system(s) 60 can develop, update / train, or refine machine-learned models based on local datasets (e.g., for model personalization / customization, as permitted by user data preference selections).

[0356] Figure 24 is a block diagram of an example computing device 98 that performs according to example embodiments of the present disclosure. Computing device 98 can be a user computing device or a server computing device (e.g., computing device 50, server computing system(s) 60, etc.). Computing device 98 can implement model host 31. For instance, computing device 98 can include a number of applications (e.g., applications 1 through N). Each application can contain its own machine learning library and machine- learned model(s). For example, each application can include a machine-learned model. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. As illustrated in Figure 24, each application can communicate with a number of other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, or additional components. In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to that application.

[0357] Figure 25 is a block diagram of an example computing device 99 that performs according to example embodiments of the present disclosure. Computing device 99 can be the same as or different from computing device 98. Computing device 99 can be a user computing device or a server computing device (e.g., computing device 50, server computing system(s) 60, etc.). Computing device 98 can implement model host 31. For instance, computing device 99 can include a number of applications (e.g., applications 1 through N). Each application can be in communication with a central intelligence layer. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. In some implementations, each application can communicate with the central intelligence layer (and model(s) stored therein) using an API (e.g., a common API across all applications).

[0358] The central intelligence layer can include a number of machine-learned models. For example, as illustrated in Figure 25, a respective machine-learned model can be provided for each application and managed by the central intelligence layer. In other implementations, two or more applications can share a single machine-learned model. For example, in some implementations, the central intelligence layer can provide a single modelfor all of the applications. In some implementations, the central intelligence layer is included within or otherwise implemented by an operating system of computing device 99.

[0359] The central intelligence layer can communicate with a central device data layer. The central device data layer can be a centralized repository of data for computing device 99. As illustrated in Figure 25, the central device data layer can communicate with a number of other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API). Additional Disclosure

[0360] The technology discussed herein makes reference to servers, databases, software applications, and other computer-based systems, as well as actions taken and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a great variety of possible configurations, combinations, and divisions of tasks and functionality between and among components. For instance, processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0361] While the present subject matter has been described in detail with respect to various specific example embodiments thereof, each example is provided by way of explanation, not limitation of the disclosure. Those skilled in the art, upon attaining an understanding of the foregoing, can readily produce alterations to, variations of, and equivalents to such embodiments. Accordingly, the subject disclosure does not preclude inclusion of such modifications, variations or additions to the present subject matter as would be readily apparent to one of ordinary skill in the art. For instance, features illustrated or described as part of one embodiment can be used with another embodiment to yield a still further embodiment. Thus, it is intended that the present disclosure cover such alterations, variations, and equivalents.

[0362] Aspects of the disclosure have been described in terms of illustrative embodiments thereof. Any and all features in the following claims can be combined or rearranged in any way possible, including combinations of claims not explicitly enumerated in combination together, as the example claim dependencies listed herein should not be readas limiting the scope of possible combinations of features disclosed herein. Accordingly, the scope of the present disclosure is by way of example rather than by way of limitation, and the subject disclosure does not preclude inclusion of such modifications, variations or additions to the present subject matter as would be readily apparent to one of ordinary skill in the art. Moreover, terms are described herein using lists of example elements joined by conjunctions such as “and,” “or,” “but,” etc. It should be understood that such conjunctions are provided for explanatory purposes only. Clauses and other sequences of items joined by a particular conjunction such as “or,” for example, can refer to “and / or,” “at least one of”, “any combination of” example elements listed therein, etc. Terms such as “based on” should be understood as “based at least in part on.”

[0363] The term “can” should be understood as referring to a possibility of a feature in various implementations and not as prescribing an ability that is necessarily present in every implementation. For example, the phrase “X can perform Y” should be understood as indicating that, in various implementations, X has the potential to be configured to perform Y, and not as indicating that in every instance X must always be able to perform Y. It should be understood that, in various implementations, X might be unable to perform Y and remain within the scope of the present disclosure.

[0364] The term “may” should be understood as referring to a possibility of a feature in various implementations and not as prescribing an ability that is necessarily present in every implementation. For example, the phrase “X may perform Y” should be understood as indicating that, in various implementations, X has the potential to be configured to perform Y, and not as indicating that in every instance X must always be able to perform Y. It should be understood that, in various implementations, X might be unable to perform Y and remain within the scope of the present disclosure.

Claims

WHAT IS CLAIMED IS:

1. A computer-implemented method for improved decoding from sequence processing models, the method comprising: for each of one or more decoding iterations: processing, by the computing system, a current sequence of tokens with a machine-learned sequence processing model to generate a plurality of scores respectively for a plurality of tokens included in a token vocabulary associated with the machine-learned sequence processing model, wherein the current sequence of tokens comprises tokens from the plurality of tokens; evaluating, by the computing system, a finite-state or push-down automaton based on a current sequence of tokens to determine a respective validity status for each of the plurality of tokens, wherein the finite-state or push-down automaton comprises a plurality of states and a plurality of edges that define token-based-transitions between the plurality of states; selecting, by the computing system, one of the plurality of tokens as an output token for the current decoding iteration based at least in part on the plurality of scores and the respective validity status for each of the plurality of tokens.

2. The computer-implemented method of claim 1, wherein the finite-state or push-down automaton comprises a finite-state automaton.

3. The computer-implemented method of claim 2, wherein the finite-state automaton comprises a token-level finite state transducer composed with a character-level finite state automaton.

4. The computer-implemented method of claim 3, wherein the character-level finite state automaton encodes one or more regular expression constraints.

5. The computer-implemented method of claim 1, wherein the finite-state or push-down automaton comprises a push-down automaton.

6. The computer-implemented method of claim 5, wherein the push-down automaton comprises a token-level finite state transducer composed with a character-levelpush-down automaton, and wherein the character-level push-down automaton encodes a deterministic grammar.

7. The computer-implemented method of any preceding claim, wherein the plurality of scores comprise a plurality of logit scores.

8. The computer-implemented method of claim 1, wherein selecting, by the computing system, one of the plurality of tokens as the output token for the current decoding iteration based at least in part on the plurality of scores and the respective validity status for each of the plurality of tokens comprises: modifying, by the computing system, one or more of the plurality of scores respectively for one or more of plurality of tokens based on the respective validity status for the one or more of plurality of tokens to obtain a modified plurality of scores; and selecting, by the computing system, one of the plurality of tokens as an output token for the current decoding iteration based on the modified plurality of scores.

9. The computer-implemented method of claim 8, wherein modifying, by the computing system, the one or more of the plurality of scores comprises penalizing the scores associated with tokens that have a negative validity status.

10. The computer-implemented method of claim 8, wherein modifying, by the computing system, the one or more of the plurality of scores comprises masking the scores associated with tokens that have a negative validity status.

11. The computer-implemented method of any preceding claim, wherein evaluating, by the computing system, the finite-state or push-down automaton based on the current sequence of tokens to determine the respective validity status for each of the plurality of tokens comprises: determining, by the computing system, a current state of the finite-state or push-down automaton, wherein the current state of the finite-state or push-down automaton is a function of the current sequence of tokens; designating, by the computing system, tokens that are associated with one or more outbound edges of the current state as having a positive validity status; anddesignating, by the computing system, tokens that are not associated with one or more outbound edges of the current state as having a negative validity status.

12. The computer-implemented method of any preceding claim, wherein the finite-state or push-down automaton comprises one or more terminal labels for wildcard matching, wherein the one or more terminal labels comprise tokens that are not included in the token vocabulary but which map to a pre-computed set of valid tokens from the token vocabulary.

13. The computer-implemented method of claim 12, wherein the one or more terminal labels comprise one or more of the following: a QUOTED_TEXT label that matches a quoted string with backslash escapes; an UNQUOTED_TEXT label that matches a YAML-style non-quoted string; an IMAGE label that matches an image generated by a multi-modal LM; a TEXT_TOKEN label that matches a single text token; a PARAGRAPH_TOKEN label that matches a single text token with no newlines; or a TEXT_UNTIL label that matches text tokens until a stop phrase appears.

14. The computer-implemented method of any preceding claim, wherein the finite-state or push-down automaton comprises one or more syntactic extensions.

15. The computer-implemented method of claim 14, wherein the one or more syntactic extensions comprise one or more of the following: a SUBSTRING_OF extension that matches any substring of a given string; a DELIMITED_LIST extension that matches a delimited list of homogeneous items; or a DELIMITED_SUBSEQUENCE_OF extension that matches a delimited subset of a heterogeneous list.

16. The computer-implemented method of any preceding claim, further comprising, for each of the one or more decoding iterations, appending, by the computing system, the selected output token to the current sequence of tokens.

17. The computer-implemented method of any preceding claim, wherein evaluating, by the computing system, the finite-state or push-down automaton comprises evaluating, by the computing system, a table-based representation of the finite-state or push- down automaton, wherein said evaluation is performed within a computation graph.

18. A computer system comprising: one or more processors; and one or more non-transitory computer-readable media that collectively store: a machine-learned sequence processing model configured to process a current sequence of tokens to generate a plurality of scores respectively for a plurality of tokens included in a token vocabulary associated with the machine-learned sequence processing model, wherein the current sequence of tokens comprises tokens from the plurality of tokens; and a finite-state or push-down automaton configured to evaluate a respective validity status for each of the plurality of tokens based on the current sequence of tokens, wherein the finite-state or push-down automaton comprises a plurality of states and a plurality of edges that define token-based-transitions between the plurality of states; and computer-executable decoding logic configured to select one of the plurality of tokens as an output token for the current decoding iteration based at least in part on the plurality of scores and the respective validity status for each of the plurality of tokens.

19. The computer system of claim 18, wherein the finite-state or push-down automaton comprises a finite-state automaton.

20. The computer system of claim 19, wherein the finite-state automaton comprises a token-level finite state transducer composed with a character-level finite state automaton.

21. The computer system of claim 20, wherein the character-level finite state automaton encodes one or more regular expression constraints.

22. The computer system of claim 18, wherein the finite-state or push-down automaton comprises a push-down automaton.

23. The computer system of claim 22, wherein the push-down automaton comprises a token-level finite state transducer composed with a character-level push-down automaton, and wherein the character-level push-down automaton encodes a deterministic grammar.

Images