Machine learning models with constrained output lexical vocabularies
By introducing a decoder plugin and a supervised machine learning model, and by using regular expressions and reward models to constrain the output vocabulary, the problems of illusion generation and autoregressive drift in machine learning models are solved, achieving more efficient and secure output control.
Patent Information
- Application Number
- CN202580029034.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-12-02
- Filing Date
- 2025-03-13
- Publication Date
- 2026-11-17
AI Technical Summary
Existing machine learning models are prone to problems such as illusions, autoregressive drift, planning difficulties, and difficulties in preventing harmful outputs when generating outputs, especially in terms of limited context window size and computational resource consumption.
By introducing decoder plugins, the output vocabulary is constrained using regular expressions, context-free grammars, or Turing-complete languages. This is combined with supervised machine learning models and reward models to control output, prevent the generation of harmful content, and optimize planning through search algorithms.
It effectively reduces illusions, improves planning accuracy, reduces computational resource consumption, prevents harmful outputs, and enhances generation efficiency and security.
Smart Images

Figure CN123125601A_ABST
Abstract
Description
Background Technology
[0001] In machine learning environments, controlling the output of Large Language Models (LLMs) and Large Multimodal Models (LMMs) using prompt words is a typical approach. When using prompt words, users typically provide text input in natural language, which is converted into input tokens. These input tokens are then processed at the machine learning model to generate a probability distribution of potential output tokens. Output tokens are sampled from this probability distribution to be included in the output presented to the user. When iteratively generating an output that includes multiple output tokens, the output tokens can also be viewed as if they were included in the tokenized input. Therefore, the output is generated in an autoregressive manner, where the machine learning model uses previously generated output tokens as input when generating later output tokens. Summary of the Invention
[0002] According to one aspect of this disclosure, a computing system is provided, including one or more processing devices configured to receive a cue word. At a machine learning model having an output lexicon including a plurality of candidate output lexicons, the one or more processing devices are further configured to compute a plurality of output lexicon probabilities on the output lexicon at least partially based on the cue word. At a decoder plug-in, the one or more processing devices are further configured to compute a constrained output lexicon, the constrained output lexicon being a proper subset of the output lexicon. The one or more processing devices are further configured to select one or more output lexicons at least partially based on the computed output lexicon probabilities. The one or more output lexicons are selected from candidate output lexicons included in the constrained output lexicon. The one or more processing devices are further configured to transmit an output including the one or more output lexicons to an additional computing process.
[0003] This synopsis is provided to introduce some concepts in a simplified form, which will be further described in the detailed embodiments below. This synopsis is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to the implementation of solutions to any or all the shortcomings mentioned in any part of this disclosure. Attached Figure Description
[0004] Figure 1 A computing system according to an example embodiment is illustrated, in which one or more processing devices are configured to execute machine learning models and decoder plugins.
[0005] Figure 2 It schematically shows the following based on Figure 1 The example in which the guiding logic is executed to constrain the output lexical vocabulary is an example.
[0006] Figure 3 It schematically shows the following based on Figure 1 The example computation system calculates the constrained output lexical probabilities.
[0007] Figure 4 It schematically shows the following based on Figure 1 The example is a decoder plugin configured to constrain machine learning models to output valid chess moves.
[0008] Figure 5A It schematically shows the following based on Figure 1 One example of the decoder plugin includes a computational system in an example of a supervised machine learning model.
[0009] Figure 5B It schematically shows the following based on Figure 5A Examples of candidate output term classification when the supervised machine learning model is a toxicity classifier.
[0010] Figure 6 It schematically shows the following based on Figure 1 The example decoder plugin includes a computational system in the example of a supervised machine learning model and a reward model.
[0011] Figure 7 It schematically shows the following based on Figure 1 One or more processing devices in the example are configured to execute a search algorithm in the example computing system when computing a constrained output lexical vocabulary.
[0012] Figure 8 It schematically shows the following based on Figure 1 The example of the computational system in which at least part of the decoder plugin is generated at the machine learning model.
[0013] Figure 9A It shows according to Figure 1 The flowchart illustrates an example of a method for computing systems to constrain the output lexical vocabulary of machine learning models.
[0014] Figure 9B-9F This shows what can be performed in some examples. Figure 9A Additional steps to the method.
[0015] Figure 10 It shows where it can be instantiated Figure 1 A schematic diagram of an example computing environment for a computing system. Detailed Implementation
[0016] While prompts offer users a flexible and easily understood means of controlling LLMs and LMMs, they also have drawbacks that limit their effectiveness in many settings. One such drawback is the illusion, where an LLM or LMM generates output that is factually inaccurate but resembles the text included in the training data of a machine learning model. For example, LLMs and LMMs sometimes cite non-existent literature when generating academic text that may contain citations.
[0017] Relying solely on cue words to control the output of a machine learning model may struggle to avoid illusions. Since the machine learning model's next lexical prediction depends on its own previously generated output lexicals, inaccuracies in the model's output can accumulate by affecting the probability distribution of subsequent output lexical selection. Therefore, as the number of lexicals in the context increases, errors may become increasingly likely in the machine learning model's output. This autoregressive drift may be unavoidable with conventional cue word engineering techniques. Furthermore, because illusions often appear to contain correct information, users may not realize the output is an illusion unless they conduct further fact-checking.
[0018] As another consequence of the autoregressive drift discussed above, planning is often difficult for existing LLMs and LMMs. For example, when a user instructs an LLM or LMM to play a board game, the machine learning model may err in its representation of the game state and may consequently attempt invalid moves. In existing machine learning models, obtaining valid moves as output may require additional cue words, which consumes extra computational resources and can be time-consuming for the user.
[0019] Limited context window size can also interfere with planning at LLM and LMM. When the total number of lexical units included in the cue words and output exceeds the context window size, the machine learning model may be unable to condition its output with earlier parts of its context, and therefore may not be able to consider information included in those earlier parts. This forgetting can be exacerbated when using multi-example cue words or when the user instructs the machine learning model to regenerate parts of its output.
[0020] Current methods for controlling machine learning models also struggle to prevent them from producing harmful outputs. For example, developers of machine learning models may intend to prevent the model from outputting responses that encourage illegal activities, leak personally identifiable information, provide advice that could lead users into danger, or be used to harass others. One existing approach to preventing harmful outputs is reinforcement learning based on human feedback (RLHF), where additional training is performed on the machine learning model using a reward model trained on human-provided classifications. However, RLHF has several drawbacks, including high cost and susceptibility to being circumvented by specially designed cue words.
[0021] Another existing approach to preventing harmful outputs relies on a harmfulness classifier, which is applied to the outputs of the machine learning model after they have been generated. If an output is classified as having a high probability of being harmful, the harmfulness classifier can instruct the machine learning model to regenerate that output. However, regenerating the output can be computationally expensive, especially in scenarios where the output is regenerated multiple times.
[0022] To address the shortcomings of existing machine learning model control techniques, a computing system 10 is provided, such as... Figure 1 The example is schematically depicted. The computing system 10 includes one or more processing devices 12 and one or more memory devices 14. The one or more processing devices 12 may include, for example, one or more central processing units (CPUs), graphics processing units (GPUs), neural processing units (NPUs), and / or other types of hardware accelerators. The one or more memory devices 14 may include, for example, one or more volatile memory devices and one or more non-volatile memory devices.
[0023] In some examples, one or more processing devices 12 and / or one or more memory devices 14 may include multiple physical components distributed across multiple different physical computing devices. For example, one or more processing devices 12 and / or one or more memory devices 14 may be included in a networked system of multiple physical computing devices located in a data center. Part of the functionality of one or more processing devices 12 and / or one or more memory devices 14 may additionally or alternatively be performed at one or more client computing devices.
[0024] One or more processing devices 12 are configured to receive the prompt 20. For example, the prompt 20 may be received in natural language. In some examples, the prompt 20 may be entered by a user at a user interface. In other examples, the prompt 20 may be generated programmatically at another computational process and may be received from that other computational process via an application programming interface (API).
[0025] One or more processing devices 12 are also configured to execute a lexer 22 to compute a lexicalized cue word 23 based at least in part on the cue word 20. The lexicalized cue word 23 includes a plurality of input lexical units 24, which may, for example, indicate a word, a part of a word, or other characters such as numbers or punctuation marks. The lexer 22 is thus configured to encode the cue word 20 in a form that can be used as input to a machine learning model 30.
[0026] The machine learning model 30 executed at one or more processing devices 12 may be, for example, an LLM or an LMM. Figure 1As shown in the example, machine learning model 30 is constructed as a deep neural network comprising multiple layers 32. For example, machine learning model 30 could be a transformer network. Alternatively, several other architectures can be used for machine learning model 30.
[0027] The machine learning model 30 has an output lexicon 40 that includes multiple candidate output lexicons 42. The candidate output lexicons 42 included in the output lexicon 40 are lexicons that can be included in the output of the machine learning model 30. In some examples, the output lexicon 40 may include the same lexicons as the input lexicon 40 used by the lexiconizer to compute the lexiconization cue word 23.
[0028] Based at least in part on the lexicalized cue word 23, the machine learning model 30 is configured to compute multiple output lexical probabilities 34 on the output lexical vocabulary 40. For example, the machine learning model 30 can be configured to generate logit values as the output of its final layer 32. These logit values are unnormalized predictions. The machine learning model 30 can also be configured to apply a normalization function to the logit values to compute the output lexical probabilities 34. The output lexical probabilities form the output lexical probability distribution 35.
[0029] The machine learning model 30 also includes a sampler 38, where one or more processing devices 12 are configured to sample output words 52 from the output word probability distribution 35. The sampler 38 may be included in the decoding module 36. Therefore, one or more processing devices 12 are configured to select one or more output words 52 based at least in part on the calculated output word probabilities 34. In some examples, the output word probabilities 34 transmitted to the sampler 38 are the highest probabilities predicted at the machine learning model 30 for a predetermined number of corresponding candidate output words 42. For example, the decoding module 36 may be configured to select the first 8 or the first 16 output word probabilities 34.
[0030] At machine learning model 30, one or more processing devices 12 are configured to compute one or more output lexical probabilities 34 in each of a plurality of autoregressive generation iterations 44. During the autoregressive generation iterations 44, the one or more processing devices 12 are configured to compute an output 50 comprising a plurality of output lexical terms 52. Figure 1 As shown in the example, the input lexical unit 24, included in the lexicalized cue word 23, is fed into the machine learning model 30 as part of the context 26. At each autoregressive generation iteration 44 following the first autoregressive generation iteration 44, the context 26 used as input to the machine learning model 30 also includes the previous output sequence 29. The previous output sequence 29 includes one or more previous output lexical units 28 that were computed as output lexical unit 52 at the previous autoregressive generation iteration 44.
[0031] After generating output 50, one or more processing devices 12 are further configured to transmit output 50 to an additional computation process 54. For example, the additional computation process 54 may be a graphical user interface (GUI) at which output 50 is displayed to a user. As another example, one or more processing devices 12 may be configured to transmit output 50 to a compiler, where the output is compiled into assembly-level instructions.
[0032] One or more processing devices 12 are also configured to execute a decoder plugin 60 when output 50 is generated at machine learning model 30. At decoder plugin 60, one or more processing devices 12 are configured to execute bootstrapping logic 62 to compute a constrained output lexical vocabulary 64. The constrained output lexical vocabulary 64 is a proper subset of the output lexical vocabulary 40. As discussed in the examples provided below, bootstrapping logic 62 can utilize various types of programming logic to compute the constrained output lexical vocabulary 64.
[0033] At the sampler 38 of the machine learning model 30, one or more processing devices 12 are configured to select one or more output words 52 from candidate output words 42 included in the constrained output word vocabulary 64. Therefore, the decoder plugin 60 provides additional control over the output 50 of the machine learning model 30 by narrowing down a set of candidate output words 42 that can be selected as one or more output words 52 to the candidate output words 42 included in the constrained output word vocabulary 64.
[0034] In some examples, at decoder plugin 60, one or more processing devices 12 may also be configured to iteratively update the constrained output lexical vocabulary 64 at each of the multiple autoregressive generation iterations 44. When additional previous output lexical terms 28 are added, the constrained output lexical vocabulary 64 may be updated at least partially based on the context 26. For example, as discussed in further detail below, the additional previous output lexical terms 28 may inform the predictions of the toxicity classifier included in decoder plugin 60 and used to select the constrained output lexical vocabulary 64.
[0035] In some examples, the bootstrapping logic 62 may be configured to compute the constrained output lexical vocabulary 64 at least in part by computed vocabulary update 66, which specifies modifications to the output lexical vocabulary 40. The vocabulary update 66 may be stored in one or more memory devices 14 for later use, such as for one or more other cue words 20 or autoregressive generation iterations 44. In such examples, the one or more memory devices 14 may store a library of pre-computed vocabulary updates 66. In some examples, the constrained output lexical vocabulary 64 may be computed at least in part by applying multiple stored vocabulary updates 66 to the output lexical vocabulary 40. In such examples, multiple pre-computed changes to the output lexical vocabulary 40 are applied accordingly.
[0036] Figure 2 An example is schematically shown in which the guiding logic 62 is executed to constrain the output lexical vocabulary 40. Figure 2 In the example, context 26 includes the 32 tokens that make up the beginning of the JavaScript Object Notation (JSON) template: { "Findings": [ { "ID": One or more processing devices 12 are configured to calculate a set of output word probabilities 34 for the corresponding candidate output word 42 predicted as the potential next word in the template. In this example, the candidate output word 42 includes “1”, “ID”, “5”, “Hi”, and “A”.
[0037] One or more processing devices 12 are also configured to execute bootstrapping logic 62 to select one or more candidate output lexics 42 and mask them as invalid output lexics. Therefore, one or more processing devices 12 are configured to compute a constrained output lexicographical vocabulary 64. For example, one or more processing devices 12 may be configured to specify the constrained output lexicographical vocabulary 64 using a regular expression 62A or a context-free grammar 62B. Figure 2 For example, regular expression 62A is the regular expression "ID": {gen(regex="[0-9]+")}, which specifies that the next output term 52 is a digit between 0 and 9, inclusive. This regular expression 62A is applied to a specific field in the template.
[0038] In the example where bootstrapping logic 62 is a context-free grammar 62B, the context-free grammar 62B can be applied as a whole to the template and can specify the properties that syntactically valid JSON has. One or more processing devices 12 are thus configured to make the machine learning model 30 more likely to generate syntactically valid templates by constraining the output lexical vocabulary 40. In other examples, bootstrapping logic 62 can be specified using a Turing-complete language instead of regular expression 62A or context-free grammar 62B.
[0039] Figure 3 The computational system 10 is illustrated schematically in an example where additional modifications are performed at the decoding module 36 of the machine learning model 30. At the decoding module 36, one or more processing devices 12 may also be configured to rescale the output lexical probabilities 34 to obtain constrained output lexical probabilities 70 for candidate output lexical 42. The constrained output lexical probabilities 70 may be normalized over a constrained output lexical vocabulary 64. Therefore, at the sampler 38, one or more processing devices 12 may be configured to sample one or more output lexical 52 based on the constrained output lexical probabilities 70.
[0040] In some examples, instead of completely excluding one or more candidate output words from candidate output words 42 from being included in output 50, one or more processing devices 12 may alternatively be configured to reduce the output word probabilities 34 of those one or more candidate output words 42 output by the machine learning model 30, rather than reducing those output word probabilities 34 to zero. In such an example, at decoder plugin 60, one or more processing devices 12 may be configured to modify the corresponding output word probabilities 34 associated with the plurality of candidate output words 42 to obtain constrained output word probabilities 70. In such an example, bootstrapping logic 62 may identify one or more candidate output words 42, and one or more processing devices 12 are configured to reduce the corresponding output word probability 34 for that one or more candidate output words 42. Bootstrapping logic 62 may additionally or alternatively identify one or more candidate output words 42, and one or more processing devices 12 are configured to increase the corresponding output word probability 34 for that one or more candidate output words 42. In such an example, one or more processing devices 12 are also configured to select one or more output words 52 at least in part based on the constrained output word probabilities 70.
[0041] exist Figure 3In the example, one or more processing devices 12 are also configured to modify one or more sampling parameters 72 of the machine learning model 30. The one or more sampling parameters are parameters used during the selection of output words 52 from the output word probability distribution 35 to determine which output words 52 are selected. For example, one or more sampling parameters 72 may include temperature. At the decoder plugin 60, one or more processing devices 12 may be configured to compute one or more modified sampling parameters 74 to replace the one or more sampling parameters 72 used at the sampler 38. For example, one or more modified sampling parameters 74 may be used when the user instructs the machine learning model 30 to generate multiple different completions of the context 26. In such an example, the decoder plugin 60 may increase the temperature to increase the variety of output 50, or it may decrease the temperature to increase the consistency of output 50.
[0042] Figure 4 Another example of decoder plugin 60 is illustrated schematically. Figure 4 In the example, machine learning model 30 is instructed to play chess, and decoder plugin 60 is configured to constrain machine learning model 30 to output legal chess moves. Existing autoregressive machine learning models trained on the following chess games often experience autoregressive drift as the game progresses. Therefore, such models frequently lose track of the game state and begin outputting illegal moves. The user may have to further provide the machine learning model with prompts to correct the representation of the game state and redirect its output back to legal chess moves. This additional prompt disrupts the user experience and consumes additional time and computational resources.
[0043] Figure 4 The bootstrapping logic 62 of the decoder plugin 60 in the example includes a state tracking module 80, which stores a description of the chessboard state, including the positions of the pieces. The state tracking module 80 also stores values of other variables affecting the legality of chess moves, such as which player's turn it is. The data stored in the state tracking module 80 is used as input to a move legality determination module 82, where one or more processing devices 12 are configured to compute a constrained output lexicon 64 as a list of legal moves. At the machine learning model 30, one or more processing devices 12 are configured to select the next output lexicon 52 from the chess moves listed as legal moves in the constrained output lexicon 64.
[0044] As the chess game progresses, one or more processing devices 12 are configured to update the state tracking module 80 and recalculate the list of legal moves at the move legality determination module 82. Therefore, by combining deterministic tracking of move legality, the decoder plugin 60 constrains the machine learning model 30 to output legal chess moves. Consequently, the machine learning model 30 is less likely to require additional cue words to successfully play chess against the user.
[0045] Figure 5A The illustration schematically shows a computational system 10 in which the decoder plugin 60 includes a supervised machine learning model 90. Figure 5A In the example, the supervised machine learning model 90 is a classifier. At the supervised machine learning model 90, one or more processing devices 12 are also configured to compute a predicted classification 92 of the output 50 of the machine learning model 30. This predicted classification 92 may be conditional on the context 26. The supervised machine learning model 90 may, for example, be smaller than the machine learning model 30 in terms of parameter count, and therefore may have lower computational costs associated with inference. The supervised machine learning model 90 can therefore be used to heuristically filter the potential outputs 50 of the machine learning model 30 in a manner lower than the cost of generating multiple different outputs 50 of the machine learning model 30 and selecting among these outputs 50.
[0046] In some instances, when estimating the predicted classification 92, one or more processing devices 12 can be configured to compute prediction completions 94 for the context 26. Prediction completions 94 may include sequences of prediction completion tokens 96. A supervised machine learning model 90 can be configured, for example, to generate prediction completions 94 in examples where the supervised machine learning model 90 is a low-capacity LLM or LMM with low inference costs. In some examples, one or more processing devices 12 can be configured to generate multiple different prediction completions 94.
[0047] exist Figure 5A In the example, one or more processing devices 12 are further configured to compute multiple completion-specific predicted classifications 98 at a supervised machine learning model 90. Each completion-specific predicted classification 98 is associated with a predicted completion 94. The supervised machine learning model 90 can compute the completion-specific predicted classifications 98 as probability vectors associated with the corresponding categories.
[0048] One or more processing devices 12 may also be configured to select a constrained output lexicon 64 based at least in part on a completion-specific prediction classification 98. For example, candidate output lexicons 42 selected for inclusion in the constrained output lexicon 64 may be one or more candidate output lexicons 42 located at the beginning of one or more prediction completions in the prediction completion 94. Those candidate output lexicons 42 may be candidate output lexicons that are predicted to produce an output 50 having an indicated completion-specific prediction classification 98, and the probability of that classification is higher or lower than a predefined probability threshold 99. For example, one or more processing devices 12 may be configured to determine the probability that the prediction completion 94 leaks personally identifiable information. One or more processing devices 12 may also be configured to exclude one or more candidate output lexicons 42 from the constrained output lexicon 64 if the completion-specific prediction classification 98 of the prediction completion 94 starting with those candidate output lexicons 42 is higher than the predefined probability threshold 99.
[0049] In some examples, the supervised machine learning model 90 can be configured to compute a predicted classification 92 for output 50 without computed prediction completion 94. For example, the supervised machine learning model 90 can be trained on training data that includes multiple context classification pairs. These context classification pairs can each indicate whether the machine learning model 30 generated a specific type of output 50 (e.g., output labeled as harmful by humans) when the context 26 included in that context classification pair is used as input. In such an example, the bootstrapping logic 62 can include a corresponding constrained output lexicon 64 used when such a class is detected, or instructions for generating such a constrained output lexicon 64.
[0050] Figure 5B An example of candidate output lexical classification is shown when the supervised machine learning model 90 is a toxicity classifier. Figure 5B In the example, the user inputs the prompt "generate internet posts that insult someone," which, if followed without supervision, would result in toxic output. One or more processing devices 12 are configured to lexicalize the prompt 20 into a lexicalized prompt 23, which is then fed into a supervised machine learning model 90.
[0051] Supervised machine learning model 90 is configured to generate multiple distinct predictive completions 94, each predictive completion including multiple predictive completion lexical units 96 selected from candidate output lexical units 42 included in the output lexical vocabulary 40 of machine learning model 30. Supervised machine learning model 90 and machine learning model 30 in... Figure 5B The examples have the same output lexical vocabulary 40.
[0052] The supervised machine learning model 90 is also configured to compute word-specific predictive classifications 98 associated with the first word of the corresponding predictive completion 94. Figure 5B In the example, the word-specific prediction classification 98 indicates the probability that output 50 is toxic given that output 50 begins with that prediction completion word 96. In some examples, the supervised machine learning model 90 is also configured to compute word-specific prediction classifications 98 for one or more subsequent prediction completion words 96.
[0053] One or more processing devices 12 are also configured to compare word-specific predicted classifications 98 with predefined probability thresholds 99, and to exclude corresponding predicted completed words 96 from the constrained output word vocabulary 64 if those predicted completed words 96 have a toxic probability higher than the predefined probability threshold. Figure 5B In the example, supervised machine learning model 90 generates predictive completions starting with the lexical units “What,” “I,” “The,” “That,” and “Are.” One or more processing devices 12 predict that outputs 50 starting with “What” or “Are” have a probability of producing toxic outputs above a predefined probability threshold 99, and therefore exclude “What” and “Are” from the constrained output lexical vocabulary 64. Machine learning model 30 is accordingly configured to select output lexical units 52 from “I,” “The,” and “That.”
[0054] In some examples, one or more processing devices 12 may be configured to execute a supervised machine learning model 90 as each output lexical unit 52 is generated. In other examples, one or more processing devices 12 may be configured to execute the supervised machine learning model 90 at specified intervals, such as every 10 lexical units or every 20 lexical units. Thus, one or more processing devices 12 may be configured to guide the generation of output 50 away from toxic responses without incurring the computational cost associated with running the supervised machine learning model 90 at each output lexical unit 52.
[0055] In some examples, such as Figure 6 As depicted in the example, a supervised machine learning model 90 can be included in the decoder plugin 60 along with a reward model 100. At the supervised machine learning model 90, one or more processing devices are configured to compute one or more prediction completions 94 of the context 26, as shown in... Figures 5A-5B In the example, prediction completion 102 each includes multiple prediction completion tokens 96.
[0056] At reward model 100, one or more processing devices 12 are further configured to compute a corresponding reward value 102 associated with each of the one or more prediction completions 94. Reward model 100 may, for example, be a machine learning model trained using human-assigned reward scores associated with samples of training text. In other examples, the reward scores used to train reward model 100 may be generated at least partially at another machine learning model receiving prompt words that include corresponding text samples. Reward model 100 may, for example, be trained to perform toxicity detection, where a higher reward value 102 corresponds to a lower toxicity output.
[0057] One or more processing devices 12 are also configured to select a constrained output lexical vocabulary 64 based at least in part on one or more reward values 102. For example, the constrained output lexical vocabulary 64 may be selected to include each predicted completion lexical 96 generated at the beginning of a corresponding predicted completion 94 having a reward value 102 higher than the reward threshold 104.
[0058] By using a decoder plugin 60 that includes a reward model 100, the output 50 of the machine learning model 30 can be guided by the reward signal without having to perform additional training on the machine learning model 30. In contrast, RLHF relies on further training of the machine learning model 30, which can be expensive. Figure 6 The configuration also allows users to switch between different reward models 100. For example, different decoder plugins 60 with different corresponding reward models 100 can be used, depending on whether the machine learning model 30 is instructed to generate text in a formal or informal writing style. Compared to reapplying RLHF with different reward models (which is slow and computationally expensive), the replacement of decoder plugins 60 to use different reward models 100 can be performed quickly (e.g., midway through a usage session in response to user input) and at low cost.
[0059] although Figure 5A-6 The examples discussed toxicity filtering, but decoder plugins can also be used to provide other types of security controls. Such security controls can, for example, constrain the output lexical vocabulary 40 to prevent behaviors such as leaking confidential information or generating malicious code. The supervised machine learning model 90 included in the decoder plugin 60 can also be used to detect and suppress unintended autonomous behavior of the machine learning model 30, such as attempts by the machine learning model 30 to leak its own parameters or enhance its control over its hardware.
[0060] Figure 7 The computational system 10 in the example is illustrated schematically, wherein at the decoder plug-in 60, one or more processing devices 12 are configured to execute a search algorithm 110 while computing a constrained output lexical vocabulary 64. Figure 7 In one example, one or more processing devices 12 are configured to perform the Monte Carlo tree search (MCTS) algorithm. In other examples, one or more processing devices 12 may be configured to perform other types of search algorithms, such as simulated annealing, simulated quantum annealing, parallel annealing, or interactive Metropolis-Hastings algorithms, to name just a few non-limiting examples.
[0061] One or more processing devices 12 are configured to execute search algorithm 110 over a predefined search domain 112. The predefined search domain 112 may, for example, be a space of candidate output terms 42. In some examples, the predefined search domain 112 may be the entire output term vocabulary 40 of the machine learning model 30. In some examples, the predefined search domain 112 may be a subset 114 of the output term vocabulary 40. For example, when the machine learning model 30 is used to perform mathematical reasoning, subset 114 may include candidate output terms 42 that frequently appear in mathematical expressions. Subset 114 may be specified via user input.
[0062] In other examples, instead of searching the space of candidate output lexical 42, one or more processing devices 12 can be configured to search the numerical range 116 of values, which are further processed at bootstrapping logic 62 to compute a constrained output lexical vocabulary 64. For example, the numerical range 116 could be a range of potential numerical values that the machine learning model 30 could use to fill fields in a template, as in Figure 2 In the example, the predefined search domain 112 can be a combination input space 118 that includes candidate combinations of discrete values other than candidate output tokens 42. For example, when the machine learning model 30 is used to prove a mathematical theorem, the combination input space 118 can be a space of transformations that can be performed on the mathematical expression.
[0063] exist Figure 7 In the example, one or more processing devices 12 are configured to construct a search tree 120 over a predefined search domain 112 that includes multiple search branches 122. The search branches 122 may be, for example, a sequence of moves in a board game or a sequence of steps in a proof. One or more processing devices 12 may be configured to iteratively expand the search tree 120 as the search proceeds through the predefined search domain 112.
[0064] When a search is performed through search tree 120, one or more processing devices 12 are configured to assign a corresponding branch score 126 to search branch 122. Each branch score 126 is computed at a scoring function 124. In some examples, the scoring function 124 may be a machine learning model trained to perform a task such as evaluating which player wins a board game. In other examples, one or more processing devices 12 are configured to compute the branch score 126 at a rule-based scoring function.
[0065] One or more processing devices 12 are also configured to obtain search results 128 from search algorithm 110 and to select a constrained output lexical vocabulary 64 based at least in part on search results 128. In an example where one or more processing devices 12 are configured to search on output lexical vocabulary 40 or a subset thereof 114, search results 128 may directly specify one or more candidate output lexical terms 42 to be included in or excluded from the constrained output lexical vocabulary 64. In other examples, guiding logic 62 may also include one or more post-processing operations, such as lexicalization of numbers. Thus, one or more processing devices 12 are configured to incorporate the planning capabilities of search algorithm 110 into the selection of one or more output lexical terms 52 by using search results 128 to guide the constraints on output lexical vocabulary 40.
[0066] Figure 8 The example shown is a computational system 10 in which the decoder plugin 60 is at least partially generated at the machine learning model 30. Figure 8 The computational operations shown in the examples can occur during the decoder plugin generation phase 140, which is performed before the computation of the constrained output lexical vocabulary 64. In some examples, the decoder plugin generation phase 140 is performed in a separate use session from the computation of output 50, and the generated decoder plugin 60 is stored in one or more memory devices 14 for later use. Alternatively, the decoder plugin generation phase 140 can be performed in the same use session as the initial phase of generating a response to prompt word 20.
[0067] In the decoder plugin generation phase 140, one or more processing devices 12 are configured to receive decoder plugin generation prompts 130. For example, decoder plugin generation prompts 130 may be received as user input and may be part of prompt 20, for which machine learning model 30 is configured to generate output 50 using a constrained output lexical vocabulary 64. Decoder plugin generation prompts 130 may specify instructions in natural language for generating at least a portion of decoder plugin 60. For example, decoder plugin generation prompts 130 may be a label assigned to a field in a template and stating that "this part of the output must be syntactically valid Python". One or more processing devices 12 are also configured to compute lexicalized decoder plugin generation prompts 132 at lexer 22.
[0068] One or more processing devices 12 are also configured to input the lexical decoder plugin-generated prompt word 132 into the machine learning model 30. At the machine learning model 30, where the prompt word 132 is generated at least partially based on the lexical decoder plugin, one or more processing devices 12 are also configured to compute at least a portion 136 of the decoder plugin 60. For example, the machine learning model 30 may compute a programmatically generated plugin portion 136 as a filler field for the decoder plugin template 134. In such an example, the decoder plugin template 134 also includes a previously generated portion specifying the structure of the decoder plugin 60. In some examples, one or more processing devices 12 may be configured to generate the programmatically generated plugin portion 136 at a different machine learning model, rather than at the machine learning model 30.
[0069] In some examples, one or more processing devices 12 may also be configured to execute a formal validator 138 during the decoder plugin generation phase 140. The formal validator 138 is configured to receive a programmatically generated plugin portion 136 as input and programmatically examine one or more properties of the programmatically generated plugin portion 136. For example, the formal validator 138 may be a type checker. The programmatically generated plugin portion 136 may be included in the decoder plugin 60 when it passes one or more validation checks performed at the formal validator 138. The machine learning model 30 may be configured to regenerate the programmatically generated plugin portion 136 if it fails one or more validation checks.
[0070] As described above, after the programmatically generated plugin portion 136 has been computed and verified, one or more processing devices can also be configured to use the decoder plugin 60 to constrain the output 50 of the machine learning model 30. Therefore, the machine learning model 30 can be used to transform attributes specified in unstructured user input into constraints on its output lexical vocabulary 40.
[0071] Figure 9A A flowchart of a method 200 for constraining the output lexical vocabulary of a machine learning model for a computing system is shown. At step 202, method 200 includes receiving a prompt word. For example, the prompt word may be received at the computing system as user input in natural language form.
[0072] Step 204 of method 200 is performed at a machine learning model. For example, the machine learning model may be a deep neural network with a transformer architecture. This machine learning model has an output lexicon that includes multiple candidate output lexicons suitable for inclusion in its output. At step 204, method 200 further includes calculating multiple output lexicon probabilities on the output lexicon vocabulary, at least in part based on the cue word. The output lexicon probabilities are associated with corresponding candidate output lexicons.
[0073] At step 206, method 200 further includes computing a constrained output lexical vocabulary at the decoder plugin. The constrained output lexical vocabulary is computed as a proper subset of the output lexical vocabulary. In some examples, at step 207, step 206 includes specifying the constrained output lexical vocabulary using regular expressions or context-free grammars. In other examples, bootstrap logic written in a Turing-complete language can be used to compute the constrained output lexical vocabulary.
[0074] At step 208, method 200 further includes selecting one or more output tokens at least in part based on the calculated output token probabilities. One or more output tokens may be probabilistically sampled from the distribution of output token probabilities at a sampler included in the decoding module of the machine learning model. One or more output tokens are selected from candidate output tokens included in a constrained output token vocabulary. Thus, the decoder plugin guides the generation of output tokens by narrowing down the output token vocabulary from which the selected tokens are fit.
[0075] At step 210, method 200 further includes transferring output, including one or more output tokens, to an additional computational process. For example, the output may be transferred to be displayed to a user at a GUI. As another example, the output may be transferred to a compiler to be compiled into assembly-level instructions.
[0076] Figure 9B-9F This shows what can be performed in some examples. Figure 9AAdditional steps for method 200. For example... Figure 9B As shown, step 212 can be performed after receiving the prompt word and step 202, and before calculating the output lexical probability at step 204. At step 212, method 200 may further include calculating a lexicalized prompt word at least partially based on the prompt word. The lexicalized prompt word may include multiple input lexicals.
[0077] At step 214, method 200 may further include calculating output lexical probabilities in each of a plurality of autoregressive generation iterations at the machine learning model. In each autoregressive generation iteration, the machine learning model may calculate a corresponding output lexical probability distribution, which is sampled to obtain the corresponding output lexical.
[0078] At step 216, method 200 may further include, at the decoder plugin, iteratively updating the constrained output lexical vocabulary at each of the multiple autoregressive generation iterations. In some examples, the decoder plugin may update the constrained output lexical vocabulary at each autoregressive generation iteration after the first autoregressive generation iteration, while in other examples, the decoder plugin may update the constrained output lexical vocabulary less frequently. Guiding logic may include logic specifying an update plan for the constrained output lexical vocabulary. When the constrained output lexical vocabulary is updated, the decoder plugin may update the constrained output lexical vocabulary at least partially based on context, which includes lexicalization cue words and previous output sequences. Previous output sequences may include one or more previously output lexical units computed at one or more corresponding previous autoregressive generation iterations. Therefore, the constrained output lexical vocabulary can be iteratively modified to reflect the addition of output lexical units to the context.
[0079] Figure 9C Additional steps of method 200 are shown in some examples where the decoder plugin includes a supervised machine learning model. At the supervised machine learning model, method 200 may also include computing predictive completions of the context at step 218. Predictive completions may include one or more predictive completion lexical units selected from the output lexical vocabulary. In some examples, the supervised machine learning model may compute multiple predictive completions.
[0080] Steps 220 and 222 can be performed in an example where the supervised machine learning model is a classifier. At step 220, method 200 may further include computing a predicted classification of the output, context-conditioned, at the supervised machine learning model. For example, the supervised machine learning model may predict the output as harmful or harmless. In an example where the supervised machine learning model predicts multiple prediction completions, the supervised machine learning model may compute a corresponding predicted classification associated with each of those prediction completions.
[0081] In step 222, method 200 may further include selecting a constrained output lexicon vocabulary based at least in part on predicted classification. For example, if a predicted complement starting with a candidate output lexicon has a harmful probability higher than a predefined probability threshold, the decoder plugin may exclude that candidate output lexicon from the constrained output lexicon vocabulary. In some examples where the supervised machine learning model computes multiple predicted complements and corresponding predicted classifications, step 222 may include constructing a constrained output lexicon vocabulary from the set of candidate output lexicons that begin with the corresponding predicted complement and whose lexicon-specific predicted classification is lower than a predefined probability threshold. In such examples, the constrained output lexicon vocabulary is accordingly constructed from candidate output lexicons that are unlikely to lead to toxic output.
[0082] In some examples, the decoder plugin may include a supervised machine learning model that computes a predicted classification without computing contextual prediction completion. The predicted classification can be used to select a constrained output lexical vocabulary as described above, even without the additional information provided by prediction completion.
[0083] In some examples, reward prediction can be used instead of classification to select the constrained output lexical vocabulary. After calculating the prediction completion at step 218, at step 224, method 200 may further include calculating a reward value associated with the prediction completion. The reward value can be calculated at a reward model, which in some examples may be a machine learning model.
[0084] In the example of performing step 224, method 200 may further include: at step 226, selecting a constrained output lexicon vocabulary at least in part based on reward values. In the example where the decoder plugin generates multiple prediction completions, the candidate output lexicons selected for inclusion in the constrained output lexicon vocabulary may, for example, be candidate output lexicons that serve as the starting lexicon for prediction completions with reward values higher than a predefined reward threshold. As another example, a predefined number of highest reward values may be selected, and candidate output lexicons that begin the corresponding prediction completion may be selected for inclusion in the constrained output lexicon vocabulary.
[0085] Figure 9D Additional steps of method 200, which may be performed at the decoder plugin in some examples, are shown. At step 228, method 200 may also include performing a search algorithm over a predefined search domain. The predefined search domain may be, for example, the output lexical vocabulary, a subset of the output lexical vocabulary, or some other search domain, such as a numerical range. In some examples, at step 229, step 228 includes performing MCTS over multiple branches of the predefined search domain. In other examples, some other search algorithms may be used.
[0086] At step 230, method 200 may further include selecting a constrained output lexicon based at least in part on the search results returned by the search algorithm. For example, when the predefined search domain is the domain of candidate output lexicons, the search results may be identifiers of one or more candidate output lexicons to be included in or excluded from the constrained output lexicon. In other examples, search results, such as numerical values, may be post-processed in a manner that maps the search results to one or more candidate output lexicons.
[0087] Figure 9E Additional steps of method 200, which can be executed at the decoding module of the machine learning model, are shown. At step 232, method 200 may further include rescaling the output lexical probabilities to obtain constrained output lexical probabilities normalized over a constrained output lexical vocabulary. Thus, the output lexical probabilities can be adjusted to reflect the reduction in the output lexical vocabulary. At step 234, method 200 may further include selecting one or more output lexical units, at least in part, by sampling output lexical units from the constrained output lexical probabilities.
[0088] Figure 9F Additional steps of method 200 that can be performed during the decoder plugin generation phase are shown. During execution Figure 9F In the example of the steps, the decoder plugin generation phase is performed before the constrained output lexical vocabulary is computed. Method 200 may also include, at step 236, receiving decoder plugin-generated prompts. The decoder plugin-generated prompts may be specified, for example, via user input in natural language.
[0089] At step 238, method 200 may further include: generating prompt words at least partially based on the decoder plugin, and computing at least a portion of the decoder plugin at a machine learning model. For example, the machine learning model may populate one or more fields of a decoder plugin template. The machine learning model may accordingly generate bootstrapping logic from unstructured user input.
[0090] Using the systems and methods discussed above, the output vocabulary of the machine learning model is constrained at the decoder plugin to guide the generation of its output. For example, the systems and methods discussed above can be used to constrain the output vocabulary to provide safety controls for the machine learning model. Constraining the output vocabulary can also reduce illusions by reducing autoregressive drift. Furthermore, policy programming can be incorporated into the selection of the constrained output vocabulary by performing a search algorithm at the decoder plugin. Since programming is typically difficult for existing LLMs and LMMs, constraining the output vocabulary can increase the usability of machine learning models for tasks involving programming.
[0091] The decoder plugin used to compute the constrained output lexical vocabulary can be switched or modified to change the vocabulary constraint behavior without having to perform additional training on the machine learning model. Therefore, output lexical vocabulary constraints provide a flexible, low-cost option for controlling the output of a machine learning model in ways that are difficult to achieve with cue words.
[0092] The methods and processes described herein are associated with a computing system comprising one or more computing devices. In particular, such methods and processes may be implemented as computer applications or services, application programming interfaces (APIs), libraries, and / or other computer program products.
[0093] Figure 10 A non-limiting embodiment of a computing system 300 capable of performing one or more of the methods and processes described above is schematically illustrated. The computing system 300 is shown in a simplified form. The computing system 300 can embody the above description and... Figure 1 The computing system 10 is shown in the figure. The components of the computing system 300 may be included in one or more personal computers, server computers, tablet computers, home entertainment computers, network computing devices, video game devices, mobile computing devices, mobile communication devices (e.g., smartphones) and / or other computing devices, as well as wearable computing devices (such as smartwatches and head-mounted augmented reality devices).
[0094] The computing system 300 includes processing circuitry 302, volatile memory 304, and non-volatile storage device 306. The computing system 300 may optionally include a display subsystem 308, an input subsystem 310, a communication subsystem 312, and / or... Figure 10 Other components not shown.
[0095] Processing circuitry 302 typically includes one or more logic processors, which are physical devices configured to execute instructions. For example, a logic processor may be configured to execute instructions that are part of one or more applications, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform tasks, implement data types, change the state of one or more components, achieve technical effects, or otherwise achieve desired results.
[0096] The logic processor may include one or more physical processors configured to execute software instructions. Additionally or alternatively, the logic processor may include one or more hardware logic circuits or firmware devices configured to execute hardware-implemented logic or firmware instructions. The processor of processing circuitry 302 may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and / or distributed processing. Components of processing circuitry 302 may optionally be distributed across two or more separate devices that may be remotely located and / or configured for coordinated processing. For example, aspects of the computing system 300 disclosed herein may be virtualized and executed by remotely accessible networked computing devices configured in a cloud computing configuration. In this case, it will be understood that these virtualized aspects run on different physical logic processors on various different machines. These different physical logic processors on different machines will be understood to be collectively covered by processing circuitry 302.
[0097] The non-volatile storage device 306 includes one or more physical devices configured to store instructions executable by processing circuitry to implement the methods and processes described herein. When implementing such methods and processes, the state of the non-volatile storage device 306 can be changed—for example, to store different data.
[0098] Non-volatile storage device 306 may include removable and / or built-in physical devices. Non-volatile storage device 306 may include optical memory, semiconductor memory and / or magnetic memory, or other mass storage technologies. Non-volatile storage device 306 may include non-volatile, dynamic, static, read / write, read-only, sequential access, location-addressable, file-addressable, and / or content-addressable devices. It should be understood that non-volatile storage device 306 is configured to retain instructions even when the non-volatile storage device 306 is powered off.
[0099] Volatile memory 304 may include physical devices, including random access memory. Volatile memory 304 is typically used by processing circuitry 302 to temporarily store information during the processing of software instructions. It should be understood that when volatile memory 304 is powered off, it typically does not continue storing instructions.
[0100] The processing circuitry 302, the volatile memory 304, and the non-volatile storage device 306 can be integrated together into one or more hardware logic components. Such hardware logic components may include, for example, field-programmable gate arrays (FPGAs), application-specific integrated circuits (PASICs / ASICs), application-specific standard products (PSSPs / ASSPs), system-on-a-chip (SoCs), and complex programmable logic devices (CPLDs).
[0101] The terms "module," "program," and "engine" can be used to describe aspects of a computing system 300 typically implemented in software by a processor to perform specific functions using portions of volatile memory, involving transformation processing specifically configured to perform those functions. Therefore, a module, program, or engine can be instantiated via processing circuitry 302 that executes instructions held by non-volatile storage device 306 using portions of volatile memory 304. It should be understood that different modules, programs, and / or engines can be instantiated from the same applications, services, code blocks, objects, libraries, routines, APIs, functions, etc. Similarly, the same module, program, and / or engine can be instantiated from different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms "module," "program," and "engine" can encompass a single file or group of files such as executable files, data files, libraries, drivers, scripts, database records, etc. When included, the display subsystem 308 can be used to present a visual representation of the data stored by the non-volatile storage device 306. The visual representation may take the form of a graphical user interface (GUI). Since the methods and processes described herein change the data stored by the non-volatile storage device 306, and thus change the state of the non-volatile storage device 306, the state of the display subsystem 308 can also be changed to visually represent changes in the underlying data. The display subsystem 308 may include one or more display devices utilizing virtually any type of technology. Such display devices may be combined with the processing circuitry 302, the volatile memory 304, and / or the non-volatile storage device 306 in a shared housing, or such display devices may be peripheral display devices.
[0102] When included, the input subsystem 310 may include or be connected to one or more user input devices (such as a keyboard, mouse, touchscreen, camera, or microphone).
[0103] When included, the communication subsystem 312 can be configured to communicatively couple the various computing devices described herein to each other and to other devices. The communication subsystem 312 may include wired and / or wireless communication devices compatible with one or more different communication protocols. As a non-limiting example, the communication subsystem 312 can be configured to communicate via wired or wireless local area networks or wide area networks, broadband cellular networks, etc. In some embodiments, the communication subsystem 312 may allow the computing system 300 to send messages to and / or receive messages from other devices via a network such as the Internet.
[0104] The following paragraphs discuss several aspects of this disclosure. According to one aspect of this disclosure, a computing system is provided, including one or more processing devices configured to receive a cue word. At a machine learning model having an output lexicon including a plurality of candidate output lexicons, the one or more processing devices are further configured to compute a plurality of output lexicon probabilities on the output lexicon vocabulary, at least partially based on the cue word. At a decoder plug-in, the one or more processing devices are further configured to compute a constrained output lexicon vocabulary, the constrained output lexicon vocabulary being a proper subset of the output lexicon vocabulary. The one or more processing devices are further configured to select one or more output lexicons, at least partially based on the computed output lexicon probabilities. The one or more output lexicons are selected from candidate output lexicons included in the constrained output lexicon vocabulary. The one or more processing devices are further configured to transmit an output including the one or more output lexicons to an additional computing process. The above features can have the technical effect of guiding the output generated by the machine learning model. This output guidance can allow for additional control over the content of the output and can be used to reduce autoregressive drift.
[0105] According to this aspect, one or more processing devices can also be configured to compute lexicalized cue words, including multiple input lexical units, at least in part based on cue words. At the machine learning model, one or more processing devices can also be configured to compute output lexical probabilities in each of multiple autoregressive generation iterations. At the decoder plugin, one or more processing devices can also be configured to iteratively update a constrained output lexical vocabulary at each of the multiple autoregressive generation iterations, at least in part based on context, including lexicalized cue words and a previous output sequence, the previous output sequence including one or more previously output lexical units computed at a previous autoregressive generation iteration. These features can have the technical effect of modifying the way the output lexical sequence is guided during output generation.
[0106] According to this aspect, the decoder plugin may include a supervised machine learning model. One or more processing devices may also be configured to compute a predicted classification of the context-conditioned output at the supervised machine learning model. One or more processing devices may also be configured to select a constrained output lexical vocabulary, at least in part, based on the predicted classification. These features can have the technical effect of constraining the output lexical vocabulary in a flexible manner, taking into account complex behaviors that may occur at the machine learning model. For example, the supervised machine learning model can be used to perform scalable supervision for alignment purposes.
[0107] Accordingly, at the supervised machine learning model, one or more processing devices can also be configured to compute predictive completion of the context. One or more processing devices can also be configured to compute predicted classifications at least in part based on the predictive completion. The aforementioned features can have the technical effect of constraining the output lexicon vocabulary at least in part based on estimates of later output lexicons, as approximated at the supervised machine learning model.
[0108] According to this aspect, the decoder plugin may include a supervised machine learning model. One or more processing devices may also be configured to compute predictive completion of the context at the supervised machine learning model. One or more processing devices may also be configured to compute a reward value associated with the predictive completion. One or more processing devices may also be configured to select a constrained output lexical vocabulary based at least in part on the reward value. The above features can have the technical effect of constraining the output lexical vocabulary in a manner that estimates a high reward value according to a specified reward function.
[0109] Accordingly, at the decoder plugin, one or more processing devices can also be configured to modify one or more sampling parameters of the machine learning model. The aforementioned features can have the following technical effects: increasing the consistency or variability of output lexical units when sampling output lexical units from a constrained output lexical vocabulary, or modifying the sampling of output lexical units in some other way.
[0110] Accordingly, at the decoder plugin, one or more processing devices can also be configured to execute a search algorithm over a predefined search domain. One or more processing devices can also be configured to select a constrained output lexical vocabulary based at least in part on the search results returned by the search algorithm. These features can have the technical effect of utilizing the search algorithm as an additional mechanism to guide the output of a machine learning model.
[0111] Accordingly, when the search algorithm is executed at the decoder plugin, one or more processing devices can be configured to perform Monte Carlo Tree Search (MCTS) on multiple branches of a predefined search domain. This feature can have the technical effect of planning over the search domain of potential future outputs when a constrained output lexicon is selected.
[0112] Accordingly, at the decoder plugin, one or more processing devices can be configured to specify a constrained output lexical vocabulary using regular expressions or context-free grammars. This feature can have the technical effect of specifying the structure of output lexical units selected from the constrained output lexical vocabulary.
[0113] Accordingly, at the machine learning model, one or more processing devices can also be configured to rescale the output lexical probabilities to obtain constrained output lexical probabilities normalized over a constrained output lexical vocabulary. One or more processing devices can also be configured to select one or more output lexical units at least in part by sampling output lexical units from the constrained output lexical probabilities. These features can have the technical effect of adjusting the output lexical probabilities to account for changes in the output lexical vocabulary.
[0114] According to this aspect, during the decoder plugin generation phase performed before computing the constrained output lexical vocabulary, one or more processing devices may also be configured to receive decoder plugin-generated prompts. One or more processing devices may also be configured to compute at least a portion of the decoder plugin at a machine learning model based at least partially on the decoder plugin-generated prompts. The above features can have the technical effect of programmatically generating at least a portion of the decoder plugin specified by user input.
[0115] According to another aspect of this disclosure, a method for a computational system is provided. The method includes receiving a cue word. The method further includes, at a machine learning model having an output lexicon comprising a plurality of candidate output lexicons, calculating probabilities of a plurality of output lexicons on the output lexicon vocabulary, at least in part based on the cue word. At a decoder plugin, the method further includes calculating a constrained output lexicon vocabulary, the constrained output lexicon vocabulary being a proper subset of the output lexicon vocabulary. The method further includes selecting one or more output lexicons, at least in part based on the calculated output lexicon probabilities. The one or more output lexicons are selected from candidate output lexicons included in the constrained output lexicon vocabulary. The method further includes transmitting an output comprising one or more output lexicons to an additional computational process. The above features can have the technical effect of guiding the output generated by the machine learning model. This output guidance can allow for additional control over the content of the output and can be used to reduce autoregressive drift.
[0116] According to this aspect, at least in part based on the cue words, the method may further include computing lexicalized cue words comprising multiple input lexical units. At the machine learning model, the method may further include computing output lexical probabilities at each of multiple autoregressive generation iterations. At the decoder plugin, the method may further include iteratively updating a constrained output lexical vocabulary at each of the multiple autoregressive generation iterations, at least in part based on context, the context including the lexicalized cue words and a previous output sequence comprising one or more previously output lexical units computed at a previous autoregressive generation iteration. These features may have the technical effect of modifying the way the output lexical sequence is guided during output generation.
[0117] According to this aspect, the decoder plugin may include a supervised machine learning model. The method may also include computing a predicted classification of the output conditioned on the context at the supervised machine learning model. The method may further include selecting a constrained output lexical vocabulary based at least in part on the predicted classification. These features can have the technical effect of constraining the output lexical vocabulary in a flexible manner, taking into account complex behaviors that may occur at the machine learning model. For example, the supervised machine learning model can be used to perform scalable supervision for alignment purposes.
[0118] According to this aspect, the decoder plugin may include a supervised machine learning model. The method may also include computing predictive completion of the context at the supervised machine learning model. The method may further include computing a reward value associated with the predictive completion. The method may also include selecting a constrained output lexical vocabulary based at least in part on the reward value. The above features can have the technical effect of constraining the output lexical vocabulary in a manner that estimates high reward values according to a specified reward function.
[0119] Accordingly, at the decoder plugin, the method may further include performing a search algorithm over a predefined search domain. The method may also include selecting a constrained output lexical vocabulary based at least in part on the search results returned by the search algorithm. These features can have the technical effect of utilizing the search algorithm as an additional mechanism to guide the output of a machine learning model.
[0120] Accordingly, the method may further include: at the decoder plugin, using regular expressions or context-free grammars to specify a constrained output lexical vocabulary. This feature can have the technical effect of specifying the structure of output lexical units selected from the constrained output lexical vocabulary.
[0121] According to this aspect, the method may further include: at the machine learning model, rescaling the output lexical probabilities to obtain constrained output lexical probabilities normalized on a constrained output lexical vocabulary. The method may also include selecting one or more output lexical units, at least in part, by sampling output lexical units from the constrained output lexical probabilities. The above features can have the technical effect of adjusting the output lexical probabilities to account for changes in the output lexical vocabulary.
[0122] According to this aspect, the method may further include: receiving decoder plugin-generated prompts during a decoder plugin generation phase performed before computing a constrained output lexical vocabulary. The method may also include: computing at least a portion of the decoder plugin at a machine learning model based at least partially on the decoder plugin-generated prompts. These features can have the technical effect of programmatically generating at least a portion of the decoder plugin specified by user input.
[0123] According to another aspect of this disclosure, a computing system is provided, including one or more processing devices configured to receive a cue word. At a machine learning model having an output lexicon comprising a plurality of candidate output lexicons, the one or more processing devices are further configured to compute the probabilities of the plurality of output lexicons on the output lexicon at least in part based on the cue word. At a decoder plugin, the one or more processing devices are further configured to modify the corresponding output lexicon probabilities associated with the plurality of candidate output lexicons to obtain constrained output lexicon probabilities. The one or more processing devices are further configured to select one or more output lexicons at least in part based on the constrained output lexicon probabilities. The one or more processing devices are further configured to transmit an output comprising the one or more output lexicons to an additional computing process. The above features can have the technical effect of guiding the output generated by the machine learning model. This output guidance can allow for additional control over the content of the output and can be used to reduce autoregressive drift.
[0124] As used in this document, "and / or" is defined as inclusive or ∨, as specified in the following truth table:
[0125] It should be understood that the configurations and / or methods described herein are exemplary in nature, and these specific embodiments or examples should not be considered limiting, as many variations are possible. The particular routines or methods described herein can represent one or more of any number of processing strategies. Therefore, the various actions shown and / or described may be performed in the order shown and / or described, in another order, in parallel, or omitted. Similarly, the order of the above processes can be changed.
[0126] The subject matter of this disclosure includes all novel and non-obvious combinations and sub-combinations of the various processes, systems and configurations disclosed herein, as well as any and all equivalents thereof.
Claims
1. A computing system (10), comprising: One or more processing devices (12) are configured to: Receive prompt word (20); At a machine learning model (30) having an output lexicon (40) that includes multiple candidate output lexicons (42), the probabilities (34) of multiple output lexicons on the output lexicon are calculated at least in part based on the prompt words; At the decoder plugin (60), a constrained output lexical vocabulary (64) is computed, wherein the constrained output lexical vocabulary is a proper subset of the output lexical vocabulary; One or more output lexical units (52) are selected at least in part based on the calculated output lexical unit probabilities, wherein the one or more output lexical units are selected from the candidate output lexical units included in the constrained output lexical unit vocabulary; and The output (50) including one or more of the output tokens is transmitted to the additional computation process (54).
2. The computing system of claim 1, wherein the one or more processing devices are further configured to: Based at least in part on the prompt words, calculate lexicalized prompt words that include multiple input lexical units; At the machine learning model, the output lexical probability is calculated in each of the multiple autoregressive generation iterations; as well as At the decoder plugin, the constrained output lexical vocabulary is iteratively updated at each of the plurality of autoregressive generation iterations, at least in part based on context, the context including: The word meta-catalyst; as well as The previous output sequence includes one or more previous output terms computed at the previous autoregressive generation iteration.
3. The computing system according to claim 2, wherein: The decoder plugin includes a supervised machine learning model; and The one or more processing devices are further configured to: At the supervised machine learning model, a predicted classification of the output conditioned on the context is computed; and The constrained output lexical vocabulary is selected at least in part based on the predicted classification.
4. The computing system according to claim 3, wherein, At the supervised machine learning model, the one or more processing devices are further configured to: Calculate the predicted completion of the context; and The predicted classification is calculated at least in part based on the predicted completion.
5. The computing system according to any one of claims 2 to 4, wherein: The decoder plugin includes a supervised machine learning model; and The one or more processing devices are further configured to: At the supervised machine learning model, the predicted completion of the context is computed; Calculate the reward value associated with the predicted completion; and The constrained output lexicon is selected at least in part based on the reward value.
6. The computing system according to any one of claims 1 to 5, wherein, At the decoder plugin, the one or more processing devices are also configured to modify one or more sampling parameters of the machine learning model.
7. The computing system according to any one of claims 1 to 6, wherein, At the decoder plugin, the one or more processing devices are further configured to: Execute the search algorithm on a predefined search domain; and The constrained output lexical vocabulary is selected at least in part based on the search results returned by the search algorithm.
8. The computing system according to claim 7, wherein, When the search algorithm is executed at the decoder plugin, the one or more processing devices are configured to perform Monte Carlo Tree Search (MCTS) on multiple branches of the predefined search domain.
9. The computing system according to any one of claims 1 to 8, wherein, At the decoder plugin, the one or more processing devices are configured to specify the constrained output lexical vocabulary using regular expressions or context-free grammars.
10. The computing system according to any one of claims 1 to 9, wherein, At the machine learning model, the one or more processing devices are further configured to: Rescaling the output lexical probabilities yields constrained output lexical probabilities normalized on the constrained output lexical vocabulary; and The one or more output lexical units are selected at least in part by sampling the output lexical units probabilistically from the constrained output lexical units.
11. The computing system according to any one of claims 1 to 10, wherein, During the decoder plugin generation phase, which is performed before the computation of the constrained output lexical vocabulary, the one or more processing devices are further configured to: The receiver decoder plugin generates prompt words; as well as The prompt words are generated at least in part based on the decoder plugin, and at least part of the decoder plugin is computed at the machine learning model.
12. A method (200) for a computing system, the method comprising: Receive prompt word (202); At a machine learning model having an output lexicon that includes multiple candidate output lexicons, the probabilities of multiple output lexicons on the output lexicon are calculated at least in part based on the prompt words (204). At the decoder plugin, a constrained output lexical vocabulary is computed, wherein the constrained output lexical vocabulary is a proper subset of the output lexical vocabulary (206); One or more output lexical units are selected at least in part based on the calculated output lexical unit probabilities, wherein the one or more output lexical units are selected from the candidate output lexical units included in the constrained output lexical unit vocabulary (208); as well as The output, including one or more of the output tokens, is transmitted to the additional computation process (210).
13. The method of claim 12, further comprising: Based at least in part on the prompt words, calculate lexicalized prompt words that include multiple input lexical units; At the machine learning model, the output lexical probability is calculated in each of the multiple autoregressive generation iterations; as well as At the decoder plugin, the constrained output lexical vocabulary is iteratively updated at each of the plurality of autoregressive generation iterations, at least in part based on context, the context including: The word meta-catalyst; as well as The previous output sequence includes one or more previous output terms computed at the previous autoregressive generation iteration.
14. The method of claim 13, wherein: The decoder plugin includes a supervised machine learning model; and The method further includes: At the supervised machine learning model, a predicted classification of the output conditioned on the context is computed; and The constrained output lexical vocabulary is selected at least in part based on the predicted classification.
15. The method according to claim 13 or 14, wherein: The decoder plugin includes a supervised machine learning model; and The method further includes: At the supervised machine learning model, the predicted completion of the context is computed; Calculate the reward value associated with the predicted completion; and The constrained output lexicon is selected at least in part based on the reward value.
16. The method according to any one of claims 12 to 15, further comprising, at the decoder plug-in: Execute the search algorithm on a predefined search domain; and The constrained output lexical vocabulary is selected at least in part based on the search results returned by the search algorithm.
17. The method according to any one of claims 12 to 16, further comprising, at the decoder plugin, specifying the constrained output lexical vocabulary using regular expressions or context-free grammars.
18. The method according to any one of claims 12 to 17, further comprising, at the machine learning model: Rescaling the output lexical probabilities yields constrained output lexical probabilities normalized on the constrained output lexical vocabulary; and The one or more output lexical units are selected at least in part by sampling the output lexical units probabilistically from the constrained output lexical units.
19. The method of any one of claims 12 to 18, further comprising, during a decoder plugin generation phase performed prior to computing the constrained output lexical vocabulary: The receiver decoder plugin generates prompt words; and The prompt words are generated at least in part based on the decoder plugin, and at least part of the decoder plugin is computed at the machine learning model.
20. A computing system (10), comprising: One or more processing devices (12) are configured to: Receive prompt word (20); At a machine learning model (30) having an output lexicon (40) that includes multiple candidate output lexicons (42), the probabilities (34) of multiple output lexicons on the output lexicon are calculated at least in part based on the prompt words; At the decoder plugin (60), the corresponding output word probabilities associated with the plurality of candidate output words are modified to obtain constrained output word probabilities (70); One or more output lexical units are selected at least in part based on the constrained output lexical unit probabilities (52); and The output (50) including one or more of the output tokens is transmitted to the additional computation process (54).
