Device and method for predicting the next token

Abstract

A computer-implemented method for predicting the next token from a token sequence using a transformer-based machine learning system, comprising: processing a multi-layer embedding of the token sequence; determining a confidence score for individual tokens at the output of a layer by multiplying the output embedding of the layer by a weight matrix; determining, at a predetermined p-th layer, the K most probable next tokens; generating a truncated weight matrix by removing rows corresponding to other tokens; determining a confidence score for predefined subsequent layers for only the K tokens using the truncated weight matrix; and returning the token with the highest confidence score exceeding a layer-specific threshold.

Description

Technical field

[0001] The invention relates to a computer-implemented method for predicting the next token from a token sequence, a corresponding system, a computer program, and a machine-readable storage medium. State of the art

[0002] It has been shown that increasing the size of large language models (LLMs) leads to better performance of these machine learning systems. However, this comes at the cost of slower and more expensive inference. Early exiting, as proposed, for example, in https: / / arxiv.org / abs / 2207.07061, is a promising approach to improving the efficiency of LLM inference by enabling token predictions at intermediate layers. A key component of early exit models is the confidence score, which is calculated at each candidate exit and determines whether the current prediction is of sufficient quality to terminate the forward pass and return the early prediction. Disclosure of the invention

[0003] In its first aspect, the invention relates to a computer-implemented method for predicting the next token based on a token sequence using a transformer-based machine learning system. The machine learning system can receive a natural language input text, tokenize the input text from its vocabulary into an initial token sequence, and then iteratively predict subsequent tokens from its vocabulary and append them to the (thus evolving) token sequence. The evolving token sequence comprises the tokenized input text and the previously predicted and appended tokens. The evolving token sequence can be the given token sequence for predicting the next token according to the method described herein.Tokenization, in this context, can be understood as the process of breaking down texts into individual units called tokens for processing by the machine learning system. These tokens can be words, word fragments, or characters. Tokens can be selected or drawn from a vocabulary, where the vocabulary is the predefined set of all possible tokens that the machine learning system recognizes and uses. The output of the machine learning system is a new token, which is appended to the token sequence. The process can be performed iteratively, effectively extending the original input sequence. Each iteration of the procedure can add a new token. The iterative token generation process can be terminated, for example, when a maximum number of appended tokens has been reached or when another termination criterion is met.

[0004] The machine learning system can further determine an embedding of the token sequence and process this embedding through a plurality of layers. In the context of this description, a layer from the plurality of layers of the transformer-based machine learning system can be understood as a composite / modular block comprising several subcomponents. Such subcomponents may include, for example, attention layers, feedforward operations, residual connections, layer normalization, and / or convolution layers. Accordingly, the concept of a layer of the transformer-based machine learning system can be understood as a higher-level building block of the machine learning system. In particular, a layer (of the transformer-based machine learning system), as mentioned herein, may comprise at least one multi-head attention block and / or one feedforward block.

[0005] An embedding of the token sequence can include the embeddings, i.e., numerical (vector) representations of each of the elements of the token sequence.

[0006] By embedding elements of a token sequence, the respective discrete text units are mapped into a continuous vector space that can, for example, capture semantic and contextual relationships between the tokens. This vector representation can then be processed by the layers of the transformer-based model.

[0007] At the output of specific, predetermined layers, a confidence value or confidence score is determined for individual tokens from the token vocabulary. The confidence value for a single token results from a corresponding entry in a confidence value vector, which is determined by multiplying the output embedding of the specific layer by a weight matrix. The weight matrix can be learned during the training of the machine learning system. In some embodiments, a confidence value can be determined at the output of each layer, so that each layer in these embodiments can be a "specific, predetermined layer." In other embodiments, however, specific but not all layers can be predetermined, and only for these predetermined layers can a confidence value for individual tokens be determined at the output of the respective layer.In other embodiments, for example, every second or third layer can be a specific, predetermined layer. In such cases, the overall computational cost for determining confidence levels can be reduced, since a confidence level might not need to be determined at the output of every layer. In other words, a "specific" layer in this context can refer to a layer that was predetermined during the design or setup of the machine learning system's architecture; which layers from the majority of the machine learning system's layers are selected / determined as specific layers may depend on the design choices and / or the overall computational cost.

[0008] The confidence score can be understood as a measure of how certain the machine learning system is that a given token is the next token.

[0009] The procedure comprises the following steps: For a predetermined p-th layer of the machine learning system, the K tokens that are most likely to be the next token are determined, based on their respective confidence values ​​calculated from the output of the p-th layer. That is, the K tokens corresponding to the elements of the confidence value vector with the K highest values ​​are identified. In a subsequent step, a truncated weight matrix is ​​derived from the weight matrix by removing all rows except those corresponding to the determined K most probable tokens. By removing, i.e., deleting, all rows except those corresponding to the K identified tokens, the dimension of the truncated weight matrix can be drastically reduced, depending on the specific choice of K. For example, if the vocabulary exceeds 30,000 or even 50, the dimensionality of the truncated weight matrix can be significantly reduced.Since the number of tokens is 000, choosing a value between one and several hundred for K for computational steps that include the truncated weight matrix instead of the (untruncated) weight matrix leads to an extreme reduction in complexity (in terms of time and memory resources).

[0010] In a next step, for specific layers after the p-th layer, a confidence value is determined for only the specified K most probable tokens. This can be done by determining a truncated confidence value vector, which is calculated by multiplying the output embedding of the respective layer by the truncated weight matrix. Similarly, as above, the layers after the p-th layer for which the confidence value (now: the confidence value for only the specified K most probable tokens) is determined can include all subsequent layers up to a final layer, or, in other embodiments, only certain predetermined subsequent layers.Finally, the token corresponding to the highest confidence value in the truncated confidence vector obtained from the output embedding of a given layer can be returned as the next token if this highest confidence value exceeds a predefined layer-specific threshold for that layer. Specifically, if this highest confidence value exceeds the predefined layer-specific threshold for that layer, the processing of a layer's output embedding by all subsequent layers can be stopped, aborted, or not started at all. The returned next token can then be appended to the (resulting) token sequence, and the steps for predicting a next token based on the (evolving) token sequence can begin again.Note that a layer-specific threshold can be given by a common value for all layers of the machine learning system, which in this case denotes a predefined threshold. The predefined (layer-specific) threshold can be determined during the training of the machine learning system. For example, a possible value for a (layer-specific) threshold could be slightly less than 1, such as 0.9, which would mean that the model should be very confident in its prediction before issuing the next token. However, a lower threshold, such as 0.5, could also be chosen. Certain values ​​may have been empirically determined during the training and fine-tuning of the machine learning system based on desired performance characteristics. In other implementations, calibration of the confidence score may not be mandatory.Instead, the confidence score can only enable an appropriate ranking by weighting more likely tokens higher than less likely ones. Then, for example, a value for a (layer-specific) threshold can be set accordingly, based on a validation dataset.

[0011] It should also be noted that the embeddings of all elements of the token sequence can be processed by the machine learning system, but only the embedding corresponding to the last token in the token sequence can be used in the mapping with the weight matrix or the truncated weight matrix.

[0012] Advantageously, the proposed method increases the efficiency of transformer-based token prediction by removing less probable tokens from consideration in later layers. This reduces the computational overhead, i.e., saves computational costs, without significant loss of accuracy, and enables faster and less resource-intensive text generation. In other words, the method described here enables an efficient estimation of confidence at each layer's output. Considering the mapping of each layer's output embeddings (hidden representations) to the full token space (i.e., the vocabulary), this would impose a significant overhead on the forward pass of the transformer-based machine learning system.

[0013] Furthermore, the method described here can advantageously accelerate the inference of any machine learning system to predict the next token, i.e., a large language model (LLM), by dynamically pruning the weight matrix that assigns the hidden representation to the logits, previously referred to as token scores, over tokens.

[0014] While the large vocabulary size in modern LLMs can make the confidence estimation required for early exit decisions computationally intensive, thus diminishing the realized efficiency gains, the procedure proposed here can help overcome / mitigate this problem. An early exit decision can be described as the decision to return a token predicted by the output of an intermediate layer as the next token, without then continuing the forward pass of the machine learning system.

[0015] The method described here is in particular independent of the exact architecture or training style of the machine learning system or the input text that is provided to / received by the machine learning system.

[0016] Preferably, the confidence vector or truncated confidence vector can be determined as the softmax of the product of the weight matrix or truncated weight matrix and the output embedding of the respective layer. In particular, the weight matrix / truncated weight matrix can map the output embedding of a layer to a vector of token scores. The token scores (sometimes called logits) can be interpreted or described as "raw," unnormalized scores for each token / the K most probable tokens in the vocabulary. The token scores are then transformed by a softmax function into a probability distribution over the vocabulary, representing the confidence values ​​for each token.

[0017] The weight matrix may have been determined during the training of the machine learning system as an unembedding matrix in the last layer of the machine learning system. Note that the softmax function can transform a vector of raw token scores (logits) into a probability distribution, where each element represents the probability of a corresponding token, and all elements sum to one.

[0018] A definition of the (unit) softmax function can be given as follows: softmax(hkl)i=ehkil∑jdmodelhk jl where hkl∈ℝdmodel stands for embedding.

[0019] Note that p and K can take values ​​defined by natural numbers that are smaller than the total number of layers and the total number of tokens in the machine learning system's vocabulary, respectively. Accordingly, p and K are hyperparameters and can be defined / chosen / set by a user or determined in advance by another algorithmic method.

[0020] Preferably, the layers in the majority of layers of the transformer-based machine learning system are decoder layers. Each decoder layer can include a multi-head attention block and a feedforward block.

[0021] Preferably, the values ​​of p and K can be optimized by evaluating the machine learning system against a validation dataset with different values ​​of p and K, and by selecting the values ​​of p and K that maximize a performance metric. In this context, the performance metric can measure the deviation between an output of the machine learning system and a corresponding element of the validation dataset.

[0022] Advantageously, such optimization of p and K based on a performance metric that measures the deviation between output and validation data can lead to improved accuracy and efficiency in predicting the next token, especially by adapting the cleaning strategy to the specific characteristics of the data, thereby reducing computational costs without significant loss of prediction performance.

[0023] Preferably, the process steps can be executed by the processor of a device, wherein the value of K and / or the value of p are determined based on the device's specific hardware resources. The hardware resource may include available processing power and / or available memory. The device may, for example, be a resource-constrained device. Such a device may be limited in its computing capacity, which can be measured in FLOPS (floating-point operations per second), available RAM (random access memory), and / or limited power consumption in watts. In particular, such a device may require optimized algorithms and possibly reduced model sizes to accommodate resource constraints.Accordingly, a smaller K-value and / or a smaller p-value can be chosen for devices with more limited resources, and a larger K-value and / or a larger p-value can be chosen for devices with more resources. In particular, the p and / or K values ​​can be determined to optimize performance within the limitations of the device. Examples of devices with limited resources include a mobile device, a personal computer or laptop, and an electronic control unit in a vehicle, a robot, or a manufacturing machine.

[0024] Advantageously, adapting p and K to the specific, available hardware resources can enable efficient execution of the next-token prediction method on a wider range of specific devices, including resource-constrained devices, by balancing prediction accuracy with computational costs and memory usage. This can facilitate deployment on devices with limited processing power, memory, or battery life.

[0025] Preferably, the machine learning system can be part of a multimodal machine learning system that receives sensor data recorded by a suitable sensor and, optionally, text data as input. Sensor data can be recorded by a suitable sensor, such as a (video) image sensor, e.g., a camera, a radar, LiDAR, temperature, or ultrasonic sensor. The multimodal machine learning system can process the sensor data based on low-level features, e.g., edges or pixels of (video) images. The machine learning system can then be instructed (i.e., instructed via input text) to generate a text classification of the received sensor data using the next token prediction method based on a token sequence. An actuator of a robot, a manufacturing machine, or a part of a manufacturing machine can be controlled based on the generated text classification.

[0026] A text classification can include a text label indicating whether the received sensor data meets a predefined criterion within predefined limits, with the label being either "OK" or "Not OK". Alternatively, the text classification can include a description of the sensor data's properties. In this case, the machine learning system may have been instructed accordingly. In other words, the token prediction algorithm can generate a sequence of tokens that form a text description of the sensor data. This description can then be compared to predefined criteria for the states "OK" and "Not OK". If the generated description meets the "OK" criteria, the classification is "OK"; otherwise, it is "Not OK". An actuator could be, for example, a motor, a robot arm, or a valve. In an example case where the generated text classification is "OK", the actuator (e.g.,The robot arm can continue its programmed operation. If, however, the classification in this example is "Not OK," the actuator can be instructed to stop or perform a corrective action. Such control logic can be implemented, for example, by a control system that receives the text classification as input and triggers corresponding commands to the actuator.

[0027] Note that the above-mentioned configuration can be implemented, for example, by using a CLIP (Contrastive Language Image Pretraining) similar framework as a multimodal machine learning system, where an image (more generally: sensor data) encoder component of the multimodal machine learning system could generate an embedding, and the text encoder could be implemented by the machine learning system described herein, which could generate the most probable label considering the (embedding of) image (more generally: sensor) data, i.e., the tokens "OK" or "Not OK".

[0028] According to another aspect, the invention relates to a system comprising a processor configured to perform a method for predicting a next token as described herein.

[0029] According to another aspect, the invention relates to a computer program comprising machine-readable instructions which, when executed by a computer, cause the computer to perform one of the computer-implemented methods described above and below. Furthermore, according to another aspect, the invention relates to a machine-readable storage medium on which the above computer program is stored.

[0030] Embodiments of the invention are discussed in more detail with reference to the following figures. The figures show: Fig. 1 a flowchart of an exemplary embodiment; Fig. 2 a flowchart of an exemplary embodiment. Description of the embodiments

[0031] For illustration, in the following description of the embodiments Y, the vocabulary space of the machine learning system is defined as having the size |Y| = d. vocabFurthermore, let (x1,...,x t ), x i ∈ Y the given token sequence, which includes both the tokenized input text and the tokens predicted and appended so far.

[0032] Fig. Figure 1 shows a flowchart of the process steps according to one embodiment. In step 100, the machine learning system 2 receives (with reference to Fig. 2) The input text 10 in natural language. This input text is tokenized into an initial token sequence from the machine learning system's vocabulary, and the machine learning system 2 iteratively predicts the next tokens from its vocabulary and appends them to the evolving token sequence. In particular, the initial token sequence can be represented by (x1, x2, ..., x i ) be given, and after (some / several) iteration steps, the developing token sequence can be represented by (x1,x2, ..., x i , x i+1 , ..., x t) may be given. Note that for the procedure described here, the sequence may be the same as the initial sequence or may be given by the sequence after several iterations. For the purpose of notation, the sequence will be denoted below by (x1,x2, ...,x). t ) denotes, and t = i or t > i.

[0033] In the next step, the machine learning system determines an embedding of the token sequence and processes this embedding through a plurality of layers. Layers 1, p, p+1, j, and L are examples in Fig. Figure 2 illustrates this. A confidence value for individual tokens from the vocabulary is then determined in step 300 at the output of specific layers. The confidence value for a single token is represented by a corresponding entry in the confidence value vector. ctl(ctk) given, which is obtained by multiplying the initial embedding htl(htk) the layer with the weight matrix W p (Wp+1 ) is determined, see also Fig. 2. In process step 400, for the predetermined layer p, the K tokens are determined which, based on their respective confidence values, are most likely to be the next token, where the confidence value is from the initial embedding. (htl,) The layer p is calculated. In the subsequent step 500, the trimmed weight matrix W is calculated. p+1 from the weight matrix W p determined by W in the weight matrix p All rows except those corresponding to the determined K most probable tokens are removed. For layers after layer p, a confidence value is calculated in step 600 only for the determined K most probable tokens. This is done by determining the truncated confidence value vector. ctk, which is achieved by multiplying the output embedding htk the respective layer with the trimmed weight matrix W p+1is determined. In step 700, the token corresponding to the highest confidence value in the truncated confidence value vector is selected as the next token x. t+1 returned if this highest confidence value exceeds a predefined layer-specific threshold for that layer (layer j in Fig. 2).

[0034] Optionally, the values ​​of p and K are optimized in steps 100a and 100b. In step 100a, the machine learning system is evaluated using a validation dataset with different values ​​of p and K. Subsequently, in step 100b, those values ​​of p and K are selected that maximize a performance metric.

[0035] With further reference to Fig. Figure 2 presents an embodiment of the method in a flowchart. The machine learning system 2 comprises a Transformer architecture, see https: / / arxiv.org / abs / 1706.03762. In general, in a Transformer model, the input sequence is passed through L layers, each consisting of a multi-head attention block and a feedforward block, resulting in a sequence of hidden representations. {htl}l=1L results, with htl∈ℝdmodel, where d model This refers to the dimension of the embedding space. After processing by all layers, the final next token distribution can be determined via p(xt+1|htL)=softmax(WhtL) This distribution is a vector that relates to Fig. 2 also with ctL can be denoted as W ∈ ℝ dvocab×dmodelIn a transformer-based machine learning system, this can generally refer to the weight matrix, also known as the unembedding matrix, which represents the final hidden state. htL can project back into the token space Y. The newly predicted token x t+1 is then added to the input sequence, and the (autoregressive) generation process can be repeated until a predetermined termination criterion is met.

[0036] Note that when determining p(xt+1|htL)=softmax(WhtL) only the embedding htL It can be taken into account that refers to the last token x t in the token sequence. This may imply the assumption that knowledge about the further, preceding tokens in the token sequence is conveyed through the attentional mechanism in the hidden representation (i.e., embedding) of the last token. htL is coded.

[0037] The machine learning system 2 in Fig. 2 can be the next token x t+1 The system can predict and return values ​​at an intermediate layer j, a process also known as early exit, if the machine learning system is sufficiently confident. A confidence value for individual tokens can be determined by the corresponding entry in the confidence value vector. ctl can be defined, where the index l refers to the respective layer. A criterion for premature exit from layer j can then be defined by max(ctj)>λtj be given, whereby ltj denotes the level-specific threshold. Note that in the case of machine processing... ctj=p(xt+1|htj)=softmax(Wp+1htj) For the first p layers (l = 1,···,p), the confidence value vector is calculated in machine learning system 2. ctl through ctl=p(xt+1|htl)=softmax(Wphtl) determined. From the confidence vector ctp, i.e., the softmax of the product of weight matrix W p and shift issue htp, The K tokens most likely to be the next token are determined as the K tokens with the highest probabilities according to their respective entries in the confidence vector. For subsequent layers, the truncated weight matrix W is used when calculating the confidence value vectors. p+1 used, which are derived from the weight matrix W p is determined by removing all rows except those corresponding to the given K most probable tokens. By choosing K << d vocabThe cost of confidence estimation is significantly reduced, while performance has been observed to remain roughly the same, which can optionally be further optimized by adjusting p. From a computational cost savings perspective, a smaller value of p may be preferable; however, it must then be ensured that the next token to be predicted is indeed among the K most probable tokens with sufficient probability. The combination of dynamic early exit after layer j, depending on the confidence that a token is the next token to cross a threshold, together with the truncated and thus much smaller weight matrix (from a certain layer onward), can lead to a significant improvement in terms of required memory and processing power.

[0038] Finally, it should be noted that a plural can generally be understood as indexed, a concept used throughout this text. That is, each element of the plural can be assigned a unique index, preferably by assigning consecutive integers to the elements contained in the plural. If a plural comprises N elements, where N is the number of elements in the plural, the elements are preferably assigned the integers from 1 to N. It can also be understood that elements of the plural can be accessed via their index. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited non-patent literature

[0000] https: / / arxiv.org / abs / 2207.07061

[0002] https: / / arxiv.org / abs / 1706.03762

[0035]

Claims

[1] Computer-implemented method for predicting the next token (x t+1 ) for a given token sequence by a transformer-based machine learning system (2), wherein the machine learning system receives a natural language input text (10), tokenizes the input text from its vocabulary into an initial token sequence, and iteratively predicts next tokens from its vocabulary and adds them to the evolving token sequence (x1,x2, ...,x t ) adds, wherein the machine learning system determines an embedding of the token sequence (200) and processes this embedding through a plurality of layers (1, p, p+1, j, L), wherein a layer from the plurality of layers includes at least one multi-head attention block and / or one feedforward block, where a confidence value for individual tokens from the vocabulary is determined at the output of specific layers (300), where the confidence value for a single token (x t+1 ) by entering a corresponding confidence value vector (ctl) is given by multiplying the output embedding (htl,htk) the respective specific layer with a weight matrix (W p , W p+1 ) is determined, characterized by that the procedure includes the following steps: - Determine (400), for a predetermined p-th layer, the K tokens that are most likely to be the next token, based on their respective confidence values ​​derived from the output (htl,) the p-th layer is calculated - Determining (500) a truncated weight matrix (W p+1 ) from the weight matrix (W p+1 ), by including in the weight matrix (W p) all lines except those corresponding to the given K most probable tokens are removed, - Determine (600), for specific layers (p+1, j, L) following the p-th layer (p), a confidence value only for the determined K most probable tokens by a truncated confidence value vector (c'tk) is determined by multiplying the output embedding (htk) the respective layer with the trimmed weight matrix (W p+1 ) is determined, - Return (700) the token corresponding to the highest confidence value in the truncated confidence value vector as the nearest token (x t+1 ), if the highest confidence value exceeds a predefined layer-specific threshold for that layer. [2] Method (1000) according to claim 1, wherein the confidence value vector / truncated confidence value vector is determined as the softmax of the product of the weight matrix / truncated weight matrix and the output embedding of the respective layer. [3] Method (1000) according to any of the preceding claims, wherein p and K take on values ​​defined by natural numbers that are smaller than the total number of layers (L) and the total number of tokens of the vocabulary of the machine learning system, respectively. [4] Method (1000) according to any one of the preceding claims, wherein the layers in the plurality of layers (1, p, p+1, j, L) are decoder layers, each decoder layer comprising a multi-head attention block and a feedforward block. [5] Method (1000) according to any one of the preceding claims, wherein the values ​​of p and K are optimized by - Evaluating (100a) the machine learning system using a validation dataset with different values ​​of p and K, and - Selecting (100b) the values ​​of p and K that maximize a performance metric. [6] Method (1000) according to any of the preceding claims, wherein the method steps are performed by the processor of a device, wherein the value of K and / or the value of p is determined based on the specific hardware resources of the device. [7] Method (1000) according to one of the preceding claims, wherein the machine learning system (2) is part of a multimodal machine learning system which receives sensor data recorded by a corresponding sensor and optionally text data as input, wherein the machine learning system (2) generates a text classification of the received sensor data using the method for predicting a next token based on a token sequence, and wherein an actuator of a robot, a manufacturing machine or part of a manufacturing machine is controlled based on the generated text classification. [8] Data processing system comprising a processor configured to perform the method (1000) according to any one of claims 1 to 7. [9] Computer program comprising instructions which, when the program is executed by a computer, cause the computer to execute the method according to any one of claims 1 to 7. [10] Computer-readable data carrier on which the computer program according to claim 9 is stored

Images