Apparatus and method for distilling attention score from full-precision into boolean in machine learning transformer

Abstract

An apparatus configured to compute a self-attention score to be used in a transformer is disclosed. The apparatus includes a query projector configured to determine query data based on an input and a key projector configured to determine key data based on the input. The apparatus further includes a value projector configured to determine value data based on the input, a module configured to compute a cross-relation score based on the query data and the key data. The query data, key data, value data and cross-relation score are real-valued tensors. The apparatus further includes a parameter projector configured to determine learnable parameters. Furthermore, the apparatus includes a binarization module configured to binarize the cross-relation score based on the learnable parameters. The apparatus further includes an output module configured to compute an output based on the value data and the binarized cross-relation score.

Description

[0001] APPARATUS AND METHOD FOR DISTILLING ATTENTION SCORE FROM FULL-PRECISION INTO BOOLEAN IN MACHINE LEARNING TRANSFORMER TECHNICAL FIELD

[0002] The present disclosure relates to machine learning transformers, specifically in the field of attention mechanisms used in machine learning transformers. Moreover, the present disclosure relates to an apparatus and a method for distilling attention scores from full precision into Boolean in machine learning transformers.

[0003] BACKGROUND

[0004] Machine learning transformers are widely used in various applications, such as natural language processing, machine translation, and text generation. The machine learning models rely heavily on self-attention mechanisms to calculate relationships between elements in input data, expressed as attention scores. Attention scores guide the model in determining which parts of the input data are most relevant to the task at hand, such as translating text or generating coherent sentences. The computation of the attention scores involves processing large amounts of real- valued tensors, leading to high computational complexity. The complexity increases significantly with the size of the input data, which is a common challenge in machine learning tasks involving large datasets. To address issue of size, efforts have been made to simplify these operations, particularly by converting real-valued attention scores into binary or Boolean representations. However, existing methods struggle with performance loss, as binarization often discards essential information from the full-precision data.

[0005] Therefore, in light of the foregoing discussion, there exists a need to overcome the aforementioned drawbacks associated with the attestation of mechanisms in the machine learning transformer.

[0006] SUMMARY

[0007] The present disclosure provides an apparatus configured to compute a self-attention score to be used in a transformer and a method for determining the self-attention score in the transformer. The present disclosure provides a solution to the existing problem of inaccurate attention mechanisms in transformer models, which may result in suboptimal performance in tasks such as natural language processing, computer vision, and other applications requiring complex pattern recognition. An objective of the present disclosure is to provide a solution that overcomes at least partially the problems encountered in the prior art and provides an improved apparatus and method for computing self-attention scores.

[0008] One or more objectives of the present disclosure are achieved by the solutions provided in the enclosed independent claims. Advantageous implementations of the present disclosure are further defined in the dependent claims.

[0009] In one aspect, the present disclosure provides an apparatus configured to compute a self-attention score to be used in a transformer. The apparatus includes a query projector configured to determine query data based on an input. Further, the apparatus includes a key projector configured to determine key data based on the input and a value projector configured to determine value data based on the input. The apparatus further includes a module configured to compute a cross-relation score based on the query data and the key data. The query data, key data, value data and cross-relation score are real-valued tensors. The apparatus further includes a parameter projector configured to determine learnable parameters. Furthermore, the apparatus includes a binarization module configured to binarize the cross-relation score based on the learnable parameters and an output module configured to compute an output based on the value data and the binarized cross-relation score. By utilizing real-valued tensors for the query data, key data, and value data, the apparatus ensures that initial computations maintain high precision. However, the parameter projector, which determines learnable parameters used to binarize the crossrelation score. The binarization reduces the complexity of the subsequent calculations by converting real-valued data into a binary format, thus decreasing the need for resource-intensive operations like floating-point multiplications. The binarization also enables faster processing with lower memory usage, making the apparatus highly effective in environments where computational efficiency is crucial, such as large-scale natural language processing models or real-time data processing tasks. Furthermore, despite the reduction in computational load, the apparatus retains significant performance benefits by using learnable parameters, which enhance the accuracy of the binarized output. The integration of these components allows the transformer to perform complex tasks more efficiently without compromising overall model performance.

[0010] In an implementation form, the parameter projector is configured to determine the learnable parameters (T) based on the input. The parameter projector dynamically adapts to the input data. The adaptability allows the apparatus to fine-tune the binarization process, making the conversion of real-valued data into binary form more accurate and context-sensitive. By basing the learnable parameters on the input, the apparatus ensures that the threshold used for binarization is optimal for each specific data set, reducing information loss during the conversion. The reduction in information loss results in improved overall performance and accuracy in the transformer, as the apparatus is able to maintain key relationships between data elements while benefiting from the computational efficiency of binary processing.

[0011] In an implementation form, the parameter projector is configured to determine the learnable parameters (T) based on the crossrelation score.

[0012] Advantageously, by basing the threshold on the actual computed attention score, the binarization may be more closely aligned with the specific characteristics of the data being processed. The context-sensitive approach allows the binarization module to retain more critical information from the original real-valued score, thereby reducing the performance gap typically associated with binarization.

[0013] In an implementation form, the learnable parameters are a threshold utilized by the binarization module.

[0014] By making the threshold the learnable parameter, the apparatus may dynamically adjust the binarization process based on the specific characteristics of the input data during training. The adaptability allows the binarization to be fine-tuned for optimal performance, preserving more relevant information from the original real-valued data. As a result, the binary representation is more accurate, leading to improved model performance and reduced information loss compared to using a fixed threshold. This flexibility enhances the overall efficiency and effectiveness of the transformer, especially in resource-constrained environments where computational power is limited.

[0015] In an implementation, the binarization module is configured to utilize the threshold for binarizing the cross-relation score (A) such that the threshold (T) is broadcasted the second dimension of the cross-relation score (A).

[0016] By broadcasting the threshold across the second dimension, the binarization process is uniformly applied to each element along that dimension, ensuring consistent treatment of data across all sequences or features. The binarization process is particularly advantageous in scenarios where input data dimensions vary or when working with high-dimensional tensors common in transformers. It ensures that the binarization is adapted to the specific structure of the data, leading to a more accurate binary representation.

[0017] In an implementation form, the parameter projector is a linear layer of weight parameters of shape [d, 1 , N] and when the input data is of dimensions [L, d, N], the resulting learnable parameters is of dimensions [L, 1, N], By having the learnable parameters vary along the sequence length and input sequence number, the parameter projector may adapt to different data sequences independently. The adaptability enables the model to capture and utilize context-specific information, leading to more precise and effective binarization for each unique sequence. The parameter projector may learn complex relationships between the dimensions of the input data without introducing excessive computational complexity. The balance between flexibility and efficiency is crucial in maintaining high performance while reducing the burden on computational resources.

[0018] In an implementation form, N is a number of input sequences, L is a length of each input sequence, and d is an embedding size. By converting the input sequences into binary format, the apparatus can perform computations and operations more efficiently. The binary representation simplifies the processing of the input sequences and enables the apparatus to handle large amounts of data more effectively. Additionally, the use of the threshold allows for fine-grained control over the binarization process, ensuring an accurate representation of the input data. The technical effect of this approach is the conversion of the input data into binary values.

[0019] In an implementation form, a projector is a learnable deep learning layer that computes data from its input data.

[0020] Advantageously, by incorporating the learnable deep learning layer as the projector, is the ability to capture and process complex relationships within the input data. The ability of the projector to capture complex relationships enhances the performance and accuracy of the apparatus in various machine learning tasks. The projector's computation of data from the input data enables the apparatus to generate informative representations of the relationships between terms, which are essential for effective learning and processing within the apparatus. This technical effect makes the apparatus highly suitable for products and systems that utilize or require machine learning transformers.

[0021] In an implementation form, the projector is implemented by a fully connected layer or a 1 -dimensional convolutional layer. In such an implementation form, implementing the projector with a fully connected layer or a 1 -dimensional convolutional layer enables the apparatus to compute the attention score efficiently. This allows for effective processing of the input data and accurate computation of the attention mechanism, which is crucial in machine learning transformer systems. By utilizing these specific layers, the apparatus can achieve improved performance and optimize the trade-off between complexity and efficiency in the self-attention module.

[0022] In an implementation form, the apparatus is a controller, or a computing device configured to execute a transformer.

[0023] The utilization of binarization structures leading to a reduction in computational complexity. As a result, the apparatus enhances the performance and efficiency of machine learning transformers that employ this technique, making them more practical and feasible for various applications that require or utilize machine learning transformers.

[0024] In another aspect, the present disclosure provides a method for determining self-attention score in a transformer. The method includes determining query data based on an input. The method further includes determining key data based on the input. The method further includes determining value data based on the input. Furthermore, the method includes computing a crossrelation score based on the query data and the key data. The query data, key data, value data and cross-relation score are real-valued tensors. The method further includes determining the learnable parameters and binarizing the cross-relation score based on learnable parameters. Further, the method includes computing an output based on the value data and the binarized crossrelation score. The method for determining self-attention score in a transformer achieves all the advantages and technical effects of the apparatus of the present disclosure.

[0025] In another aspect, the present disclosure provides a computer program product comprising program instructions for performing the method when executed by one or more processors in a transformer.

[0026] It is to be appreciated that all the aforementioned implementation forms can be combined.

[0027] It has to be noted that all devices, elements, circuitry, units, and means described in the present application could be implemented in the software or hardware elements or any kind of combination thereof. All steps which are performed by the various entities described in the present application as well as the functionalities described to be performed by the various entities are intended to mean that the respective entity is adapted to or configured to perform the respective steps and functionalities. Even if, in the following description of specific embodiments, a specific functionality or step to be performed by external entities is not reflected in the description of a specific detailed element of that entity which performs that specific step or functionality, it should be clear for a skilled person that these methods and functionalities can be implemented in respective software or hardware elements, or any kind of combination thereof. It will be appreciated that features of the present disclosure are susceptible to being combined in various combinations without departing from the scope of the present disclosure as defined by the appended claims.

[0028] Additional aspects, advantages, features, and objects of the present disclosure would be made apparent from the drawings and the detailed description of the illustrative implementations construed in conjunction with the appended claims that follow. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The summary above, as well as the following detailed description of illustrative embodiments, is better understood when read in conjunction with the appended drawings. For the purpose of illustrating the present disclosure, exemplary constructions of the disclosure are shown in the drawings. However, the present disclosure is not limited to specific methods and instrumentalities disclosed herein. Moreover, those in the art will understand that the drawings are not to scale. Wherever possible, like elements have been indicated by identical numbers.

[0030] Embodiments of the present disclosure will now be described, by way of example only, with reference to the following diagrams wherein:

[0031] FIG. 1 is a block diagram of an apparatus configured to compute a self-attention score to be used in a transformer, in accordance with an embodiment of the present disclosure;

[0032] FIG. 2 is a flowchart depicting a method for determining self-attention score in a transformer, in accordance with an embodiment of the present disclosure;

[0033] FIG. 3 is an exemplary diagram depicting dimensions of the processed data tensors, in accordance with an embodiment of the present disclosure; and

[0034] FIG. 4 is an exemplary diagram depicting an apparatus configured to compute a self-attention score to be used in a transformer in another configuration, in accordance with another embodiment of the present disclosure.

[0035] In the accompanying drawings, an underlined number is employed to represent an item over which the underlined number is positioned or an item to which the underlined number is adjacent. A non-underlined number relates to an item identified by a line linking the non-underlined number to the item. When a number is non-underlined and accompanied by an associated arrow, the non-underlined number is used to identify a general item at which the arrow is pointing. DETAILED DESCRIPTION OF EMBODIMENTS

[0036] The following detailed description illustrates embodiments of the present disclosure and ways in which they can be implemented. Although some modes of carrying out the present disclosure have been disclosed, those skilled in the art would recognize that other embodiments for carrying out or practicing the present disclosure are also possible.

[0037] FIG. 1 is a block diagram that depicts an apparatus configured to compute a self-attention score to be used in a transformer, in accordance with an embodiment of the present disclosure. With reference to FIG. 1, there is shown a block diagram 100 that includes an apparatus 102. The apparatus 102 includes a query projector 106, a key projector 108, a value projector 110, a module 112, a parameter projector 114, a binarization module 116 and an output module 118.

[0038] The apparatus 102 is a specialized computing device or controller that serves as the execution environment for transformer models incorporating an innovative binarization technique. In accordance with an embodiment, the apparatus 102 is a controller or a computing device configured to execute the transformer. The apparatus 102 combines hardware capabilities with specialized software modules to efficiently process and transform input data, compute and binarize attention scores, and produce outputs, all while supporting the learning and adaptation processes crucial to transformer model performance. The query projector 106 refers to a computational mechanism or algorithm that transforms input data into a format suitable for querying or searching within a given system or database. In accordance with an embodiment, a projector is a learnable deep learning layer that computes data from its input data. In accordance with an embodiment, the projector is implemented by a fully connected layer or a 1 -dimensional convolutional layer. Advantageously, by incorporating the learnable deep learning layer as the projector, is the ability to capture and process complex relationships within the input data. The ability of the projector to capture complex relationships enhances the performance and accuracy of the apparatus in various machine learning tasks. The projector's computation of data from the input data enables the apparatus to generate informative representations of the relationships between terms, which are essential for effective learning and processing within the apparatus. This technical effect makes the apparatus highly suitable for products and systems that utilize or require machine learning transformers. The term "query data" refers to the information or data that is used as input for a query projector, typically consisting of specific parameters, criteria, or conditions that define the desired search or retrieval results. The term "input" refers to any data, information, or signals that are provided to a system, device, or algorithm for processing, analysis, or manipulation.

[0039] The key projector 108 is a component in neural network models, particularly in attention mechanisms, which transforms input data into a specific representation called key data. The key data serves as a reference or anchor point for evaluating how relevant different parts of the input are in relation to the query data.

[0040] The value projector 110 is a component in neural networks, particularly in attention mechanisms, which transforms input data into "value data". The value data represents the information that will be weighted and combined based on the attention scores calculated from the query and key data. The value projector typically applies a linear transformation or passes the input through a neural network layer to generate the value data. The value data is then used in attention mechanisms to produce the final output, with each value being weighted according to how relevant it is to the corresponding query data. The weighted combination of these values helps the model focus on the most essential information for a given task, such as translating a sentence or identifying key features in a sequence.

[0041] The module 112 refers to a self-contained unit or component that performs a specific function within the apparatus 102. The parameter projector 114 is a component in a neural network that is designed to project or transform input data into a set of parameters that are learnable during the training process. This typically involves applying a linear transformation or a neural network layer to the input data to generate these parameters. The parameter projector 114 takes input data with specific dimensions and produces an output that represents certain parameters, such as weights, biases, or thresholds, which the model can adjust during training to improve its performance on a given task. The learnable parameters are crucial for tasks like classification, regression, or attention mechanisms, as they enable the model to adapt to the data and improve accuracy. The binarization module 116 is a component in a system or neural network that converts continuous or multi-valued data into binary values, typically 0 and 1. This process is known as binarization. The binarization module 116 uses a threshold value to determine the binary output: if the input data exceeds the threshold, it is assigned a value of 1 ; if it is below the threshold, it is assigned a value of 0. The binarization is used in scenarios where decisions are based on whether certain conditions are met or not, simplifying the data into a binary format that can be more easily processed or interpreted in subsequent stages of the apparatus 102.

[0042] In operation, the query projector 106 is configured to determine query data based on an input 104. The query projector receives input 104, whichmay be a matrix or tensor containing multiple data points. For example, in natural language processing (NLP), the input 104 may be a sequence of word embeddings, where each embedding is a vector representing a word in a highdimensional space.

[0043] In some implementations, the query projector 106 consists of a linear transformation, usually implemented as a fully connected layer. The fully connected layer is characterized by a matrix of learnable weights and a bias vector. The input 104 is multiplied by the weight matrix of the query projector 106. If the input 104 has a shape of [L, d] (where L is the sequence length and d is the dimension of the input features), and the weight matrix has a shape of [d, d_q] (where d_q is the dimension of the query data), the operation can be represented as: Q=XxWq+bq;

[0044] where:

[0045] X is the input data [L, d],

[0046] Wm is the weight matrix [d, d_q],

[0047] be is the bias vector [d_q],

[0048] Q is the resulting query data [L, d_q] .

[0049] The linear transformation adjusts the input data into a new space (the query space), where specific aspects of the input 104 are highlighted according to the learned parameters in the projector. After the linear transformation, sometimes an activation function is applied to introduce non-linearity. This allows the model to learn more complex patterns. Common activation functions include Rectified Linear Unit (ReLU) or Gaussian Error Linear Unit (GELU). The output of the linear transformation (and optional activation) is the query data. The query data now exists in a space where it can effectively interact with other elements in the attention mechanism, such as keys.

[0050] The key projector 108 is configured to determine key data based on the input 104. The key projector takes input 104. The input 104 may represent various features depending on the application, such as word embeddings in NLP, image patches in computer vision, or any other data that needs to be processed. The key projector 108 is typically implemented as a linear layer consisting of a weight matrix and an optional bias vector. The linear layer transforms the input 104 into key data. The transformation involves multiplying the input 104 by the weight matrix of the key projector 108. If the input has a shape of [L, d] (where L is the sequence length and d is the dimension of the input features), and the weight matrix has a shape of [d, d_k] (where d_k is t

[0051]

[0052] he dimension of the key data), the operation can be represented as: K=X*Wk+bk ;

[0053] where:

[0054] X is the input [L, d],

[0055] Wk is the weight matrix [d, d_k],

[0056] bk is the bias vector [d_k],

[0057] K is the resulting key data [L, d_k].

[0058] The transformation projects the input 104 into a new space (the key space), where it is optimized for use in the attention mechanism. The output of the linear transformation (and optional activation) is the key data denoted by “K”. The key data is now in a form that allows it to interact effectively with query data during the attention process. Each key vector represents the input data in a way that will be compared with query vectors to determine their relevance or importance. Using attention mechanisms, the key data is compared with the query data to calculate attention scores. The scores dictate how much attention each query should pay to different parts of the input, represented by the keys.

[0059] The value projector 110 is configured to determine value data (V) based on the input 104. The value projector 110 is a learnable deep learning layer. The value projector 110 determines value data (V) based on the input 104. The value projector 110 can be implemented using a Full Connected (Linear) layer or a ID convolutional layer. The purpose of computing the self-attention score is to enable the transformer to process and understand the input data effectively. By determining the value data using the value projector 110, the apparatus 102 may enhance the transformer's ability to attend to relevant information and make accurate predictions or classifications. Advantageously, by using the value projector 110 in the apparatus 102, the improved computation of the self-attention score was achieved. This allows the transformer to better capture the relationships and dependencies within the input data, leading to enhanced performance and accuracy in various machine-learning tasks.

[0060] Additionally, the flexibility in implementing the value projector 110 using different layer types provides options for optimizing the complexity and performance trade-off in the self-attention module. The input to the transformer is typically a sequence of vectors representing words or tokens in the case of natural language processing. Each token has its corresponding embedding, which is a numerical representation of its meaning in the context of the task.

[0061] The value projector is a layer, typically a fully connected layer (also called a linear layer) or sometimes a ID convolutional layer. It transforms the input data into the value data (V) by applying a learnable weight matrix and a bias. Mathematically, this transformation can be represented as: V = X W_v + b_v:

[0062] where:

[0063] o “X” is the input data,

[0064] o Wvis the learnable weight matrix specific to the value projector,

[0065] o b_v is the bias term.

[0066] The result of this transformation is the value data, which represents the information content the model will use later in the computation of the self-attention mechanism. In the self-attention mechanism, value data is important because it is combined with the attention score (computed from the query and key data) to generate the final output. The output will then influence how the model understands and processes the input sequence.

[0067] The module 112 is configured to compute a cross-relation score based on the query data and the key data. The query data, key data, value data and cross-relation score are real-valued tensors. The apparatus 102 is configured to compute a self-attention score for use in a transformer. It includes a module that computes a cross-relation score by performing a computation between the query data and the key data. This computation involves using a real- valued multiplicator (MULT) and applying the SoftMax function to the result. The resulting cross-relation score is a real- valued tensor. The module 112 address the computational complexity associated with the large dimension of the processed data in self-attention modules. By using real-valued tensors and performing the necessary computations, the apparatus aims to accurately compute the self-attention score while minimizing the performance gap that may arise from using Boolean structures. Advantageously, the configuration can compute the selfattention score efficiently and accurately. By utilizing real- valued tensors and the appropriate computations, the apparatus can handle the computational intensity of the self-attention module, even with large data dimensions.

[0068] The parameter projector 114 is configured to determine learnable parameters. The parameter projector 114 generates learnable parameters. In accordance with an embodiment, the parameter projector 114 is configured to determine the learnable parameters based on the input 104. The learnable parameters are adaptable, meaning that during the training of the transformer model, the values of the learnable parameters are adjusted to optimize the binarization process. The learnable parameters are essential because they control the threshold used to binarize the attention scores. In simpler terms, they determine whether the real-valued attention score will be converted to a 0 or a 1 based on the comparison between the correlation score and learnable parameter. In operation, the parameter projector 114 takes the input 104 and processes it through a linear transformation or similar operation to produce the thresholds learnable parameters.

[0069] The parameter projector 114 serves to generate learnable parameters from the cross-relation score. The learnable parameters are calculated via a linear transformation followed by an activation function and are optimized during training to act as thresholds or modulation factors in the model. By learning these parameters, the model can better interpret and process the input data, ultimately improving performance in tasks such as attention mechanisms or binary classification. The transformation can be done using different types of layers, such as a fully connected layer or a 1 D convolutional layer, which contains trainable weight parameters. These layers effectively learn the best thresholds based on the specific task and dataset. For example, the parameter projector 114 may output parameters that vary across different dimensions of the input data. For example, the input tensor has dimensions [L, d, N] where:

[0070] o “L” is the length of the input sequence (number of tokens),

[0071] o “d”is the embedding size (dimensionality of each token's vector),

[0072] o “N” is the number of sequences being processed.

[0073] In accordance with an embodiment, the parameter projector 114 is a linear layer of weight parameters of shape [d, 1, N] and when the input data is of dimensions [L, d, N], the resulting learnable parameters is of dimensions [L, 1, N], The parameter projector 114 may produce a threshold tensor of dimension [L,l, N], This means that for each sequence and each token in that sequence, a different threshold value is learned. The threshold values are used by the binarization module to convert the real-valued attention scores into binary values. In accordance with an embodiment, N is a number of input sequences, L is a length of each input sequence, and d is an embedding size. The binarization module 116 is configured to binarize the cross-relation score (A) based on the learnable parameters (T). For example, the cross-relation score has a 3x3 cross-relation score matrix “A”:

[0074] A = [[0.7, 0.2, 0.5],

[0075] [0.3, 0.8, 0.1],

[0076] [0.6, 0.4, 0.9]].

[0077] The parameter projector produces a threshold vector “T” of shape (3, 1 ):

[0078] T = [[0.4],

[0079] [0.5],

[0080] [0.6]].

[0081] The learnable parameters are a threshold (T) utilized by the binarization module 116. The binarization module 116 applies the threshold T to each row of “A”. Elements greater than or equal to the threshold become 1 , while those below become 0. For the first row:

[0082] • Threshold T[0] = 0.4

[0083] • A[0,0] = 0.7 > 0.4, so A_bin[0,0] = 1

[0084] • A[0,l] = 0.2 < 0.4, so A_bin[0,l] = 0

[0085] • A[0,2] = 0.5 > 0.4, so A_bin[0,2] = 1

[0086] Applying this process to all rows, we get:

[0087] A_bin = [[1, 0, 1],

[0088] [0, 1, 0],

[0089] [0, 0, 1]].

[0090] The threshold T (3x1) is broadcast across the second dimension of A (3x3). Each row of A uses its corresponding threshold value from T. The resulting binarized cross-relation score A_bin maintains the overall structure of the original “A” but with binary values. The process reduces the precision but potentially speeds up subsequent computations in the transformer model. The learnable parameters allow the model to adapt the binarization thresholds during training. The binarization preserves some of the relational information from the original cross-relation score. The binary output may lead to more efficient computations in subsequent layers of the transformer. The binarization approach aims to balance the trade-off between computational efficiency and maintaining the essential relational information captured by the attention mechanism in transformer models. Binarized Cross-relation Score (A):

[0091] the binarized A matrix = [[1, 0, 1],

[0092] [0, 1, 0],

[0093] [0, 0, 1]].

[0094] This is a 3x3 binary matrix.

[0095] For example, a 3x2 value matrix “V” (value data):

[0096] V = [[0.5, 0.2],

[0097] [0.3, 0.8],

[0098] [0.1, 0.6]].

[0099] In accordance with an embodiment, the binarization module 116 is configured to utilize the threshold (T) for binarizing the cross-relation score (A) such that the threshold (T) is broadcasted the second dimension of the cross-relation score (A). The output module 118 is configured to compute an output (AV) based on the value data (V) and the binarized cross-relation score (A). The output module 118 computes AV by performing matrix multiplication between “A” and “V”. The resulting “AV” will have dimensions 3x2 (the number of rows from A and the number of columns from V).

[0100] F or the first row of AV :

[0101] AV[0,0] = A[0,0] * V[0,0] + A[0,l] * V[l,0] + A[0,2] * V[2,0]

[0102] = 1 * 0.5 + 0 * 0.3 + 1 * 0.1 = 0.6

[0103] AV[0,l] = A[0,0] * V[0,l] + A[0,l] * V[l,l] + A[0,2] * V[2,l]

[0104] = 1 * 0.2 + 0 * 0.8 + 1 * 0.6 = 0.8

[0105] Continuing this process for all rows, we get:

[0106] AV= [[0.6, 0.8],

[0107] [0.3, 0.8],

[0108] [0.1, 0.6]].

[0109] Each row in “AV” represents the weighted sum of values from “V,” where the weights are determined by the binary attention scores in “A.” The binarization of “A” simplifies this computation, as only adding values from “V” where “A” is 1 and ignoring values where “A” is 0. The process effectively "attends" to certain parts of the input sequence more than others, based on the learned attention patterns.

[0110] The binarization of “A” simplifies the matrix multiplication, potentially speeding up computations. Despite the simplification, the output still captures the essence of the attention mechanism, focusing on relevant parts of the input. The resulting “AV” maintains the same dimensionality as “V” in terms of the number of columns, preserving the feature representation while incorporating the attention information. The output computation is a crucial part of the self-attention mechanism in transformers. By using the binarized attention scores, the model can efficiently aggregate information from different parts of the input sequence, allowing it to capture complex dependencies and relationships in the data.

[0111] FIG. 2 is a flowchart depicting a method for determining self-attention score in the transformer, in accordance with an embodiment of the present disclosure. With reference to FIG. 2, there is shown a flowchart of a method 200 for determining self-attention score in a transformer. The method 200 includes steps 202 to 214.

[0112] At step 202, the method 200 includes determining query data based on an input 104. The apparatus 102 may perform the determining step by executing a query projector 106, for example, the apparatus 102 takes the input 104 and processes it through the query projector 106 to generate query data. The query projector 106 is the neural network layer that maps the input data to a new space. The new space is designed to highlight or extract specific features from the input 104 that are relevant for the subsequent operations, such as attention mechanisms or comparisons with other data points. The query data represents the processed version of the input data that will be used to search or "query" other data (like key data) in a way that identifies similarities, relationships, or matches, depending on the context of the application.

[0113] At step 204, the method 200 further includes determining key data based on the input 104. The apparatus 102 may perform the determining step by executing a key projector 108, for example, the apparatus 102 may process the input 104 through the key projector 108 to generate the key data. The key projector maps the input data into a new representation. The key data is crafted to act as a "reference" or "anchor" in subsequent operations, particularly in attention mechanisms. The key data helps in determining how relevant or similar other data points (such as query data) are to it. In essence, the key projector 108 transforms the input data into a format that makes it easier to compare and match with other data, facilitating tasks like information retrieval, pattern recognition, or decision-making within the apparatus 102.

[0114] At step 206, the method 200 further includes determining value data based on the input 104. The apparatus 102 may perform the determining step by executing a value projector 110. For example, the input 104, which may be a set of features, embeddings, or any other data representation, is fed into the value projector 110. The value projector 110 applies a specific transformation to the input 104. The transformation is designed to map the input 104 into a new space or format known as value data. The result of the transformation is the value data. The value data is important in attention mechanisms because it represents the information that will be aggregated or attended to base on the relevance determined by comparing the query data and the key data. The value data is used to produce the final output of the attention mechanism or other processes in the apparatus 102. The value data carries the essential information that will be weighted and combined according to the relevance scores determined by the interaction between the query data and key data.

[0115] At step 208, the method 200 further includes computing the cross-relation score based on the query data and the key data. The query data, key data, value data and cross-relation score are real-valued tensors. The input 104, which may be a set of features, embeddings, or any other data representation, is fed into the value projector 110. The query data and key data are real-valued tensors. The real-valued tensors are derived from the input 104 through their respective projectors (i.e., the query projector 106 and the key projector 108). The cross-relation score is computed by taking the dot product (or another similarity measure) between the query data and the key data. The dot product measures how similar each query is to each key, resulting in a score that indicates the relevance or importance of each key to the corresponding query. The output of the operation using crossproduct is the cross-relation score (also a real-valued tensor). The dimensions of the cross-relation score tensor represent the relationships between each element in the query set with each element in the key set. The cross-relation score determines how much attention the transformer should pay to each key when processing the corresponding query. Higher scores indicate stronger relevance, which means that the value data associated with that key will have more influence on the final output. In some implementations, the cross-relation score may be further normalized (e.g., using a SoftMax function) to ensure that the scores are on a consistent scale, such as probabilities. The cross-relation score plays an important role in determining how much each piece of value data contributes to the final output, effectively guiding the focus of the apparatus 102 during the processing of information.

[0116] At step 210, the method 200 further includes determining the learnable parameters. The apparatus 102 may perform the determining step by executing a parameter projector 114. For example, the parameter projector 114 receives input data, which may be the original input 104 or another intermediate result from previous steps in the method 200. The input data typically carries information relevant to the task the apparatus 102 is designed to perform. The parameter projector 114 applies a set of weight parameters (which are themselves learnable) to the input data. The operation usually involves a matrix multiplication followed by the addition of a bias term. The result of the transformation is a set of learnable parameters. The learnable parameters are typically tailored to the specific characteristics of the input data and are used in subsequent steps of the method to influence the decisions of the transformer. The learnable parameters serve as flexible components that the model can adjust during training to optimize performance. They can be used in various ways, such as setting thresholds, adjusting weights, or controlling the behaviour of other modules within the model. During training, the model learns the optimal values for learnable parameters by minimizing a loss function. The learning process involves backpropagation, where gradients are calculated with respect to the learnable parameters and used to update them iteratively.

[0117] At step 212, the method 200 further includes binarizing the cross-relation score based on learnable parameters. The crossrelation score is a real-valued tensor that results from earlier computations, such as the interaction between the query data (Q) and key data (K). It represents the strength or relevance of the relationship between different elements in the input 104. The learnable parameters have been determined the previously using the parameter projector 114. The learnable parameters are designed to act as thresholds or other forms of control that will guide the binarization process. Binarization is a process that converts continuous values into binary values, typically 0 or 1. Each element of the cross-relation score is compared to the corresponding threshold value from the learnable parameters. The binarization results in a binary tensor, where each element is either 0 or 1 , depending on whether the corresponding element in the cross-relation score (A) meets or exceeds the threshold. The binary tensor represents a simplified or discretized version of the original data, focusing on whether certain relationships meet the criteria defined by the learnable parameters. Binarizing the cross-relation score can be useful for making discrete decisions or simplifying the data for further processing. For instance, in classification tasks, binarization might help in determining which categories are relevant or in selecting specific features for attention.

[0118] At step 214, the method 200 further includes computing an output 120 based on the value data and the binarized cross-relation score. The output from the value projector 110, i.e., value data (V), typically of shape [L, d, N], the binarized cross-relation score, the output from the binarization module 116, i.e., cross-relation score (A), typically of shape [L, L, N], but now contains only binary values (0 or 1 ).

[0119] The computation process involves Matrix Multiplication: Firstly, the primary operation is a matrix multiplication between “A’ and “V” represented as:

[0120] A [L*, L*, N*] x V [L*, d*, N*] = [L*, d*, N*].

[0121] Further, for each sequence in the batch (N), Each row of “A” represents attention weights for a specific position in the sequence. The binary weights are used to select or mask values from “V.” Furthermore, the binary operation optimization is performed, as “A “is binary, the multiplication may be optimized:

[0122] Where “A” is “1”, the corresponding values from “V” are added to the output. Where “A” is “0”, those values from “V” are ignored (effectively multiplied by 0). Each position in the output is a weighted sum of values, where the weights are binary. This can be thought of as a selective aggregation of information from different parts of the sequence. Traditional attention uses floating-point multiplication between “A” and “V.” With binary A, this becomes a series of additions and omissions, which is computationally more efficient. The resulting output maintains the same shape as the input [L, d, N], preserving the sequence length and embedding dimension. Despite using binary attention, the output still captures relationships between different parts of the sequence. The learnable binarization process aims to optimize which connections are maintained. This computation step is crucial as it is where the efficiency gains from binarization are realized while still attempting to preserve the essential function of the attention mechanism. The challenge lies in balancing the computational efficiency gained from binarization with the potential loss of fine-grained attention information.

[0123] The steps 202 to 214 are only illustrative, and other alternatives can also be provided where one or more steps are added, one or more steps are removed, or one or more steps are provided in a different sequence without departing from the scope of the claims herein.

[0124] There is provided a computer program product comprising program instructions for performing the method 200, when executed by one or more processors in a transformer. In an example, the instructions are implemented on the computer-readable media, which include, but are not limited to, Electrically Erasable Programmable Read-Only Memory (EEPROM), Random Access Memory (RAM), Read-Only Memory (ROM), Hard Disk Drive (HDD), Flash memory, a Secure Digital (SD) card, Solid-State Drive (SSD), a computer-readable storage medium, and / or CPU cache memory.

[0125] FIG. 3 is an exemplary diagram depicting the dimensions of the processed data tensors, in accordance with an embodiment of the present disclosure. FIG. 3 is described in conjunction with elements from FIGs. 1 and 2. With reference to FIG. 3, there is shown an exemplary diagram 300 depicting the processing of the four data tensors, i.e., a first data tensor 302 having dimensions of length “L” of “N”, breadth “d” and width “N”. A second data tensor 304 has dimensions of length “d”, breadth “1” and width “N”. A third data tensor is 306. Further, a fourth data tensor 308 has length “L”, breadth “1”, and width “N”. In operation, the first data tensor 302, i.e., input 104 having Dimensions: [L, d, N], where:

[0126] “L”: Sequence length (e.g., number of words in a sentence)

[0127] “d”: Embedding dimension (size of the vector representing each element)

[0128] “N”: Batch size (number of sequences processed in parallel)

[0129] The second tensor 304, i.e., weight tensor 310, has dimensions [d*, d*, N*], which represent the weight matrices used in the projection of input to query data, key data, and value data. The d* x d* dimensions allow for transformation within the embedding space.

[0130] The “N” dimension suggests batch-specific weights.

[0131] A tensor dimension [L*, L*, N*] represents the attention scores L* x L* shows. Attention weights between all pairs of positions in the sequence. “N” maintains the batch dimension.

[0132] Dimensions: [L*, d*, N*].

[0133] It has the same shape as the input, preserving sequence length and embedding dimension.

[0134] The input 104 is projected using “W” to create “Q”, “K”, and “V”. “Q” and “K” are used to compute the attention scores “A.” “A” is then used with “V” to produce the output. Binarizing “A” to improve computational efficiency. Data flows through the self-attention mechanism and how tensor shapes change or remain consistent, which is crucial for understanding the binarization process proposed in the invention. When applying on the INPUT data, which is of dimensions [L, d, N], the resulting threshold T is of dimensions [L, 1, N], Here, “L,” “d,” and “N” are the components of the data dimensions corresponding to specific physical quantities depending on specific applications. In common applications, “N” is the number of input sequences, “L” is the length of each input sequence, and “d” is the embedding size. The binarization module 116 uses a threshold for binarizing “A” such that the threshold is broadcasted over the second dimension of “A”.

[0135] FIG. 4 is an exemplary diagram depicting an apparatus in another configuration, in accordance with an embodiment of the present disclosure. FIG. 4 is described in conjunction with the FIGs 1 to 3. With reference to FIG. 4, there is shown a block diagram 400 of an apparatus 402. The apparatus 102 includes the query projector 106, the key projector 108, the value projector 110, the module 112, the parameter projector 114, the binarization module 116 and the output module 118. The input 104 is not directly connected to the parameter projector 114.

[0136] In operation, the input 104 in the form of a data tensor is fed into three separate projectors, i.e., the query projector 106, which generates the query tensor (Q), the key projector 108 generates the key tensor, the value projector generates the value tensor ( V) The projectors transform the input into different representations needed for self-attention. The module 112 takes the outputs Q and K from their respective projectors. The module 112 computes the attention scores by performing operations (usually dot product) between the “Q” and the “K”. The parameter projector 114 takes the input and generates the learnable parameter “T”. “T" will be used to guide the binarization process. Further, the binarization module 116 takes two inputs, i.e., the QK scores from the module 112 and the learnable parameter “T” from the parameter projector 114. The binarization module 116 converts the continuous attention scores into binary (0 or 1 ) values using “T” as thresholds. The binarized attention represents the output of the binarization module 116. The binarized attention is the binarized version of the attention scores. Further, the output module 118 takes two inputs namely the binarized attention scores (A) and the value tensor (V) from the parameter projector 114.

[0137] In accordance with an embodiment, the parameter projector is configured to determine the learnable parameters (T) based on the cross-relation score. The configuration where the parameter projector determines the learnable parameters (T) based on the cross-relation score offers a significant technical advantage in improving the accuracy of the binarization process. By basing the threshold (“T”) on the actual computed attention score (“A”), the binarization can be more closely aligned with the specific characteristics of the data being processed. The context-sensitive approach allows the binarization module 116 to retain more critical information from the original real-valued score, thereby reducing the performance gap typically associated with binarization. Ultimately, this leads to more accurate and efficient attention mechanisms in the transformers, particularly in scenarios where computational complexity needs to be minimized without sacrificing too much precision. The parameter projector 114 computes the final output by combining A and V (typically through matrix multiplication). The output 404 is the final result of the self-attention mechanism, incorporating binarized attention. Advantageously, the presence of the parameter projector 114 and the binarization module 116 reduces computational complexity while maintaining performance. The learnable parameters T allow for adaptive binarization, potentially optimizing the information preserved after binarization. The overall flow remains similar to standard self-attention but with the added binarization step. The apparatus 402 makes transformer models more efficient by simplifying the attention mechanism through binarization while using learnable parameters to maintain adaptability and performance.

[0138] Modifications to embodiments of the present disclosure described in the foregoing are possible without departing from the scope of the present disclosure as defined by the accompanying claims. Expressions such as "including", "comprising", "incorporating", "have", "is" used to describe, and claim the present disclosure are intended to be construed in a non-exclusive manner, namely allowing for items, components or elements not explicitly described also to be present. Reference to the singular is also to be construed to relate to the plural. The word "exemplary" is used herein to mean "serving as an example, instance or illustration". Any embodiment described as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments or to exclude the incorporation of features from other embodiments. The word "optionally" is used herein to mean "is provided in some embodiments and not provided in other embodiments". It is appreciated that certain features of the present disclosure, which are, for clarity, described in the context of separate embodiments, may also be provided in combination in a single embodiment. Conversely, various features of the invention, which are, for brevity, described in the context of a single embodiment, may also be provided separately or in any suitable combination or as suitable in any other described embodiment of the disclosure.

Claims

CLAIMS1. An apparatus (102) configured to compute a self-attention score to be used in a transformer, the apparatus comprising:a query projector (106) configured to determine query data (Q) based on an input (104),a key projector (108) configured to determine key data (K) based on the input,a value projector (110) configured to determine value data (V) based on the input,a module (112) configured to compute a cross-relation score (A) based on the query data (Q) and the key data (K), wherein the query data, key data, value data and cross-relation score (Q, K, V, and A) are real- valued tensors, a parameter projector (114) configured to determine learnable parameters (T),a binarization module (116) configured to binarize the cross-relation score (A) based on the learnable parameters (T), andan output module (118) configured to compute an output (AV) (120) based on the value data (V) and the binarized cross-relation score (A).

2. The apparatus (102) according to claim 1, wherein the parameter projector is configured to determine the learnable parameters (T) based on the input.

3. The apparatus (102, 402) according to claim 1 or 2, wherein the parameter projector is configured to determine the learnable parameters (T) based on the cross-relation score.

4. The apparatus (102, 402) according to any preceding claim, wherein the learnable parameters are a threshold (T) utilized by the binarization module.

5. The apparatus (102, 402) according to claim 4, wherein the binarization module is configured to utilize the threshold (T) for binarizing the cross-relation score (A) such that the threshold (T) is broadcasted the second dimension of the crossrelation score (A).

6. The apparatus (102, 402) according to any preceding claim, wherein the parameter projector (114) is a linear layer of weight parameters of shape [d, 1, N] and when the input data is of dimensions [L, d, N], the resulting learnable parameters is of dimensions [L, 1, N],7. The apparatus (102, 402) according to claim 6, wherein N is a number of input sequences, L is a length of each input sequence, and d is an embedding size.

8. The apparatus (102, 402) according to any preceding claim, wherein a projector is a learnable deep learning layer that computes data from its input data.

9. The apparatus (102, 402) according to claim 8, wherein the projector is implemented by a Full-Connected layer or a 1-dimensional convolutional layer.

10. The apparatus (102, 402) according to any preceding claim, wherein the apparatus (102) is a controller, or a computing device configured to execute a transformer.

11. A method (200) for determining self-attention score in a transformer, the method comprising:determining query data (Q) based on an input (104),determining key data (K) based on the input (104),determining value data (V) based on the input (104),computing a cross-relation score (A) based on the query data (Q) and the key data (K), wherein the query data, key data, value data and cross-relation score (Q, K, V, and A) are real-valued tensors and wherein the method (200) further comprises:determining the learnable parameters (T),binarizing the cross-relation score (A) based on learnable parameters (T), andcomputing an output (AV) based on the value data (V) and the binarized cross-relation score (A).

12. The method (200) according to claim 11, wherein the method (200) comprises determining the learnable parameters (T) based on the input (104).

13. The method (200) according to claim 11 or 12, wherein the method (200) comprises determining the learnable parameters (T) based on the cross-relation score.

14. The method (200) according to claim 11 , 12 or 13, wherein the learnable parameters are a threshold (T) utilized by the binarization module (116).

15. A computer program product, comprising program instructions for performing the method according to any of claims 11 to 14.

Images