Computing an attention score using boolean logic

Abstract

In some examples, a self-attention module for processing input data in a neural network comprises a first logic gate arranged to process a first input value and a second input value, whereby to calculate a cross-relation score between the first input value and a second input value, and a second logic gate arranged to process the calculated cross-relation score and a third input value, whereby to generate an attention score.

Description

[0001] COMPUTING AN ATTENTION SCORE USING BOOLEAN LOGIC

[0002] TECHNICAL FIELD

[0003] The present disclosure relates, in general, to reducing the computational complexity of the self-attention module by using Boolean weights. Aspects of the disclosure relate to the method of determining the Boolean weights.

[0004] BACKGROUND

[0005] Transformers are a class of machine learning models that were initially developed for tasks in natural language processing (NLP), such as machine translation. Since their introduction, transformers have become widely used in a variety of machine learning applications, including text classification, language generation, and more recently in generative models such as ChatGPT. The fundamental principle of the transformer model is its ability to learn and represent relationships between elements within sequential input data. This enables transformers to process complex patterns within the data effectively, making them crucial for tasks such as translating languages, generating coherent text, and understanding context. A key feature of the transformer model is its ability to compute "attention scores," which indicate the relative importance of different elements in the input sequence.

[0006] The attention scores are computed through a mechanism known as self-attention. Self-attention allows the model to determine how each element of the input data (for example, a word or token) relates to all other elements in the sequence. To achieve this, the input data is transformed into three key representations: query (Q), key (K), and value (V), which are computed by a learnable deep learning layer referred to as the projector. The projector is typically implemented using a fully connected (or linear) layer, though it may also be realised with a ID convolutional layer. Using these Q, K, and V representations, selfattention calculates the attention scores, which direct how information is processed and propagated through the model. Despite the effectiveness of self-attention in capturing relationships within data, a major challenge associated with transformers is the computational complexity of this mechanism. The self-attention process involves calculating pairwise interactions between all elements of the input sequence, resulting in computational complexity that scales quadratically with the sequence length. In practice, this means that as the size of the input data grows, the time and resources required to compute the attention scores increase significantly. This problem is particularly pronounced when dealing with large-scale tasks, such as processing long sequences of text in translation or generation applications, where the computational demands can become a major bottleneck. Therefore, managing the computational complexity of self-attention is a critical concern in the ongoing development and deployment of transformer models.

[0007] SUMMARY

[0008] An objective of the present disclosure is to reduce the computational complexity of the self-attention module by using Boolean weights.

[0009] The foregoing and other objectives are achieved by the features of the independent claims.

[0010] Further implementation forms are apparent from the dependent claims, the description and the Figures.

[0011] A first aspect of the present disclosure provides a self-attention module for processing input data in a neural network, the selfattention module comprising a first logic gate arranged to process a first input value and a second input value, whereby to calculate a cross-relation score between the first input value and a second input value, and a second logic gate arranged to process the calculated cross-relation score and a third input value, whereby to generate an attention score. Accordingly, the technical problem addressed by this invention is the development of a self-attention module wherein activations and weights are represented as Boolean numbers (i.e., values of TRUE or FALSE, consuming only 1 bit). The solution involves operating these Boolean values with Boolean logic gates. This approach aims to significantly reduce the computational complexity of the self-attention module while achieving performance levels comparable to those of fullprecision models.

[0012] The first input value, the second input value and the third input value may each comprise a Boolean value.

[0013] The self-attention module may further comprise an activation module arranged to convert the input data into a Boolean format, whereby to obtain converted input data.

[0014] The self-attention module may further comprise multiple transformation layers, wherein each of the multiple transformation layers is arranged to generate, based on the converted input data, an output comprising one of: the first input value, the second input value, or the third input value.

[0015] Each of the multiple transformation layers may comprise a Boolean neuron, wherein an output of the Boolean neuron is based on an associated Boolean weight, wherein the associated Boolean weight may be obtained by inverting a current weight of the Boolean neuron in response to reception of a predefined trigger signal.

[0016] The self-attention module may further comprise multiple normalisation layers, wherein each one of the multiple normalisation layers is arranged to normalise the output of a respective transformation layer of the multiple transformation layers.

[0017] The first logic gate may comprise an XOR gate.

[0018] The second logic gate may comprise an XNOR gate.

[0019] The self-attention module may further comprise a scaling module arranged to modify the cross-relation score.

[0020] A second aspect of the present disclosure provides a method for processing input data in a neural network, comprising processing, by a first logic gate, a first input value and a second input value, whereby to calculate a cross-relation score between the first input value and a second input value, and processing, by a second logic gate, the calculated cross-relation score and a third input value, whereby to generate an attention score.

[0021] The method may further comprise converting the input data into a Boolean value, whereby to obtain converted input data, and processing the converted input data using multiple transformation layers, whereby to generate an output comprising one of: the first input value, the second input value, or the third input value.

[0022] The method may further comprise converting the output of a transformation layer of the multiple transformation layers into a Boolean value.

[0023] Each of the multiple transformation layers may comprise a Boolean neuron, wherein an output of the Boolean neuron is based on an associated Boolean weight, and the method may further comprise obtaining the associated Boolean weight by inverting a current weight of the Boolean neuron in response to reception of a predefined trigger signal.

[0024] A third aspect of the present disclosure provides a computer program stored on a non-transitory medium and including code instructions, which, when executed on more or more processors, cause the one or more processors to execute the method described herein.

[0025] These and other aspects of the invention will be apparent from the embodiment(s) described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order that the present invention may be more readily understood, embodiments of the invention will now be described, by way of example, with reference to the accompanying drawings, in which:

[0027] Figure 1 is a schematic representation of a self-attention module according to an example;

[0028] Figure 2 is a schematic representation of a self-attention module according to another example;

[0029] Figure 3 is a schematic representation of a self-attention module according to yet another example; and

[0030] Figure 4 is a flow chart of a method for processing input data in a neural network according to an example.

[0031] DETAILED DESCRIPTION

[0032] Example embodiments are described below in sufficient detail to enable those of ordinary skill in the art to embody and implement the systems and processes herein described. It is important to understand that embodiments can be provided in many alternate forms and should not be construed as limited to the examples set forth herein.

[0033] Accordingly, while embodiments can be modified in various ways and take on various alternative forms, specific embodiments thereof are shown in the drawings and described in detail below as examples. There is no intent to limit to the particular forms disclosed. On the contrary, all modifications, equivalents, and alternatives falling within the scope of the appended claims should be included. Elements of the example embodiments are consistently denoted by the same reference numerals throughout the drawings and detailed description where appropriate.

[0034] The terminology used herein to describe embodiments is not intended to limit the scope. The articles “a,” “an,” and ‘The” are singular in that they have a single referent, however the use of the singular form in the present document should not preclude the presence of more than one referent. In other words, elements referred to in the singular can number one or more, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises,” “comprising,” “includes,” and / or “including,” when used herein, specify the presence of stated features, items, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, items, steps, operations, elements, components, and / or groups thereof.

[0035] Unless otherwise defined, all terms (including technical and scientific terms) used herein are to be interpreted as is customary in the art. It will be further understood that terms in common usage should also be interpreted as is customary in the relevant art and not in an idealized or overly formal sense unless expressly so defined herein.

[0036] The attention score is computed by a specific component known as self-attention, where a learnable deep learning layer, referred to as the projector, computes the Q (query), K (key), and V (value) representations from the input data. The projector is typically implemented using a fully connected (or linear) layer, though it may also be realised through a ID convolutional layer. A fundamental element of any deep learning layer is the neuron. In the current state of the art, a neuron is characterised by weights w1;... , wm, and a bias w0. For an input vector x1:... , xm, the neuron computes an output given as:

[0037] m

[0038] y = W0+ ^ WlX Xl,

[0039]

[0040] i=l

[0041] where the operator "x" represents real- valued multiplication (MULT). Since the structure of deep learning layers is not altered, it is assumed that the conventional architecture of such layers is well-known. The weights of the layers are learned during the training phase, and the current method for weight optimisation is based on the commonly known principles of gradient backpropagation and gradient descent. During the backward pass of training, each component in the module, upon receiving gradients from downstream components, computes gradients with respect to its own weights and sends gradients to upstream components. These gradients are used by gradient descent-based optimisers to update the weights.

[0042] In this self-attention architecture, all data and parameters are real-valued, and dot products are computed using real-valued multiplication (MULT). More specifically, the computation of the attention score involves two primary steps. The first step, referred to as OPERATION-1, calculates a cross-relation score between a single query, which corresponds to a row in tensor Q, and a single key, corresponding to a row in tensor K.

[0043] Mathematically, each 2D slice of A is first given as: Ap= QpKp / / d, where the sub index p denotes the slice index in the 3D tensors of Q, K, and A, and d denotes the number of columns of Qp. This dot product in a computing system comprises the following base operation:

[0044] Ap

[0045]

[0046] After that, A is applied to the softmax function in the row-basis of each slide as follows:

[0047] Ap[i, :] := softmax(Ap[i, : ]).

[0048] It is observed that the fundamental computing operations involved in obtaining A are multiplication (MULT) and summation (SUM), with MULT being recognised as the most computationally intensive operation. The second part of the attention score computation, referred to as OPERATION-2, involves a dot product between A and V In the same manner, a single attention score is computed as:

[0049] L

[0050] = ^>lp[i,c] x Vp[c,j],

[0051]

[0052] C=1

[0053] wherein L is the number of rows of Qp. Given the typically large dimensions of the processed data, the computational complexity of self-attention presents a significant challenge. This complexity is particularly problematic in scenarios where real-time processing or resource efficiency is required.

[0054] The most relevant prior art includes solutions that propose a binary architecture for the self-attention module, in which certain data and / or parameters are represented as binary numbers, consuming only 1 bit, instead of real- valued numbers, which require a multi-bit format, such as 16 bits. The four most related solutions in this area are outlined below.

[0055] A first solution in the prior art is the BIT architecture, proposed in "BiT: Robustly Binarized Multi-distilled Transformer," presented atNeurlPS 2022. This architecture applies an existing binarisation method, in which the projectors of the Q, K, and V matrices are fully connected layers with binarised weights. Learning binarised weights Wblnis based on the use of continuous latent weights Wrealthat are learnt by the standard gradient descent method, i.e.,

[0056] update the continuous latent weights Wrealby a gradient descent-based method;

[0057] extract binarised weights: Wbln= sign(Wreal— a), where a =

[0058]

[0059] W™al.

[0060] The activations such as Q, K, V, and A are all also binarised, resulting in 1-bitby 1-bit A second solution, presented in "Binarized Neural Machine Translation" atNeurlPS 2023, introduces a similar approach to BIT but with two notable differences. First, it applies a layer normalisation before binarising Q, K, and V Second, it modifies the method used to binarise the continuous weights and activations. A third solution is described in "BitNet: Scaling 1 -bit Transformers for Large Language Models" (2023). This approach also utilises a binarised design, though only the weights are binarised, while the activations remain in real-number format. The method for binarising the weights from continuous latent weights follows the same process as in the BIT architecture mentioned above. Finally, a fourth solution, "BiViT: Exploring Binary Vision Transformers" (2023), proposes a binarisation method similar to the previous solution but specifically replaces the softmax function with a sigmoid function in the self-attention mechanism. This solution is primarily applied to vision transformers.

[0061] The existing solutions that utilise binarisation techniques inherit significant limitations inherent to this approach. One major drawback is the substantial performance gap between binarised models and their full-precision counterparts. While binarisation reduces memory usage and computational costs during inference, it leads to a loss of precision that can result in diminished model accuracy and overall performance. Additionally, these binarised architectures do not address the complexity of model training. The training process still relies on full-precision operations, particularly during the gradient descent phase, which means that the computational load and time required for training remain high. As a result, while binarisation offers some benefits during inference, it falls short in providing a comprehensive solution for reducing computational complexity in both training and deployment phases.

[0062] According to an example, there is provided a mechanism to drastically reduce the computational complexity in self-attention modules. More specifically, aspects relate to the use of Boolean projectors and logic gates to optimise the process. For example, Boolean projectors can be utilised to reduce the complexity associated with projecting the Q, K, and V matrices in the selfattention mechanism. The mechanism provided advantageously computes attention scores using logic gates, thereby reducing the complexity of the dot product operations, which are typically the dominant source of computational overhead in selfattention. In particular, Boolean weights are learned directly using Boolean logic backpropagation, which allows the selfattention module to be trained entirely in the Boolean domain. This approach avoids the complexity associated with continuous latent weights and the need for a gradient-descent-based optimiser. Thus, in an example, the training of the Boolean selfattention module becomes significantly more efficient, while still maintaining performance in attention-based tasks.

[0063] Examples in the present disclosure can be provided as methods, systems or machine-readable instructions, such as any combination of software, hardware, firmware or the like. Such machine-readable instructions may be included on a computer readable storage medium (including but not limited to disc storage, CD-ROM, optical storage, etc.) having computer readable program codes therein or thereon.

[0064] The present disclosure is described with reference to flow charts and / or block diagrams of the method, devices and systems according to examples of the present disclosure. Although the flow diagrams described above show a specific order of execution, the order of execution may differ from that which is depicted. Blocks described in relation to one flow chart may be combined with those of another flow chart. In some examples, some blocks of the flow diagrams may not be necessary and / or additional blocks may be added. It shall be understood that each flow and / or block in the flow charts and / or block diagrams, as well as combinations of the flows and / or diagrams in the flow charts and / or block diagrams can be realized by machine readable instructions.

[0065] The machine-readable instructions may, for example, be executed by a machine such as a general-purpose computer, user equipment such as a smart device, e.g., a smart phone, a special purpose computer, an embedded processor or processors of other programmable data processing devices to realize the functions described in the description and diagrams. In particular, a processor or processing apparatus may execute the machine-readable instructions. Thus, modules of apparatus (for example, a module implementing a comparator unit, or a firewall structure and so on) may be implemented by a processor executing machine readable instructions stored in a memory, or a processor operating in accordance with instructions embedded in logic circuitry. The term 'processor' is to be interpreted broadly to include a CPU, processing unit, ASIC, logic unit, or programmable gate set etc. The methods and modules may all be performed by a single processor or divided amongst several processors.

[0066] Such machine-readable instructions may also be stored in a computer readable storage that can guide the computer or other programmable data processing devices to operate in a specific mode. For example, the instructions may be provided on a non-transitory computer readable storage medium encoded with instructions, executable by a processor.

[0067] Figure 1 is a schematic representation of a self-attention module according to an example. The self-attention module 100 comprises a self-attention module for processing input data in a neural network. The neural network may comprise an artificial neural network. As mentioned earlier in the specification, the self-attention module plays a critical role in allowing the neural network to focus on and model dependencies between different elements of the input data, regardless of their distance within the input sequence. This mechanism enables the network to weigh the relevance of each input element when making predictions or generating outputs, making it particularly effective in tasks such as natural language processing, where contextual understanding is essential.

[0068] The self-attention module 100 comprises a first logic gate 101 arranged to process a first input value and a second input value, whereby to calculate a cross-relation score between the first input value and the second input value. Based on an input 110 of the self-attention module, three values may be computed - the query (Q), a key (K) and a value (V). These computations may be performed by applying distinct linear transformations to the input data 110.

[0069] The input data 110 may be converted into a Boolean signal through the use of a first Boolean activator 111, whereby to obtain converted input data. The self-attention module 100 may comprise multiple Boolean activators. Each one of the Boolean activator may perform a logic operation on Boolean values (TRUE or FALSE) to produce an output based on predefined logical conditions. In Figure 1, five distinct Boolean activators are shown, but the invention is not limited thereto.

[0070] The self-attention module 100 may comprise multiple transformation layers, wherein each of the multiple transformation layers is arranged to generate, based on the converted input data, an output comprising one of: the first input value, the second input value, or the third input value. The transformation layers may also be referred to as projectors. As used herein, the term “projector” may refer to a component that implement these linear transformations. They can be realised as fully connected (linear) layers or other types of neural network layers designed to map the input data to the required feature spaces. As will be described in more detail below, the multiple projectors 103-105 (i.e., the first projector 103, the second projector 104 and the third projector 105) may each comprise a Boolean projector. In other words, each of the projectors 103-105 may comprise at least one Boolean neuron, the weights of which is a Boolean number, instead of real numbers.

[0071] In particular, the multiple projectors 103, 104, 105 may be composed of Boolean neurons whose kernel logic is XNOR. In particular, the structure of the projectors may not be limited to the common choice of fully-connected layer and may comprise any other applicable structure, such as, a ID convolutional layer.

[0072] The utilised Boolean neuron can be described by its Boolean weights w;, its kernel logic, and its output y given as:

[0073] y = w0+ XN0R(wi,xi),

[0074]

[0075] i=l

[0076] wherein m is the neuron’s input size, x,- are input activations, and w0is the bias. In the embodiment of Figure 1, the weights w;, bias w0, and input activations xtmay all comprise Boolean numbers. This base setting may extend directly to the case with real- valued bias w0, as well as real- valued input activations xt. The Boolean activation (realised by the multiple Boolean activators 111, 112, 113, 114, 115) may function in a way similar to the existing binary activation. The Boolean activators may convert an input comprising a real value into a Boolean output based on the following base principle:

[0077] Boolean output = TRUE if real input > Threshold, and FALSE otherwise

[0078] where the threshold comprises a pre-specified threshold or a learnable threshold. In this context, the expression “base principle” may refer to the fundamental approach of converting real-valued inputs into Boolean outputs, upon which any method of Boolean data conversion may be based, allowing for various implementation-specific adaptations and optimisations. The Boolean activator may thus output TRUE when the real input meets or exceeds the threshold, and FALSE otherwise. This conversion facilitates the use of Boolean values (TRUE or FALSE) in subsequent processing steps. It is noted that the first, second, and third input values may each comprise Boolean values, adhering to this principle of conversion.

[0079] While the figures depict the presence of five Boolean activators 111-115, it is to be understood that certain Boolean activations may be omitted under specific conditions. Specifically, some signals may be retained as real-valued, allowing for more flexible processing. In this case, Boolean activations may be selectively applied, such that the first logic gate processes a first input value and a second input value, calculating a cross-relation score, using a combination of real-valued and Boolean data (referred to as "REAL-BOOL") rather than purely real- valued data ("REAL-REAL"). Similarly, the second logic gate, which processes the calculated cross-relation score and a third input value to generate an attention score, may also utilise REAL-BOOL data where appropriate. This approach can help in reducing computational complexity by avoiding purely REAL-REAL operations while enhancing the performance as compared to the purely BOOL-BOOL processing. For instance, Boolean activator 111 may be optional, depending on the specific data type and the operational requirements.

[0080] The first input value may comprise the query (Q). The query (Q) may be derived by applying a first projector 103 to the converted input data (i.e., the input data 110 converted into a Boolean signal using the first Boolean activator 111), mapping it into a query space. The query (Q) may represent the aspect of the input data 110 used to assess its relevance relative to other parts of the input. The output of the first projector 103 may be converted into a Boolean signal through the use of a second Boolean activator 112.

[0081] The second input value may comprise the key (K). Similarly to the query (Q), the key (K) may be computed using a second projector 104, serving as a reference against which the query (Q) is compared. The key may represent the criteria or features by which the relevance of the input data 110 is evaluated. The second projector 104 may be applied to the converted input data. The output of the second projector 104 may be converted into a Boolean signal through the use of a third Boolean activator 113.

[0082] The third input value may comprise the value (V). The value (V) may be derived from the converted input data using a third projector 105 and may represent the actual information or content that is utilised in generating the output of the self-attention mechanism. In other words, the value (V) may comprise the value that gets weighted and aggregated based on the attention scores computed from the queries (Q) and keys (K). The output of the third projector 105 may be converted into a Boolean signal through the use of a fourth Boolean activator 114.

[0083] As mentioned earlier, the first logic gate 101 is arranged to process the first input value and the second input value, whereby to calculate the cross-relation score between the first input value and a second input value. In other words, the first logic gate 101 may be arranged to process the query and the key and calculate the cross-relation score between those. The cross-relation score may be used to quantify the relevance or similarity between two different input values - in this case, between the query (Q) and the key (K). This score may be used to determine how much focus should be placed on one input in relation to another, thereby guiding the aggregation of information in the self-attention module 100 to produce contextually relevant outputs. Importantly, the calculated cross-relation score may be converted into a Boolean signal through the use of a fifth Boolean activator 115. The first logic gate 101 may comprise an exclusive OR (XOR) gate.

[0084] The cross-relation score (i.e., the score between the first input value Q and the second input value K) may be calculated using an XOR gate (i.e., the first logic gate) as follows:

[0085]

[0086] Advantageously, this significantly reduces the computational complexity, since the complex mathematical operations performed on real numbers (for example, the multiplicator MULT) are replaced using much less computationally expensive logic gates.

[0087] The self-attention module 100 also comprises a second logic gate 102 arranged to process the calculated cross-relation score and a third input value, whereby to generate an attention score. As mentioned earlier, the third input value may comprise the value (V). The attention score may be generated based on the calculated cross-relation score between the query (Q) and the key (K), and based on the value (V). The attention score may reflect the importance of the value (V) in the context of the querykey relationship, and may be used to weight and aggregate the values, thereby determining the contribution of each value to the final output of the self-attention module 100. The second logic gate 102 may comprise an exclusive NOR (XNOR) gate.

[0088] The attention score may be calculated on the same basis as the cross-relation score, using an XNOR gate as follows:

[0089] = ^XN0R(X;,Lj.

[0090]

[0091] The skilled person would appreciate that any existing method for backpropagating through the Boolean activation can be used. Here, the term "backpropagating" may refer to the process of propagating error gradients from the output layer of a neural network back through the network’s layers to update the weights based on the error. This may involve calculating how changes in the activation function impact the overall error, and then using these gradients to adjust the weights in the direction that minimises the error, thereby refining the model’s performance. However, the focus of the invention does not he therein, and so a detailed description will be omitted.

[0092] The backpropagation through the logic gates and through the Boolean components may follow the Boolean logic backpropagation principle. In other words, this principle may involve adjusting the weights and parameters by calculating the gradients in the Boolean domain, rather than the continuous domain. This process may employ Boolean logic operations to determine how changes in the Boolean activators 111-115 affect the overall error, and subsequently update the weights accordingly to optimise the model’s performance while operating within the constraints of Boolean logic.

[0093] Figure 2 is a schematic representation of a self-attention module according to another example. The self-attention module 200 of Figure 2 largely corresponds to the self-attention module 100 described above in relation to Figure 1. In both figures, the same elements are denoted using the same reference numerals and function likewise. In contrast to the self-attention module 100 of Figure 1, the self-attention module 200 may comprise a scaling module 210 arranged to modify the cross-relation score. The scaling module 210 may be arranged to adjust or scale the cross-relation score generated by the first logic gate 101 before it is processed by the second logic gate 102. The scaling module 210 may be learnable, meaning that its parameters can be optimised during training to improve the performance of the self-attention module 200. By applying a learned scaling factor to the cross-relation score, the scaling module can be used to fine-tune the sensitivity of the attention mechanism, enabling more precise control over how different parts of the input data 110 contribute to the final self-attention score.

[0094] Figure 3 is a schematic representation of a self-attention module according to yet another example. Again, the self-attention module 300 of Figure 3 largely corresponds to the self-attention modules 100 and 200, described above in relation to Figures 1 and 2. In all figures, the same elements are denoted using the same reference numerals and function likewise. The selfattention module 300 builds on the self-attention module of Figure 2, also comprising the scaling module 210. However, additionally, after each of the first projector 103, the second projector 104, and the third projector 105, a normalisation layer 311, 312, 313 may be present. Each of these normalisation layers 311, 312, 313 may be arranged to normalise the output of its respective projector, ensuring that the outputs are scaled and centred appropriately before further processing.

[0095] In other words, each one of the normalisation layers 311, 312, 313 adjust the activations produced by the respective projector by applying a normalisation process that typically involves scaling and shifting. This process may help in stabilising and accelerating training by reducing internal co variate shift, improving convergence, and ensuring that the outputs of the projectors have a consistent distribution.

[0096] Figure 4 is a flow chart of a method for processing input data in a neural network according to an example. The method comprises, in step 401, processing, by a first logic gate, a first input value and a second input value, whereby to generate a cross-relation score between the first input value and a second input value. The first logic gate used in the method may comprise the first logic gate 101, described above in relation to Figures 1-3. In step 402, the method comprises processing, by a second logic gate, the calculated cross-relation score and a third input value, whereby to generate an attention score. The second logic gate used in the method may comprise the second logic gate 102 described in this specification.

[0097] As discussed earlier in relation to the apparatus, the method may comprise converting the input data into a Boolean value, whereby to obtain converted input data. This conversion may be performed using, for example, the Boolean activator 111. The method may further comprise processing the converted input data using multiple transformation layers, whereby to generate an output comprising one of the first input value, the second input value, or the third input value. The characteristics of these values have been described earlier in the description. The method may also comprise converting the output of each one of the multiple transformation layers (i.e., projectors) into a Boolean value. Each of the multiple transformation layers may comprise at least one Boolean neuron.

[0098] During the training phase, the output of the Boolean neuron may be based on an associated Boolean weight, which may be obtained by inverting the current weight of the Boolean neuron in response to the reception of a predefined trigger signal. This trigger signal may be generated when specific conditions are met during training. By inverting the Boolean weight, the influence of the neuron on subsequent computations can be modified, allowing the model to adapt and learn relevant patterns in the input data. This process can enable efficient training while maintaining the computational simplicity of Boolean operations within the self-attention module.

[0099] According to an example, machine-readable instructions can be loaded onto a computer or other programmable data processing devices, so that the computer or other programmable data processing devices perform a series of operations to produce computer-implemented processing, thus the instructions executed on the computer or other programmable devices provide an operation for realizing functions specified by flow(s) in the flow charts and / or block(s) in the block diagrams.

[0100] Further, the teachings herein may be implemented in the form of a computer or software product, such as a non-transitory machine-readable storage medium, the computer software or product being stored in a storage medium and comprising a plurality of instructions, e.g., machine readable instructions, for making a computer device implement the methods recited in the examples of the present disclosure.

[0101] In some examples, some methods can be performed in a cloud-computing or network-based environment. Cloud-computing environments may provide various services and applications via the Internet. These cloud-based services (e.g., software as a service, platform as a service, infrastructure as a service, etc.) may be accessible through a web browser or other remote interface of the user equipment for example. Various functions described herein may be provided through a remote desktop environment or any other cloud-based computing environment.

[0102] While various embodiments have been described and / or illustrated herein in the context of fully functional computing systems, one or more of these exemplary embodiments may be distributed as a program product in a variety of forms, regardless of the particular type of computer-readable-storage media used to actually carry out the distribution. The embodiments disclosed herein may also be implemented using software modules that perform certain tasks. These software modules may include script, batch, or other executable files that may be stored on a computer-readable storage medium or in a computing system. In some embodiments, these software modules may configure a computing system to perform one or more of the exemplary embodiments disclosed herein. In addition, one or more of the modules described herein may transform data, physical devices, and / or representations of physical devices from one form to another.

[0103] The preceding description has been provided to enable others skilled in the art to best utilize various aspects of the exemplary embodiments disclosed herein. This exemplary description is not intended to be exhaustive or to be limited to any precise form disclosed. Many modifications and variations are possible without departing from the spirit and scope of the instant disclosure. The embodiments disclosed herein should be considered in all respects illustrative and not restrictive. Reference should be made to the appended claims and their equivalents in determining the scope of the instant disclosure.

Claims

CLAIMS1. A self-attention module (100) for processing input data (110) in a neural network, the self-attention module (100) comprising:a first logic gate (101) arranged to process a first input value and a second input value, whereby to calculate a crossrelation score between the first input value and a second input value; anda second logic gate (102) arranged to process the calculated cross-relation score and a third input value, whereby to generate an attention score.

2. The self-attention module of claim 1, wherein the first input value, the second input value and the third input value each comprises a Boolean value.

3. The self-attention module of claim 1 or 2, further comprising an activation module (111) arranged to convert the input data (110) into a Boolean format, whereby to obtain converted input data.

4. The self-attention module of claim 3, further comprising multiple transformation layers (103, 104, 105), wherein each of the multiple transformation layers (103, 104, 105) is arranged to generate, based on the converted input data, an output comprising one of: the first input value, the second input value, or the third input value.

5. The self-attention module of claim 4, wherein each of the multiple transformation layers (103, 104, 105) comprises a Boolean neuron, wherein an output of the Boolean neuron is based on an associated Boolean weight,wherein the associated Boolean weight is obtained by inverting a current weight of the Boolean neuron in response to reception of a predefined trigger signal.

6. The self-attention module of claim 4 or 5, further comprising multiple normalisation layers (311, 312, 313), wherein each one of the multiple normalisation layers (311, 312, 313) is arranged to normalise the output of a respective transformation layer of the multiple transformation layers (103, 104, 105).

7. The self-attention module of any one of claims 1 to 6, wherein the first logic gate (101) comprises an XOR gate.

8. The self-attention module of any one of claims 1 to 7, wherein the second logic gate (102) comprises an XNOR gate.

9. The self-attention module of any one of claims 1 to 8, further comprising a scaling module (210) arranged to modify the cross-relation score.

10. A method for processing input data in a neural network, comprising:processing, by a first logic gate, a first input value and a second input value, whereby to calculate a cross-relation score between the first input value and a second input value (401); andprocessing, by a second logic gate, the calculated cross-relation score and a third input value, whereby to generate an attention score (402).

11. The method of claim 10, further comprising:converting the input data into a Boolean value, whereby to obtain converted input data; andprocessing the converted input data using multiple transformation layers, whereby to generate an output comprising one of: the first input value, the second input value, or the third input value.

12. The method of claim 11, further comprising:converting the output of a transformation layer of the multiple transformation layers into a Boolean value.

13. The method of claim 11 or 12, wherein each ofthe multiple transformation layers comprises a Boolean neuron, wherein an output of the Boolean neuron is based on an associated Boolean weight,the method further comprising:obtaining the associated Boolean weight by inverting a current weight of the Boolean neuron in response to reception of a predefined trigger signal.

14. A computer program stored on a non-transitory medium and including code instructions, which, when executed on more or more processors, cause the one or more processors to execute the method of any of claims 10 to 13.

Images