Devices and methods for providing a transformer neural network model

Abstract

A data processing apparatus comprising a processing circuitry configured to operate a transformer neural network, wherein the transformer neural network comprises a plurality of processing layers for processing a plurality of input elements, including an attention layer. One or more of the plurality of processing layers upstream of the attention layer are configured to determine for each of the plurality of input elements a query vector q i , a key vector k i , and a value vector v i , wherein the plurality of query vectors q i define a query matrix Q, the plurality of key vectors k i define a key matrix K, and the plurality of value vectors v i define a value matrix V. The attention layer is configured to determine as an intermediate result the matrix product A = K' T V and output as a final result the matrix product Q'A.

Description

[0001]Devices and methods for providing a transformer neural network model TECHNICAL FIELD The present disclosure relates to data processing. More specifically, the present disclosure relates to devices and methods for providing and operating a transformer neural network model with an attention layer. BACKGROUND Neural networks, which are being implemented in more and more electronic devices for a variety of different purposes, are usually demanding with respect to computational resources. Therefore, it is a challenge to implement large neural networks on electronic devices with limited hardware capabilities in terms of processing power, memory and energy resources, such as smartphones, intelligent cameras or other types of IoT devices. Even for cloud computing, where the processing capabilities, the memory and the power resources of cloud servers are usually not an issue, reducing the computational complexity of neural networks may provide advantages. Recently, transformer neural network models have emerged as a very popular type of neural networks for analysing data that is in or can be processed into sequential form. It has been applied in various fields, such as natural language processing, computer vision, neural machine translation, document summarization, protein structure prediction and the like. Generally, a key building block of a transformer neural network model is an attention layer or self-attention layer. An attention or self-attention layer may, for instance, identify complex dependencies between the different elements of an input sequence and thereby efficiently capture short and long-term correlations between these tokens. Despite its success, the conventional attention layer of a transformer model generally has a quadratic complexity both in memory and time with respect to the length of the input sequence which makes it very challenging, if not impossible to train transformer models with very long sequences. There have been some recent attempts to enable an efficient training of transformer models with long input sequences. This is because, long input sequences are important for many applications, such as document summarization, document retrieval, and DNA sequence analysis to name a few. Although these attempts allow to reduce the computational and memory complexity of the attention layer, this is achieved by at least a slight degradation of the attention layer performance. Thus, there is still a need for devices and methods allowing to reduce the computational complexity of the attention layer of a transformer neural network model without significantly reducing its performance. SUMMARY It is an object of the present disclosure to provide improved devices and methods for operating transformer neural network models with an attention layer requiring less computational resources without a significant degradation of its performance. The foregoing and other objects are achieved by the subject matter of the independent claims. Further implementation forms are apparent from the dependent claims, the description and the figures. Generally, a transformer neural network model with an improved attention layer is provided that scales linearly with the length of the input sequence, i.e. with the number of elements of the input sequence. Thus, the transformer neural network model with the improved attention layer may be trained with long sequences preserving the memory and latency with a linear dependence on the input sequence length, instead of the quadratic dependency of a conventional attention layer, which is generally implemented using the SoftMax function. More specifically, according to a first aspect a data processing apparatus is provided, comprising a processing circuitry configured to operate a transformer neural network model, wherein the transformer neural network model comprises a plurality of processing layers for processing a plurality, i.e. a sequence of input elements, wherein the plurality of processing layers include an attention layer for identifying short and long-term correlations between the plurality of input elements. One or more of the plurality of processing layers upstream of the attention layer are configured to determine for each of the plurality of input elements of the transformer neural network model a query vector ^^, a key vector ^^, and a value vector ^^, wherein the plurality of query vectors ^^define a query matrix ^, the plurality of key vectors ^^define a key matrix ^, and the plurality of value vectors ^^define a value matrix ^. For instance, the plurality of query vectors ^^define the rows or columns of the query matrix ^, the plurality of key vectors ^^define the rows or columns of the key matrix ^, and the plurality of value vectors ^^define the rows of the value matrix ^. The attention layer is configured to determine as an intermediate result the matrix product ^ = ^′^^ and output as a final result the matrix product ^′^, wherein ^^= ^^^, and ^^= ^^^, and wherein ^ denotes a first parameter and ^ denotes a second parameter. As will be appreciated, for instance, the expression ^^^denotes a matrix of the same size as the matrix ^ with the function ^^applied to a respective element of the matrix ^. Thus, a data processing apparatus is provided allowing to reduce the computational complexity of the attention layer of the transformer neural network model without significantly reducing its performance. In a further possible implementation form the attention layer is configured to output the matrix product ^′^ as the final result based on the following equation: ^^^^^^^ ∑^^^^^^^^, wherein ^^denotes the transpose of the key matrix ^. In a further possible implementation form, the attention layer is configured to determine the first parameter ^ based on the following equation: ^ wherein: ^^denotes ^^^^(^)with ^^^(^)the variance of the matrix ^; ^^denotes ^^^^(^)with ^^^(^)the variance of the matrix ^; and ^, ^ denote constant values. In a further possible implementation form, the attention layer is configured to determine the second parameter ^ based on the following equation: ^ In a further possible implementation form, the attention layer is further configured to determine for a plurality of respective subblocks of the query matrix ^, the key matrix ^, and the value matrix ^ a plurality of intermediate matrices based on the following equation: wherein: ^^^^^^^(_) denotes the softmax function, ^^^[^] denotes a subblock of size ^^^ of the matrix ^, and ^ denotes a dimension of the query matrix ^ and the key matrix ^. In a further possible implementation form, the attention layer is further configured to output as the final result a combination, in particular an average of the matrix product ^′^ of size ^^^ and a further matrix of size ^^^ based on the plurality of intermediate matrices of size ^^^ determined using the softmax function. In a further possible implementation form, the attention layer is further configured to determine the further matrix of size ^^^ based on the plurality of intermediate matrices of size ^^^ by stacking or concatenating the plurality of intermediate matrices of size ^^^. According to a second aspect a data processing method is provided, wherein the data processing method comprises the step of: operating a transformer neural network model, wherein the transformer neural network model comprises a plurality of processing layers for processing a plurality, i.e. a sequence of input elements, including an attention layer for identifying short and long-term correlations between the plurality of input elements. The method further comprises the following step performed by one or more of the plurality of processing layers upstream of the attention layer: determining for each of the plurality of input elements a query vector ^^, a key vector ^^, and a value vector ^^, wherein the plurality of query vectors ^^define a query matrix ^, the plurality of key vectors ^^define a key matrix ^, and the plurality of value vectors ^^define a value matrix ^. Moreover, the method comprises the following step performed by the attention layer: determining as an intermediate result the matrix product ^ = ^′^^ and outputting as a final result the matrix product ^′^, wherein ^^= ^^^, and ^^= ^^^, and wherein ^ denotes a first parameter and ^ denotes a second parameter. Thus, a data processing method is provided allowing to reduce the computational complexity of the attention layer of the transformer neural network model without significantly reducing its performance. In a further possible implementation form, the step of outputting the final result comprises outputting the matrix product ^′^ as the final result based on the following equation: ^^^^^^^ ∑^^^^^^^^, wherein ^^denotes the transpose of the key matrix ^. In a further possible implementation form, the method further comprises determining the first parameter ^ based on the following equation: ^ wherein: ^^denotes ^^^^(^); ^^denotes ^^^^(^); and ^, ^ denote constant values. In a further possible implementation form, the method further comprises the step of determining the second parameter ^ based on the following equation: ^ wherein: ^^denotes ^^^^(^); ^^denotes ^^^^(^); and ^, ^ denote constant values. In a further possible implementation form, the method further comprises the step of determining by the attention layer for a plurality of respective subblocks of the query matrix ^, the key matrix ^, and the value matrix ^ a plurality of intermediate matrices based on the following equation: ^^ wherein: ^^^^^^^(_)denotes the softmax function, ^^^[^] denotes a subblock of size ^^^ of the matrix ^, and ^ denotes a dimension of the query matrix ^ and the key matrix ^. In a further possible implementation form, the step of outputting the final result comprises outputting as the final result a combination, in particular an average of the matrix product ^′^ of size ^^^ and a further matrix of size ^^^ based on the plurality of intermediate matrices of size ^^^. In a further possible implementation form, the method further comprises the step of determining the further matrix of size ^^^ based on the plurality of intermediate matrices of size ^^^ by stacking or concatenating the plurality of intermediate matrices of size ^^^. The data processing method according to the second aspect can be performed by the data processing apparatus according to the first aspect. Thus, further features of the data processing method according to the second aspect result directly from the functionality of the data processing apparatus according to the first aspect and its different implementation forms described above and below. According to a third aspect a computer program or a computer program product is provided, comprising a computer-readable storage medium carrying program code which causes a computer or a processor to perform the data processing method according to the second aspect when the program code is executed by the computer or the processor. The different aspects of the present disclosure can be implemented in software and / or hardware. Details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS In the following embodiments of the present disclosure are described in more detail with reference to the attached figures and drawings, in which: Fig.1 is a schematic diagram illustrating a data processing apparatus according to an embodiment for operating a transformer neural network model with an improved attention layer; Fig.2 is a schematic diagram illustrating an exemplary transformer neural network model implemented by the data processing apparatus according to an embodiment including an improved attention layer; Fig.3a is a diagram illustrating the operation of a conventional attention layer; Fig.3b is a diagram illustrating the operation of an improved attention layer of a transformer neural network model implemented by a data processing apparatus according to an embodiment; Fig.3c is a diagram illustrating the variance of an improved attention layer implemented by a data processing apparatus according to an embodiment; Fig.3d is a histogram for the improved attention layer implemented by a data processing apparatus according to an embodiment with and without moment matching compared with the histogram of a conventional attention layer; Fig.4 is a diagram illustrating further details of the operation of an improved attention layer of a transformer neural network model implemented by a data processing apparatus according to an embodiment; Fig.5 is a flow diagram illustrating a computer-implemented data processing method according to an embodiment for operating a transformer neural network model with an improved attention layer; Fig.6a illustrates the memory consumption of the improved attention layer implemented by a data processing apparatus according to an embodiment in comparison with the conventional self-attention for the Roberta model; Fig.6b illustrates the memory and latency of the improved attention layer implemented by a data processing apparatus according to an embodiment in comparison with the conventional self-attention for the Bert model. In the following identical reference signs refer to identical or at least functionally equivalent features. DETAILED DESCRIPTION OF THE EMBODIMENTS In the following description, reference is made to the accompanying figures, which form part of the disclosure, and which show, by way of illustration, specific aspects of embodiments of the present disclosure or specific aspects in which embodiments of the present disclosure may be used. It is understood that embodiments of the disclosure may be used in other aspects and comprise structural or logical changes not depicted in the figures. The following detailed description, therefore, is not to be taken in a limiting sense, and the scope of the present disclosure is defined by the appended claims. For instance, it is to be understood that a disclosure in connection with a described method may also hold true for a corresponding device or system configured to perform the method and vice versa. For example, if one or a plurality of specific method steps are described, a corresponding device may include one or a plurality of units, e.g. functional units, to perform the described one or plurality of method steps (e.g. one unit performing the one or plurality of steps, or a plurality of units each performing one or more of the plurality of steps), even if such one or more units are not explicitly described or illustrated in the figures. On the other hand, for example, if a specific apparatus is described based on one or a plurality of units, e.g. functional units, a corresponding method may include one step to perform the functionality of the one or plurality of units (e.g. one step performing the functionality of the one or plurality of units, or a plurality of steps each performing the functionality of one or more of the plurality of units), even if such one or plurality of steps are not explicitly described or illustrated in the figures. Further, it is understood that the features of the various exemplary embodiments and / or aspects described herein may be combined with each other, unless specifically noted otherwise. Figure 1 is schematic diagram illustrating a data processing apparatus 100 according to an embodiment. In the embodiment shown in figure 1 the data processing apparatus 100 is, by way of example, a server 100, for instance, a cloud server 100. As will be appreciated, however, in other embodiments the data processing apparatus 100 may be implemented, for instance, as a server farm, a desktop computer, a laptop computer, a tablet computer, a smartphone, an IoT device or another device having the computational resources for implementing a neural network. As illustrated in figure 1, the server 100 may comprise a processing circuitry 110, such as one or more processors or cores 110 for processing data and a memory 130 for storing and retrieving data. Furthermore, the server 100 may comprise a communication interface 150, for instance, for exchanging data with other devices, such as a smartphone 160, via a wired and / or wireless communication channel 170 provided, for instance, by the Internet. The processing circuitry 110 of the server 100 may be implemented in hardware and / or software. The hardware may comprise digital circuitry, or both analog and digital circuitry. Digital circuitry may comprise components such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), or general-purpose processors. The memory 130 may store executable program code which, when executed by the processing circuitry 110, causes the server 100 to perform the functions and methods described herein. As will be described in more detail in the following under further reference to figure 2, the processing circuitry 110 of the data processing apparatus 100, e.g. server 100 is configured to implement, i.e. operate a transformer neural network model 111, wherein the transformer neural network model 111 comprises a plurality of processing layers 113 for processing a plurality, i.e. a sequence of input elements 201. By way of example, the plurality of input elements 201 (also referred to as tokens) may comprise a plurality of words 201 making up a sentence or a query. One or more of the plurality of processing layers 113 of the transformer neural network model 111 may comprise a plurality of neural network weights, which are usually determined or adapted during a training phase of the transformer neural network model 111. The plurality of neural network weights of a respective processing layer 113 may be stored in the memory 130 of the data processing apparatus 100. The plurality of processing layers 113 may comprise different types of neural network layers, such as fully connected, convolution, deconvolution, recurrent layers and the like. As will be appreciated, the transformer neural network model 111 may be a part or component of a larger neural network implemented by the data processing apparatus 100. As illustrated in figure 2 and as a will be described in the following in more detail under further reference to figures 3a and 3b, one of the plurality of processing layers 113 of the transformer neural network model 111 is implemented as an attention layer 113a for identifying short and long-term correlations between the plurality of input elements 201. One or more of the plurality of processing layers 113 upstream of the attention layer 113a are configured to determine for each of the plurality of input elements 201 a query vector ^^, a key vector ^^, and a value vector ^^, wherein the plurality of query vectors ^^define a query matrix ^, the plurality of key vectors ^^define a key matrix ^, and the plurality of value vectors ^^define a value matrix ^. For instance, the plurality of query vectors ^^may define the rows or columns of the query matrix ^, the plurality of key vectors ^^may define the rows or columns of the key matrix ^, and the plurality of value vectors ^^may define the rows of the value matrix ^. Figure 3a is a diagram illustrating the operation of a conventional attention layer based on the query matrix ^, the key matrix ^, and the value matrix ^ as the input. A main component of the conventional attention layer is the so-called scaled dot-product attention function which has the following form: Wherein ^^^^^^^(_)denotes the softmax function and ^ denotes the so-called embedding dimension of the query matrix ^ and the key matrix ^. As will be appreciated, computing the scaled dot-product attention function requires to form a ^ × ^ matrix, where N denotes the sequence length, i.e. the number of elements of the input sequence 201. Consequently, the memory and time complexity of the conventional attention layer grow quadratically with the length of the input sequence 201. Figure 3b is a diagram illustrating the operation of an improved attention layer 113a of the transformer neural network model 111 implemented by the data processing apparatus 100 according to an embodiment. As can be taken from figure 3b, the design of the attention layer 113a is based on the idea of decomposing ^^^^^^^(^^^)^ into ^′^′^^ and performing multiplication in the opposite direction, i.e. from right to left. Since the relevant dimensions of the query matrix ^ and the key matrix ^ are defined by the input sequence length ^ and the embedding dimension with usually ^ ≪ ^ by such a linearization of the attention layer 113a a memory consumption is obtained which scales linearly with the length ^ of the input sequence 201. More specifically, as illustrated in figure 3b, the attention layer 113a of the transformer neural network model 110 implemented by the processing circuitry 101 of the data processing apparatus 110 is configured to determine as an intermediate result the matrix product ^ = ^′^^ and output as a final result the matrix product ^′^, wherein ^^= ^^^, and ^^= ^^^, and wherein ^ denotes a first parameter and ^ denotes a second parameter. As will be appreciated, for instance, the expression ^^^denotes a matrix of the same size as the matrix ^ with the function ^^applied to a respective element of the matrix ^. In an embodiment, the attention layer 113a is configured to output the matrix product ^′^ as the final result based on the following equation: ^^^^^^^ ∑^^^^^^^^, wherein ^^denotes the transpose of the key matrix ^. Thus, as will be appreciated the probabilities, i.e. elements of the attention matrix ^′^′^used by the attention layer 113a (herein also referred to as log-normal attention (LLN) layer 113a) are given by: ^ In an embodiment, the data processing apparatus 110 is configured to determine the first parameter ^ and / or the second parameter ^ such that the statistics of the LLN attention layer 113a of the data processing apparatus 100 are close to statistics of the conventional softmax attention layer. As used herein, statistics of the LLN attention layer 113a and the conventional softmax attention layer may be a respective mean and / or variance of the element of the output tensor of the LLN attention layer 113a and the conventional softmax attention layer. Thus, for matching the statistics between the LLN attention layer 113a and the conventional attention layer the data processing apparatus 100 may determine the variance of the output of the LLN attention layer 113a and the conventional softmax attention layers as a function of the variance of the input and may match both variances using the first and second parameters ^, ^. Assuming that the input to the LLN attention layer 113a, i.e. the query matrix ^, the key matrix ^, and the value matrix ^ have a Gaussian distribution of their respective elements, the output of the probabilities matrix ^^^^^^^ ^^^^follows a log-normal distribution. As will be appreciated, for a log-normal distribution ^^^^may be represented as the probabilities matrix for the conventional softmax attention layer may be represented as ^^^^^^^^^. Analyzing the parameters as a function of the input matrices one obtains the following relations: ^ where σ2qand σ2 k are the variance of the query matrix ^ and the key matrix ^, respectively. Equating the two variances and solving for ^, ^ one obtains: ^ ^ Figure 3c illustrates the variance of the LLN attention layer 113a and the variance of the conventional softmax attention layer. Figure 3d illustrates a histogram of the LLN attention layer 113a with and without moment matching, that is, finding α and β such that σ2SMand σ2LLNare identical, compared with the histogram of the conventional softmax attention. In an embodiment, in order to determine the constants ^ and ^ the data processing apparatus 100 may be configured to determine values of ^ ^ ^^^for different values of ^^^, ^^ ^ and, because the expression above for ^ ^ ^^^is linear in ^^^, ^^ ^ , by fitting the line may determine the values for the constants ^ and ^. In a further embodiment illustrated in figure 4, the transformer neural network model 110 implemented by the processing circuitry 101 of the data processing apparatus 100 may implement an attention layer 113a comprising a combination of the LLN attention layer 401 described above and a conventional softmax attention layer 405. More specifically, as illustrated in figure 4, in an embodiment, the attention layer 113 is further configured to determine for a plurality of respective subblocks of size ^^^ of the query matrix ^, the key matrix ^, and the value matrix ^ a plurality of intermediate matrices based on the following equation: wherein: ^^^^^^^(_)denotes the softmax function, and ^^^[^] denotes a subblock of size ^^^ of the matrix ^. As already described above, ^ denotes the embedding size, i.e. a dimension of the query matrix ^ and the key matrix ^. As illustrated in figure 4, the attention layer 113 may further implement a processing stage 403 for generating the respective subblocks of size ^^^ of the query matrix ^, the key matrix ^, and the value matrix ^. As further illustrated in figure 4, the attention layer 113 may be further configured to output as the final result 409 a combination 407, in particular an average of the matrix product ^′^ of size ^^^ (provided by the LLN attention component 401) and the further matrix of size ^^^ (provided by the softmax attention component 405) based on the plurality of intermediate matrices of size ^^^ determined using the softmax function. In an embodiment, the attention layer 113a is configured to determine the further matrix of size ^^^ based on the plurality of intermediate matrices of size ^^^ by stacking or concatenating the plurality of intermediate matrices of size ^^^. Described in more mathematical terms, the attention layer 113a illustrated in figure 4 is configured for computing the D-block softmax attention 405 for each of the input matrices Q, K and V of size Nxd to slice into n blocks of size dxd Q1, …Qn, K1…Kn and V1…Vn respectively (see processing stage 403 of figure 4). The in processing stage 405 of figure 4 the softmax attention is applied independently on Qi *KiTand the output is multiplied by Vi as shown in Figure 4. The outputs of both the LLN attention layer component 401 and the D- block softmax attention 405 have a size of Nxd. These two outputs are averaged in processing stage 407 of figure 4 to produce the final output 409 of the attention layer 113a. As will be appreciated, the overall asymptotic memory and computational complexity of the attention layer 113a illustrated in figure 4 is still linear in N i.e. ^(^^) + ^(^^^ / ^) = ^(^^). Figure 5 is a flow diagram illustrating a computer-implemented data processing method 500 according to an embodiment. The data processing method 500 comprises a step 501 of operating the transformer neural network model 111, wherein, as already described above, the transformer neural network model 111 comprises the plurality of processing layers 113 for processing a plurality, i.e. a sequence of input elements 201 and wherein the plurality of processing layers 113 include an attention layer 113a for identifying short and long-term correlations between the plurality of input elements 201. The method 500 further comprises the step 503 of determining, by one or more of the plurality of processing layers 113 upstream of the attention layer 113a, for each of the plurality of input elements 201 a query vector ^^, a key vector ^^, and a value vector ^^, wherein the plurality of query vectors ^^define a query matrix ^, the plurality of key vectors ^^define a key matrix ^, and the plurality of value vectors ^^define a value matrix ^. Moreover, the method 500 comprises the step 505 of determining, by the attention layer 113a, as an intermediate result the matrix product ^ = ^′^^ and outputting as a final result the matrix product ^′^, wherein ^^= ^^^, and ^^= ^^^, and wherein ^ denotes a first parameter and ^ denotes a second parameter. For evaluating the memory requirements and running time of the data processing apparatus 100 and the data processing method 500 according to an embodiment memory consumption and time per iteration for Roberta and Bert base models has been measured, as illustrated in figures 6a and 6b. As expected, the conventional softmax attention (referred to as vanilla attention) has a quadratic memory and runtime dependency on N, while the attention layer 113a of figure 3b (referred to as LLN) and the attention layer 113a of figure 4 (referred to as LLN+) has a linear dependency for these measurables. For long sequences of 2k and more the attention layer 113a achieves more than 3x times reduction in memory and latency. For a specific GPU with 40Gb of memory the attention layer 113a allows to fit x3.5 times longer sequences than the conventional softmax attention. The person skilled in the art will understand that the "blocks" ("units") of the various figures (method and apparatus) represent or describe functionalities of embodiments of the present disclosure (rather than necessarily individual "units" in hardware or software) and thus describe equally functions or features of apparatus embodiments as well as method embodiments (unit = step). In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus, and method may be implemented in other manners. For example, the described apparatus embodiment is merely exemplary. For example, the unit division is merely logical function division and may be other division in actual implementation. For example, a plurality of units or components may be combined or integrated into another system, or some features may be ignored or not performed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections may be implemented by using some interfaces. The indirect couplings or communication connections between the apparatuses or units may be implemented in electronic, mechanical, or other forms. The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one position, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments. In addition, functional units in the embodiments of the present disclosure may be integrated into one processing unit, or each of the units may exist alone physically, or two or more units are integrated into one unit.

Claims

CLAIMS 1. A data processing apparatus (100), comprising a processing circuitry (110) configured to operate a transformer neural network (111), wherein the transformer neural network (111) comprises a plurality of processing layers (113) for processing a plurality of input elements (201), including an attention layer (113a); wherein one or more of the plurality of processing layers (113) upstream of the attention layer (113a) are configured to determine for each of the plurality of input elements (201) a query vector ^^, a key vector ^^, and a value vector ^^, wherein the plurality of query vectors ^^define a query matrix ^, the plurality of key vectors ^^define a key matrix ^, and the plurality of value vectors ^^define a value matrix ^; and wherein the attention layer (113a) is configured to determine as an intermediate result the matrix product ^ = ^′^^ and output as a final result the matrix product ^′^, whereinwherein ^ denotes a first parameter and ^ denotes a second parameter.

2. The data processing apparatus (100) of claim 1, wherein the attention layer (113a) is configured to output the matrix product ^′^ as the final result based on the following equation: ^^^^^^^ ∑^^^^^^^^.

3. The data processing apparatus (100) of claim 1 or 2, wherein the attention layer (113a) is configured to determine the first parameter ^ based on the following equation: ^wherein: ^^denotes ^^^^(^);^^denotes^, ^ denote constant values.

4. The data processing apparatus (100) of any one of the preceding claims, wherein the attention layer (113a) is configured to determine the second parameter ^ based on the following equation: ^wherein: ^^denotes ^^^^(^); ^^denotes ^^^^(^); and ^, ^ denote constant values.

5. The data processing apparatus (100) of any one of the preceding claims, wherein the attention layer (113a) is further configured to determine for a plurality of subblocks of the query matrix ^, the key matrix ^, and the value matrix ^ a plurality of intermediate matrices based on the following equation:wherein: ^^^^^^^(_)denotes the softmax function, ^^^[^] denotes a subblock of size ^^^ of the matrix ^, and ^ denotes an embedding dimension of the query matrix ^ and the key matrix ^.

6. The data processing apparatus (100) of claim 5, wherein the attention layer (113a) is further configured to output as the final result a combination of the matrix product ^′^ and a further matrix based on the plurality of intermediate matrices.

7. The data processing apparatus (100) of claim 6, wherein the attention layer (113a) is further configured to determine the further matrix based on the plurality of intermediate matrices by stacking or concatenating the plurality of intermediate matrices.

8. A data processing method (500), wherein the method comprises: operating (501) a transformer neural network (111), wherein the transformer neural network (111) comprises a plurality of processing layers (113) for processing a plurality of input elements (201), including an attention layer (113a); determining (503), by one or more of the plurality of processing layers (113) upstream of the attention layer (113a), for each of the plurality of input elements (201) a query vector ^^, a key vector ^^, and a value vector ^^, wherein the plurality of query vectors ^^define a query matrix ^, the plurality of key vectors ^^define a key matrix ^, and the plurality of value vectors ^^define a value matrix ^; and determining (505), by the attention layer (113a), as an intermediate result the matrix product ^ = ^′^^ and outputting as a final result the matrix product ^′^, whereinwherein ^ denotes a first parameter and ^ denotes a second parameter.

9. The data processing method (500) of claim 8, wherein outputting the final result comprises outputting the matrix product ^′^ as the final result based on the following equation: ^^^^^^^ ∑^^^^^^^^.

10. The data processing method (500) of claim 8 or 9, wherein the method (500) further comprises determining the first parameter ^ based on the following equation: ^wherein: ^^denotes ^^^^(^); ^^denotes ^^^^(^); and ^, ^ denote constant values.

11. The data processing method (500) of any one of claims 8 to 10, wherein the method (500) further comprises determining the second parameter ^ based on the following equation: ^wherein: ^^denotes ^^^^(^); ^^denotes ^^^^(^); and ^, ^ denote constant values.

12. The data processing method (500) of any one of the claims 8 to 11, wherein the method (500) further comprises the step of determining by the attention layer (113a) for a plurality of subblocks of the query matrix ^, the key matrix ^, and the value matrix ^ a plurality of intermediate matrices based on the following equation:wherein: ^^^^^^^(_) denotes the softmax function, ^^^[^] denotes a subblock of size ^^^ of the matrix ^, and ^ denotes an embedding dimension of the query matrix ^ and the key matrix ^.

13. The data processing method (500) of claim 12, wherein outputting the final result comprises outputting as the final result a combination of the matrix product ^′^ and a further matrix based on the plurality of intermediate matrices.

14. The data processing method (500) of claim 13, wherein the method (500) further comprises determining the further matrix based on the plurality of intermediate matrices by stacking or concatenating the plurality of intermediate matrices.

15. A computer program product comprising a computer-readable storage medium for storing program code which causes a computer or a processor to perform the method (500)of any one of claims 8 to 14, when the program code is executed by the computer or the processor.

Images