Computing device, chip, board card and method

Abstract

The invention discloses a computing device, a chip, a board card and a method. The computing device may be included in a combination processing device, which may further include an interface device and a processing device. And the computing device interacts with the processing device to jointly complete the computing operation specified by the user. The combined processing device can further comprise a storage device, and the storage device is connected with the computing device and the processing device and used for storing data of the computing device and the processing device. According to the scheme disclosed by the invention, a hardware implementation scheme of self-attention mechanism operation is provided, the operation performance in a throughput mode can be optimized, and IO (Input / Output) and computing power bandwidth of hardware are fully utilized.

Description

Technical Field

[0001] The present disclosure generally relates to the field of computing. More specifically, the present disclosure relates to a computing device, a chip, a board, and a method for performing self-attention mechanism operations using the computing device. Background Art

[0002] At present, Transformer networks have been widely used in the field of natural language processing (NLP), such as machine translation, question-answering systems, text summarization, and speech recognition. The Transformer network adopts an encoder-decoder architecture, and both the encoder and decoder include a self-attention mechanism. The self-attention mechanism is the core of the Transformer network, and its calculation process involves multiplication operations between three tensors: query (Q), key (K), and value (V).

[0003] Traditional AI chips use Tensor cores to accelerate the multiplication operations between the three tensors Q, K, and V when implementing the self-attention mechanism of the Transformer network decoder. Tensor core is a special functional unit introduced by NVIDIA in its Volta architecture. It is a specific hardware module of AI chips used to accelerate matrix multiplication operations. This method enables the multiplication operations between the three tensors Q, K, and V to be implemented efficiently.

[0004] However, when in throughput mode, that is, when the input data scale is very large, especially when the data batch is large, the Tensor core processes in batches in a cyclic manner, which will result in lower data input and output (IO) and computing power bandwidth utilization, leading to poor performance.

[0005] In view of this, it is desired to provide an improved solution so that the operation of the self-attention mechanism can be efficiently implemented even if the input data scale is very large. Summary of the invention

[0006] In order to at least solve one or more of the technical problems mentioned above, the present disclosure proposes an operation scheme of the self-attention mechanism in multiple aspects, which decomposes the matrix multiplication operation in the self-attention mechanism so that the operation can be split and executed on two operation components, thereby facilitating pipeline operations and improving data throughput.

[0007] In a first aspect, the present disclosure provides a computing device configured to perform a self-attention mechanism operation, characterized in that the matrix multiplication operation in the self-attention mechanism operation is decomposed into a positional multiplier operation and a matrix multiplier operation, and the computing device includes: a processing circuit configured to perform dimensional operations to convert related tensor data into a dimensional order and / or shape required for the operation or output, wherein the dimensional operations include: dimensional transposition and / or dimensional deformation; a vector operation circuit VFU configured to perform the positional multiplier operation, wherein the batch dimension n of the tensor data participates in the positional multiplier operation; and a tensor operation circuit TFU configured to perform the matrix multiplier operation, wherein the matrix multiplier operation performs a low-dimensional reduction sum operation on the corresponding input data.

[0008] In a second aspect, the present disclosure provides a chip comprising the computing device of the first aspect of the present disclosure.

[0009] In a third aspect, the present disclosure provides a board comprising the chip of the second aspect of the present disclosure.

[0010] In a fourth aspect, the present disclosure provides a method for performing self-attention mechanism operations by a computing device of the first aspect of the present disclosure.

[0011] Through the self-attention mechanism operation scheme provided above, the disclosed embodiment decomposes the matrix multiplication operation in the self-attention mechanism operation into a positional multiplier operation and a matrix multiplier operation for the throughput mode, which is convenient for execution on two different operation components, thereby realizing pipeline operation and improving data throughput. Furthermore, this operation decomposition is also conducive to integrating multiple batches of self-attention mechanism operations into a single self-attention mechanism operation, avoiding the problem of inefficient computing power and bandwidth caused by cyclic calculations, making full use of the parallel computing capabilities of intelligent computing hardware, and improving processing efficiency. Improvements in the performance of self-attention mechanism operations can effectively enhance product competitiveness. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] By reading the detailed description below with reference to the accompanying drawings, the above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0013] Figure 1 A structural diagram of a board according to an embodiment of the present disclosure is shown;

[0014] Figure 2 A structural diagram showing a combined processing device according to an embodiment of the present disclosure;

[0015] Figure 3A schematic diagram showing the internal structure of a single processor core of a single-core computing device according to an embodiment of the present disclosure;

[0016] Figure 4 A simplified schematic diagram showing the internal structure of a multi-core computing device according to an embodiment of the present disclosure;

[0017] Figure 5 The operation flow of the self-attention mechanism is schematically shown;

[0018] Figure 6 A schematic diagram of the operation process of the self-attention mechanism according to some embodiments of the present disclosure is shown;

[0019] Figure 7 An exemplary structural block diagram of a computing device configured to perform self-attention mechanism operations according to some embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0021] It should be understood that the terms "first", "second", "third", and "fourth" etc. that may appear in the claims, specifications, and drawings of the present disclosure are used to distinguish different objects rather than to describe a specific order. The terms "include" and "comprise" used in the specifications and claims of the present disclosure indicate the presence of the described features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their collections.

[0022] It should also be understood that the terms used in this disclosure are only for the purpose of describing specific embodiments and are not intended to limit the disclosure. As used in this disclosure and claims, the singular forms of "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It should also be further understood that the term "and / or" used in this disclosure and claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations.

[0023] As used in this specification and claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0024] The specific implementation of the present disclosure is described in detail below with reference to the accompanying drawings. Exemplary Hardware Environment

[0025] Figure 1 FIG. 1 is a schematic diagram showing the structure of a board 10 according to an embodiment of the present disclosure. Figure 1 As shown, the board 10 includes a chip 101, which is a system-level chip, or a system on chip (SoC), integrated with one or more combined processing devices, which are an artificial intelligence computing unit to support various deep learning and machine learning algorithms, and meet the intelligent processing needs in complex scenarios in the fields of computer vision, speech, natural language processing, data mining, etc. In particular, deep learning technology is widely used in the field of cloud intelligence. A significant feature of cloud intelligence applications is the large amount of input data, which has high requirements for the storage capacity and computing power of the platform. The board 10 of this embodiment is suitable for cloud intelligence applications, and has huge off-chip storage, on-chip storage and powerful computing power.

[0026] The chip 101 is connected to an external device 103 via an external interface device 102. The external device 103 is, for example, a server, a computer, a camera, a display, a mouse, a keyboard, a network card, or a wifi interface. The data to be processed can be transmitted from the external device 103 to the chip 101 via the external interface device 102. The calculation result of the chip 101 can be transmitted back to the external device 103 via the external interface device 102. According to different application scenarios, the external interface device 102 can have different interface forms, such as a PCIe interface.

[0027] The board 10 also includes a storage device 104 for storing data, which includes one or more storage units 105. The storage device 104 is connected to the control device 106 and the chip 101 through a bus and transmits data. The control device 106 in the board 10 is configured to control the state of the chip 101. To this end, in an application scenario, the control device 106 may include a microcontroller (MCU).

[0028] Figure 21 is a block diagram showing the combined processing device in the chip 101 of this embodiment. Figure 2 As shown in , the combined processing device 20 includes a computing device 201 , an interface device 202 , a processing device 203 and a storage device 204 .

[0029] The computing device 201 is configured to execute user-specified operations, and is mainly implemented as a single-core intelligent processor or a multi-core intelligent processor to perform deep learning or machine learning calculations. It can interact with the processing device 203 through the interface device 202 to jointly complete the user-specified operations.

[0030] The interface device 202 is used to transmit data and control instructions between the computing device 201 and the processing device 203. For example, the computing device 201 can obtain input data from the processing device 203 via the interface device 202 and write it into the storage device on the computing device 201 chip. Further, the computing device 201 can obtain control instructions from the processing device 203 via the interface device 202 and write them into the control cache on the computing device 201 chip. Alternatively or optionally, the interface device 202 can also read data in the storage device of the computing device 201 and transmit it to the processing device 203.

[0031] The processing device 203, as a general processing device, performs basic controls including but not limited to data handling, starting and / or stopping the computing device 201. Depending on the implementation, the processing device 203 can be a central processing unit (CPU), a graphics processing unit (GPU) or one or more types of processors in other general and / or special processors, these processors include but are not limited to digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, only with respect to the computing device 201 disclosed in the present invention, it can be regarded as having a single-core structure or a homogeneous multi-core structure. However, when the computing device 201 and the processing device 203 are integrated and considered together, the two are regarded as forming a heterogeneous multi-core structure.

[0032] The storage device 204 is used to store data to be processed, which may be DRAM or DDR memory, and is usually 16G or larger in size, and is used to store data of the computing device 201 and / or the processing device 203 .

[0033] When the computing device 201 runs the neural network, it is generally necessary to first compile the neural network using the processing device 203 to obtain an executable file, which contains device information, that is, which device in the heterogeneous computer system the executable file needs to be executed on. After the executable file is assembled and linked, an executable program of the neural network can be obtained, and the executable program is stored in the storage device 204.

[0034] The processing device 203 can read the executable program from the storage location of the executable program and obtain multiple tasks of the program according to the executable program. These tasks are distributed to the computing device 201 for execution via the interface device 202, and finally obtain the operation result.

[0035] Figure 3 Shows Figure 2 The internal structure diagram of the processing core when the computing device 201 in FIG. 2 is a single-core device. The computing device 301 is used to process input data such as computer vision, speech, natural language, and data mining. The computing device 301 includes three modules: a control module 31 (also called a controller), an operation module 32 (also called an operator), and a storage module 33 (also called a memory).

[0036] The control module 31 is used to coordinate and control the operation of the operation module 32 and the storage module 33 to complete the deep learning task, and includes an instruction fetch unit (IFU) 311 and an instruction decode unit (IDU) 312. The instruction fetch unit 311 is used to obtain instructions from the processing device 203, and the instruction decode unit 312 decodes the obtained instructions and sends the decoding results to the operation module 32 and the storage module 33 as control information.

[0037] The operation module 32 may include a vector operation unit VFU 321 and a tensor operation unit TFU 322. The vector operation unit 321 is used to perform vector operations and can support complex operations such as vector multiplication, addition, and nonlinear transformation; the tensor operation unit 322 is responsible for the core calculation of the deep learning algorithm, such as matrix multiplication and convolution.

[0038] The storage module 33 is used to store or transfer relevant data, including a neuron RAM (NRAM) 331, a weight RAM (WRAM) 332, and a direct memory access module (DMA) 333. NRAM 331 is used to store input neurons, output neurons, and intermediate results after calculation; WRAM 332 is used to store the convolution kernel of the deep learning network, that is, the weight; DMA 333 is connected to DRAM 204 via bus 34, and is responsible for data transfer between the computing device 301 and DRAM 204. It should be noted that the NRAM and WRAM here can be two storage areas formed by dividing the same memory in the logical storage space, or they can be two independent memories, which are not specifically limited here.

[0039] Figure 4 Shows Figure 2 The simplified schematic diagram of the internal structure of the computing device 201 in FIG. 2 is a multi-core computing device. A multi-core computing device can be abstracted using a hierarchical hardware model. As shown in the figure, the multi-core computing device 400 is a system on chip, which includes at least one computing cluster, and each computing cluster includes multiple processor cores. In other words, the multi-core computing device 400 is composed of a hierarchy of system on chip-computing cluster-processor core.

[0040] From the perspective of the system on chip (SoC) level, as shown in the figure, the multi-core computing device 400 includes an external storage controller 41 , a peripheral communication module 42 , an on-chip interconnect module 43 , a global synchronization module 44 , and multiple computing clusters 45 .

[0041] There may be multiple external storage controllers 41, and two are shown in the figure as an example, which are used to respond to access requests issued by the processor core and access external storage devices (such as Figure 2 The peripheral communication module 42 is used to communicate with the DRAM 204 in the chip to read data from the chip or write data. Figure 2 202) receives data from the processing device ( Figure 2 203) control signal, start the computing device ( Figure 2201) to perform tasks. The on-chip interconnect module 43 connects the external storage controller 41, the peripheral communication module 42 and multiple computing clusters 45 to transmit data and control signals between the modules. The global synchronization module 44 is, for example, a global synchronization barrier controller (GBC), which is used to coordinate the work progress of each computing cluster and ensure the synchronization of information. Multiple computing clusters 45 are the computing cores of the multi-core computing device 400. The figure shows 4 on each bare chip by way of example. With the development of hardware, the multi-core computing device 400 disclosed in the present disclosure may also include 8, 16, 64, or even more computing clusters 45. The computing clusters 45 are used to efficiently execute deep learning algorithms.

[0042] From the perspective of the computing cluster level, as shown in the figure, each computing cluster 45 includes multiple processor cores 406 as control and computing units, and a shared storage core 407 as a storage unit. Furthermore, each computing cluster may also include a local synchronization module 412 to coordinate the work progress of each processor core in the computing cluster and ensure information synchronization. The processor cores 406 are shown in the figure as an example, and the present disclosure does not limit the number of processor cores 406.

[0043] The storage core 407 is mainly used for storage and communication, that is, to store shared data or intermediate results between the processor cores 406, and to perform communication between the computing cluster 45 and the DRAM 204, communication between the computing clusters 45, and communication between the processor cores 406. In other embodiments, the storage core 407 has the ability of scalar operations and is used to perform scalar operations.

[0044] The storage core 407 includes a shared memory unit (SRAM) 408, a broadcast bus 409, a cluster direct memory access module (cluster direct memory access, CDMA) 410 and a global direct memory access module (global direct memory access, GDMA) 411. The SRAM 408 plays the role of a high-performance data transfer station. The data reused between different processor cores 406 in the same computing cluster 45 does not need to be obtained from the DRAM 204 by each processor core 406, but is transferred between the processor cores 406 through the SRAM 408. The storage core 407 only needs to quickly distribute the reused data from the SMEM 408 to multiple processor cores 406 to improve the efficiency of inter-core communication and greatly reduce on-chip and off-chip input / output access. The broadcast bus 409, CDMA 410 and GDMA 411 are used to perform communication between processor cores 406, communication between computing clusters 45 and data transmission between computing clusters 45 and DRAM 204 respectively.

[0045] At the processor core level, the structure of a single processor core can be similar to Figure 3 The structural diagram of the single-core computing device shown is not described in detail here. Self-attention mechanism operation scheme

[0046] Figure 5 The schematic diagram shows the operation flow of the self-attention mechanism.

[0047] In the self-attention mechanism, tensors are used to calculate self-attention. In the self-attention calculation, three tensors or intermediate variables are involved: query Q (Query) tensor, key K (Key) tensor and value V (Value) tensor. Q, K, V are obtained by linearly transforming the input of the self-attention mechanism, that is, each input feature is projected onto the three tensors Q, K, and V. Simply put, the Q tensor is used to match other units, the K tensor is used to be matched by other units, and the V tensor represents the extracted information.

[0048] As shown in the figure, in the operation of the self-attention mechanism, the correlation between the Q tensor and the K tensor is first calculated, that is, a matrix multiplication operation 510 is performed on the Q tensor and the K tensor to obtain an attention score, which is also the similarity between the two tensors. Next, the attention score is adjusted by scaling 520. Then, the scaled attention score is normalized by softmax 530. Finally, the normalized attention score is used as a weight to weight the V tensor, that is, a matrix multiplication operation 540 is performed with the V tensor to obtain an attention output.

[0049] The operation process of the above self-attention mechanism can be expressed as the following formula:

[0050] Among them, Q, K, and V are the query Q (Query) tensor, key K (Key) tensor, and value V (Value) tensor mentioned above, respectively. The superscript T indicates transposition. k Represents the dimension of K. The dimension of the Q tensor is [n,1,h], the dimension of the K tensor is [n,t,h], and the dimension of the V tensor is [n,t,h], where n represents the batch dimension, t represents the sequence length dimension, the input sequence length of the self-attention mechanism Q tensor in the decoder is 1, and h represents the number of heads in the multi-head attention layer.

[0051] As mentioned earlier, Tensor core is a dedicated processing unit for matrix calculations, which accelerates matrix multiplication and addition operations. When implementing the above self-attention mechanism, Tensor core will cyclically perform matrix multiplication operations between Q, K, and V according to batches. When in throughput mode, that is, when the input data scale is very large, especially when the data batch is large, the batch cyclic operation of Tensor core will result in lower data input and output (IO) and computing power bandwidth utilization, resulting in poor computing performance.

[0052] In view of this, in the disclosed embodiment, the hardware implementation of the self-attention mechanism operation is optimized to improve the operation performance in the throughput mode. On the one hand, by splitting the operation of the self-attention mechanism into a bitwise multiplication operation and a matrix multiplication operation, the self-attention mechanism can be combined with the previous Figure 3 These operations are implemented on the vector unit (VFU) and tensor unit (TFU) described in the paper, and the two operation steps can be separated to facilitate pipeline execution, thereby achieving greater throughput. On the other hand, based on the operation mode supported by VFU and TFU, by integrating the batch dimension into other dimensions for operation, the cyclic calculation process that requires multiple batches can be converted into a single calculation, thereby making full use of the computing power and IO bandwidth of the computing hardware and improving the computing efficiency.

[0053] Figure 6 A schematic diagram of the operation process of the self-attention mechanism according to some embodiments of the present disclosure is shown.

[0054] As shown in the figure, in step S1, three tensors Q, K, and V to be calculated are provided. Specifically, the Q tensor, K tensor, and V tensor are input into the Transformer decoder self-attention model, and the corresponding tensor dimensions are [n,1,h], [n,t,h], and [n,t,h], respectively, where n, t, and h have the same meanings as before. According to the principle of the Transformer model, it can be easily understood that these three tensors are obtained by linearly transforming the input of the self-attention mechanism. The improvement of the disclosed embodiment does not lie in this, so it will not be described in detail here.

[0055] Next, in step S2, dimension operations are performed on the three tensors Q, K, and V respectively to convert the relevant tensors into the dimension order and / or shape required for subsequent calculations. The dimension operation may include dimension transposition and / or dimension deformation.

[0056] Specifically, a dimensional transformation is performed on the Q tensor to transform its tensor dimension from three-dimensional [n, 1, h] to two-dimensional [n, h]. It can be seen that in this transformation, the batch dimension n is fused into the sequence length dimension t, so that operations that previously needed to be performed in multiple batches can be fused into one operation. This merging of adjacent dimensions does not change the storage order of the elements of the tensor data in memory, but only changes the description of the shape of the tensor.

[0057] Perform a dimension transposition on the K tensor to transform its tensor dimension from [n, t, h] to [t, n, h]. It can be seen that in this transformation, the batch dimension n and the sequence length dimension t are swapped. This dimension transposition will correspondingly change the storage order of each element of the tensor data in memory.

[0058] Similarly, the V tensor is transposed to transform its tensor dimension from [n, t, h] to [h, n, t]. It can be seen that each dimension has changed in this transformation. Similarly, this dimension transposition will correspondingly change the storage order of each element of the tensor data in memory.

[0059] Next, in step S3, the Q tensor (dimension is [n, h]) and the K tensor (dimension is [t, n, h]) obtained after the dimension operation in step S2 are respectively subjected to a bitwise multiplication operation, and the tensor dimension of the output operation result is [t, n, h]. The bitwise multiplication operation can be performed using a vector operation unit VFU, for example, by executing a vector multiplication vector instruction. As can be seen from this step, in the bitwise multiplication operation, the multiplication operation is performed on n batches of data at one time.

[0060] Next, in step S4, the operation result of step S3 is provided to the tensor operation unit TFU, and its matrix multiplication and addition operation function is used to implement low-dimensional reduction summation, and the tensor dimension of the reduction summation result is [t,n]. It can be seen that the operation of step S3 and step S4 is equivalent to splitting the matrix multiplication operation into two steps: positional multiplication and reduction summation, and integrating the batch dimension n into the operation, thereby completing the matrix multiplication operation between multiple batches of Q tensors and K tensors.

[0061] Next, in step S5, a softmax operation is performed on the operation result of step S4 to normalize the result. The tensor dimension of the output result of the softmax operation is [n, t].

[0062] Next, the matrix multiplication operation between the normalized result and the V tensor is performed. Similar to the previous step, this matrix multiplication operation is also split into two steps: bitwise multiplication and reduced summation, which makes it easier to merge the matrix multiplication operation between the normalized results of n batches and the V tensor into one operation.

[0063] Specifically, in step S6, the V tensor (dimension [h, n, t]) obtained after the dimension operation in step S2 and the result obtained in step S5 (dimension [n, t]) are subjected to a bitwise multiplication operation, and the tensor dimension of the output operation result is [h, n, t]. Similar to step S3, this step can also be performed using a vector operation unit VFU.

[0064] Next, in step S7, the operation result of step S6 is provided to the tensor operation unit TFU, and its matrix multiplication and addition operation function is used to implement low-dimensional reduction summation, and the tensor dimension of the reduction summation result is [h, n]. It can be seen that the operation of step S6 and step S7 is equivalent to splitting the matrix multiplication operation into two steps: positional multiplication and reduction summation, thereby completing the matrix multiplication operation between the normalized results of multiple batches and the V tensor.

[0065] Finally, in step S8, the dimension operation is performed on the operation result of step S7 to obtain the final output result, whose tensor dimension changes from [h,n] to [n,1,h], that is, the two-dimensional data is restored to three-dimensional data. It can be seen from the dimension change result that the dimension operation here includes dimension transposition and dimension deformation.

[0066] It can be seen from the embodiments described above that the implementation scheme uses a combination of VFU operation units to perform bitwise multiplication operations and TFU operation units to perform matrix multiplication operations (low-dimensional reduction summation) to replace the matrix operations of the traditional tensorcore module, which can make full use of the hardware IO and computing power bandwidth, and can handle a large amount of data at a time, and can maintain an absolute performance advantage without the need for powerful tensor core computing power.

[0067] Figure 7 An exemplary structural block diagram of a computing device configured to perform self-attention mechanism operations according to some embodiments of the present disclosure is shown.

[0068] As shown in the figure, the computing device 700 includes a processing circuit 710 , a vector operation circuit VFU 720 and a tensor operation circuit TFU 730 .

[0069] The processing circuit 710 may be responsible for processing various functions on the computing device 700, including but not limited to control (e.g., control unit 711), decoding (e.g., decoding unit 712), operation (e.g., operation unit 713), etc. The processing circuit 710 may include, for example Figure 3 The control module 31 in.

[0070] In some embodiments, the processing circuit 710 may be configured to perform dimensional operations on multidimensional data blocks to convert the related tensor data into a dimensional order and / or shape required for subsequent operations or outputs. Specifically, the dimensional operations may include dimensional transposition and / or dimensional deformation.

[0071] As described above, dimensional deformation only changes the description of the shape of the tensor and does not change the storage order of the elements of the tensor data in the memory, so the processing circuit only needs to adjust the shape description according to the corresponding instructions. Dimension transposition involves changing the storage order of data in the memory, so the processing circuit can work with the storage circuit (not shown in the figure) to realize data transfer, thereby adjusting the storage order to realize dimensional transposition.

[0072] The vector operation circuit VFU 720 is used to perform vector operations, which can support multiple operation modes of multiple data types, for example, it can support various common operations such as table lookup, addition, multiplication, pooling, sampling, edge expansion, vector comparison, and data format conversion.

[0073] In some embodiments, the vector operation circuit VFU 720 can be configured to perform vector-by-vector operations, thereby implementing the positional multiplier operations decomposed from the matrix multiplication operations in the disclosed embodiments. In these positional multiplier operations, the batch dimension n of the tensor data involved participates in the positional multiplier operations, that is, the n-dimensional data of the two tensor data involved in the operation will perform the positional multiplication operation, so that the operation that requires n batches to complete in the prior art can be implemented in a single operation.

[0074] The tensor operation circuit TFU 730 is used to perform matrix operations, such as convolutional layer and fully connected layer operations in deep learning algorithms. The tensor operation circuit can support matrix-vector multiplication, vector-matrix multiplication, matrix-scalar multiplication, outer product multiplication, matrix-matrix addition, matrix-matrix subtraction and other matrix-related operations.

[0075] In some embodiments, the tensor operation circuit TFU 730 may be configured to perform matrix-vector multiplication operations, thereby implementing matrix multiplier operations decomposed from matrix multiplication operations in the disclosed embodiments. In these matrix multiplier operations, low-dimensional reduction sum operations are performed on corresponding input data.

[0076] Therefore, through the cooperation of the vector operation circuit VFU and the tensor operation circuit TFU, the batch dimension n can be integrated into the operation, and the matrix multiplication operation that originally required n operations can be achieved through one operation.

[0077] In some embodiments, the matrix multiplication operation between the Q tensor and the K tensor can be decomposed as follows: the processing circuit 710 is further configured to: perform a first dimension operation on the key tensor K to convert the batch dimension n to the operation dimension of the bitwise multiplier operation; and perform a second dimension operation on the query tensor Q to convert the batch dimension n to the operation dimension of the bitwise multiplier operation.

[0078] For example, the dimension of the key tensor K is [n, t, h], and the first dimension operation transposes the dimension of the key tensor K to [t, n, h]. The dimension of the query tensor Q is [n, 1, h], and the second dimension operation transforms the dimension of the query tensor Q to [n, h].

[0079] Then, the vector operation circuit VFU is further configured to perform a bitwise multiplier operation on the key tensor K operated by the first dimension and the query tensor Q operated by the second dimension to output a first VFU result. That is, the n-dimension and h-dimension of the key tensor K are bitwise multiplied with the n-dimension and h-dimension of the query tensor Q, respectively, so as to realize n batches of operations at the same time. The dimension of the first VFU result is [t, n, h].

[0080] Accordingly, the tensor operation circuit TFU is further configured to perform a matrix multiplication operation on the first VFU result to sum the low-dimensionality of the first VFU result and output the first TFU result. The low dimension of the first VFU result is h, and the dimension of the first TFU result after the low-dimensionality reduction is [t, n]. Thus, the matrix multiplication operation of multiple batches of data between the Q tensor and the K tensor is completed.

[0081] Based on the operation principle of the self-attention mechanism, the computing device 700 also includes a softmax operation circuit 740, which is used to perform a softmax operation on the first TFU result to output a softmax operation result. The dimension of the softmax operation result is [n, t]. The scaling operation in the self-attention mechanism can be performed before the softmax operation or integrated into the softmax operation. The disclosed embodiment has no limitation in this regard.

[0082] Similarly, the matrix multiplication operation between the Softmax operation result and the V tensor can be similarly decomposed to incorporate the batch dimension n in a single operation.

[0083] In some embodiments, the matrix multiplication operation between the Softmax operation result and the V tensor can be decomposed as follows: The processing circuit 710 is further configured to: perform a third dimension operation on the value tensor V to convert the batch dimension n to the operation dimension of the bitwise multiplier operation.

[0084] For example, the dimension of the value tensor V is [n, t, h], and the third dimension operation transposes the dimension of the value tensor V to [h, n, t].

[0085] Then, the vector operation circuit VFU is further configured to perform a bitwise multiplier operation on the value tensor V and the softmax operation result after the third dimension operation to output a second VFU result. That is, the n-dimension and t-dimension of the value tensor V are bitwise multiplied with the n-dimension and t-dimension of the softmax operation result, respectively, so as to realize n batches of operations at the same time. The dimension of the second VFU result is [h, n, t].

[0086] Accordingly, the tensor operation circuit TFU is further configured to perform a matrix multiplication operation on the second VFU result to sum the low-dimensionality of the second VFU result and output the second TFU result. The low dimension of the second VFU result is t, and the dimension of the second TFU result after summing the low-dimensionality is [h, n]. Thus, the matrix multiplication operation of multiple batches of data between the Softmax operation result and the V tensor is completed.

[0087] Finally, according to the dimension requirement of the final output data, the processing circuit 710 can be further configured to perform a fourth dimension operation on the second TFU result to generate the result of the self-attention mechanism operation. For example, the dimension of the second TFU result is [h, n], and the fourth dimension operation transposes and transforms the dimension of the second TFU result to [n, 1, h].

[0088] Therefore, the disclosed embodiment provides a hardware implementation solution for the self-attention mechanism operation, which supports the fusion of multiple batches of data operations into a single operation in throughput mode, can make full use of the hardware's IO and computing bandwidth, maintain a large data throughput, and improve computing performance.

[0089] The present disclosure also provides a chip, which may include the aforementioned computing device. Furthermore, the present disclosure also provides a board, which may include the aforementioned chip.

[0090] According to different application scenarios, the electronic equipment or device disclosed herein may include servers, cloud servers, server clusters, data processing devices, robots, computers, printers, scanners, tablet computers, smart terminals, PC devices, IoT terminals, mobile terminals, mobile phones, driving recorders, navigators, sensors, cameras, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, automatic driving terminals, transportation, household appliances, and / or medical equipment. The transportation includes airplanes, ships and / or vehicles; the household appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, electric lights, gas stoves, and range hoods; the medical equipment includes magnetic resonance imaging, ultrasound machines and / or electrocardiographs. The electronic equipment or device disclosed herein may also be applied to the Internet, IoT, data centers, energy, transportation, public administration, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, and medical fields. Further, the electronic equipment or device disclosed herein may also be used in cloud, edge, and terminal applications related to artificial intelligence, big data, and / or cloud computing. In one or more embodiments, electronic devices or devices with high computing power according to the disclosed solution can be applied to cloud devices (such as cloud servers), while electronic devices or devices with low power consumption can be applied to terminal devices and / or edge devices (such as smart phones or cameras). In one or more embodiments, the hardware information of the cloud device and the hardware information of the terminal device and / or edge device are compatible with each other, so that according to the hardware information of the terminal device and / or edge device, appropriate hardware resources can be matched from the hardware resources of the cloud device to simulate the hardware resources of the terminal device and / or edge device, so as to complete the unified management, scheduling and collaborative work of end-to-end or cloud-edge-to-end.

[0091] It should be noted that, for the purpose of simplicity, the present disclosure describes some methods and embodiments thereof as a series of actions and combinations thereof, but those skilled in the art will appreciate that the scheme of the present disclosure is not limited by the order of the actions described. Therefore, based on the disclosure or teaching of the present disclosure, those skilled in the art will appreciate that some of the steps therein may be performed in other orders or simultaneously. Further, those skilled in the art will appreciate that the embodiments described in the present disclosure may be regarded as optional embodiments, i.e., the actions or modules involved therein may not necessarily be necessary for the implementation of one or some of the schemes of the present disclosure. In addition, depending on the different schemes, the present disclosure also has different focuses on the description of some embodiments. In view of this, those skilled in the art may appreciate the parts that are not described in detail in a certain embodiment of the present disclosure, and may also refer to the relevant descriptions of other embodiments.

[0092] In terms of specific implementation, based on the disclosure and teaching of this disclosure, those skilled in the art can understand that several embodiments disclosed in this disclosure can also be implemented by other methods not disclosed herein. For example, with respect to the various units in the electronic device or device embodiments described above, this article splits them on the basis of considering the logical functions, and there may be other splitting methods in actual implementation. For another example, multiple units or components can be combined or integrated into another system, or some features or functions in the units or components can be selectively disabled. In terms of the connection relationship between different units or components, the connection discussed in the above text in conjunction with the accompanying drawings can be a direct or indirect coupling between units or components. In some scenarios, the aforementioned direct or indirect coupling involves a communication connection using an interface, wherein the communication interface can support electrical, optical, acoustic, magnetic or other forms of signal transmission.

[0093] In the present disclosure, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed on multiple network units. In addition, according to actual needs, some or all of the units may be selected to achieve the purpose of the scheme described in the embodiments of the present disclosure. In addition, in some scenarios, multiple units in the embodiments of the present disclosure may be integrated into one unit or each unit may exist physically separately.

[0094] In some other implementation scenarios, the above-mentioned integrated unit can also be implemented in the form of hardware, that is, a specific hardware circuit, which may include digital circuits and / or analog circuits, etc. The physical implementation of the hardware structure of the circuit may include but is not limited to physical devices, and the physical devices may include but are not limited to devices such as transistors or memristors. In view of this, the various devices described herein (such as computing devices or other processing devices) can be implemented by appropriate hardware processors, such as central processing units, GPUs, FPGAs, DSPs, and ASICs, etc. Further, the aforementioned storage unit or storage device may be any appropriate storage medium (including magnetic storage media or magneto-optical storage media, etc.), which may be, for example, a variable resistive memory (Resistive Random Access Memory, RRAM), a dynamic random access memory (Dynamic Random Access Memory, DRAM), a static random access memory (Static Random Access Memory, SRAM), an enhanced dynamic random access memory (Enhanced Dynamic Random Access Memory, EDRAM), a high bandwidth memory (High Bandwidth Memory, HBM), a hybrid memory cube (Hybrid Memory Cube, HMC), ROM and RAM, etc.

[0095] The embodiments of the present disclosure are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present disclosure. The description of the above embodiments is only used to help understand the method of the present disclosure and its core idea. At the same time, for those skilled in the art, according to the ideas of the present disclosure, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present disclosure.

Claims

1. A computing device configured to perform a self-attention mechanism operation, characterized in that: The matrix multiplication operation in the self-attention mechanism operation is decomposed into a positional multiplier operation and a matrix multiplier operation, and the computing device includes: Processing circuitry configured to perform dimensional operations to convert the relevant tensor data into a dimensional order and / or shape required for operation or output, wherein the dimensional operations include: dimensional transposition and / or dimensional deformation; a vector operation circuit VFU configured to perform the bitwise multiplier operation, wherein the batch dimension n of the tensor data participates in the bitwise multiplier operation; and The tensor operation circuit TFU is configured to perform the matrix multiplier operation, wherein the matrix multiplier operation performs a low-dimensional reduction summation operation on the corresponding input data.

2. The computing device of claim 1 , wherein the processing circuit is further configured to: Performing a first dimension operation on the key tensor K in the self-attention operation to convert the batch dimension n to the operation dimension of the bitwise multiplier operation; and A second dimension operation is performed on the query tensor Q in the self-attention operation to convert the batch dimension n to the operation dimension of the bitwise multiplier operation.

3. The computing device of claim 2, wherein: The vector operation circuit is further configured to perform a bitwise multiplier operation on the key tensor K operated by the first dimension and the query tensor Q operated by the second dimension to output a first VFU result; The tensor operation circuit is further configured to perform a matrix multiplication operation on the first VFU result to sum the low-dimensional reduction of the first VFU result and output a first TFU result.

4. The computing device of claim 3, wherein: The dimension of the key tensor K is [n, t, h], and the first dimensional operation transposes the dimension of the key tensor K to [t, n, h]; The dimension of the query tensor Q is [n, 1, h], and the second dimension operation transforms the dimension of the query tensor Q into [n, h], where n represents the batch, t represents the sequence length, and h represents the number of heads.

5. The computing device according to any one of claims 3 to 4, further comprising: The softmax operation circuit is configured to perform a softmax operation on the first TFU result to output a softmax result.

6. The computing device of claim 5, wherein the processing circuit is further configured to: Perform a third dimension operation on the value tensor V in the self-attention operation to convert the batch dimension n to the operation dimension of the bitwise multiplier operation.

7. The computing device of claim 6, wherein: The vector operation circuit is further configured to perform a bitwise multiplier operation on the value tensor V operated on the third dimension and the softmax result to output a second VFU result; The tensor operation circuit is further configured to perform a matrix multiplication operation on the second VFU result to sum the low-dimensional reduction of the second VFU result and output a second TFU result.

8. The computing device according to any one of claims 6-7, wherein: The dimension of the value tensor V is [n, t, h], and the third dimension operation transposes the dimension of the value tensor V to [h, n, t], where n represents the batch, t represents the sequence length, and h represents the number of heads.

9. A computing device according to any one of claims 6-7, wherein the processing circuit is further configured to perform a fourth dimension operation on the second TFU result to generate a result of the self-attention mechanism operation.

10. The computing device of claim 9, wherein: The dimension of the second TFU result is [h, n], and the fourth dimension operation transposes and transforms the dimension of the second TFU result to [n, 1, h], where n represents the batch and h represents the number of heads.

11. A chip, characterized in that: The chip includes a computing device as described in any one of claims 1-10.

12. A board, characterized in that: The board includes the chip according to claim 11.

13. A method for performing self-attention mechanism operations using a computing device as described in any one of claims 1-10.

Images