Arithmetic processing unit

Abstract

To provide an arithmetic processing device capable of accelerating calculation of a network model which is based on an attention mechanism.SOLUTION: In each inside of M pieces of arithmetic units 11 of an arithmetic processing device 10, one set of column vectors [am, bm] among a first matrix a and a second matrix b is stored in a m-th arithmetic unit. The M pieces of arithmetic units are connected with a series network 14, and from an arbitrary arithmetic unit, array data of an N-dimensional column vector bm is transmitted and is sequentially transferred to arithmetic units adjoining a subsequent stage. A dot product of each of column vectors b0, b1,b2,-, bM-1 of the second matrix and am stored inside of the arithmetic unit, being an M-dimensional column vector xm=[am x b0, am x b1, am x b2,---, am x bM-1]T composing a part of an array of an M×M-dimensional third matrix is calculated in an m-th arithmetic unit. The M-dimensional column vector xm is stored in the m-th arithmetic unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a computing system, and more particularly to a computing processing unit that efficiently performs machine learning processing. [Background technology]

[0002] General-purpose computing on GPUs (GPGPUs), powered by Graphical Processing Units (GPUs), has dramatically improved the speed of machine learning.

[0003] On the other hand, machine learning network models, including those that use not only simple convolution operations but also dot product operations of vectors known as attention mechanisms, are increasingly being recognized for their useful applications in areas such as automatic translation and image processing (see, for example, Patent Document 1, Non-Patent Documents 1 and 2). [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2022-019422 [Non-patent literature]

[0005] [Non-Patent Document 1] A. Vaswani et al., “Attention is all you need”, 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA, Dec. 4-9, 2017. https: / / proceedings.neurips.cc / paper / 2017 / file / 3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf [Non-Patent Document 2] A. Dosovitskiy et al., “An image is worth 16x16 words: transformers for image recognition at scale”, The 9th International Conference on Learning Representations (ICLR 2021), Virtual Only, May 3-7, 2021. https: / / iclr.cc / virtual / 2021 / poster / 3013 [Overview of the project] [Problems that the invention aims to solve]

[0006] Image processing units are generally designed to be highly versatile, capable of performing many types of matrix operations, and their specific processing is carried out by software libraries running on the unit. In this case, it is known that versatility leads to increased power consumption and reduced speed due to the inclusion of software processing. On the other hand, creating a completely dedicated circuit has the disadvantage of being unable to perform other calculations.

[0007] The object of the present invention is to provide a processing unit that can speed up the calculation of network models based on attention mechanisms (hereinafter also referred to as "attention"). [Means for solving the problem]

[0008] According to one aspect of the present invention, the processing unit is an M × N dimensional first matrix a = [a 0, a 1, a 2,…, a M-1 ] and the second matrix b = [ b 0, b 1, b 2,…, b M-1 Regarding ], a third M × M dimension matrix x = [x ] is formed by M M-dimensional column vectors whose elements are the dot product of the first matrix and all the column vectors belonging to the second matrix.0, x 1, x 2,…, x M-1 for the calculation of the matrix product x = a T b, when calculating by the above arithmetic processing unit, inside each of the M arithmetic units which are the minimum unit of the configuration of the above arithmetic processing unit, the above first matrix a = [a 0, a 1, a 2,…, a M-1 and the above second matrix b = [b 0, b 1, b 2,…, b M-1 among the column vector groups [a m, b m are stored one set in the m-th arithmetic unit, the above M arithmetic units are connected in a serial network, and the array data of the N-dimensional column vector b m is sent out from any of the above arithmetic units and propagated sequentially to the adjacent arithmetic unit in the subsequent stage, and each column vector b 0, b 1, b 2,…, b M-1 of the above second matrix and a m stored inside the arithmetic unit are dot-producted, that is, the M-dimensional column vector x<src="https: / / patentscope.wipo.int / search / en / detail.jsf?docId=WO2019030762A1&recNum=0000042" label="0000042"> = [a m ·b 0, a m ·b 1, a m ·b 2,…, a m ·b M-1 T is calculated inside the m-th arithmetic unit, and this M-dimensional column vector x​​​​​​​​​​​​​​​0, c 1, c 2,…, c M-1 ] and the matrix product y=[y 0, y 1, y 2,…, y M-1 The column vector y, which makes up this part of the data of the fifth matrix generated by ]=cx, is stored in the m-th operation unit. m In other words

number

[0010] In each of the above embodiments, the processing unit uses column vectors d that constitute the M×L-dimensional matrix. m =[d m0, d m1, d m2,…, d m(L-1) ] T When performing linear layer calculations for M channels, an N×L dimensional coefficient matrix U or an N dimensional bias vector V is sequentially input from an external source to the series-connected network and propagated, and the column vector d stored in advance in each m-th processing unit is used. m Between,

number

[0011] In each of the above embodiments, the processing unit, when performing multi-head attention, has a network for the number of heads H and a set of the M processing units, divides the number of rows of the coefficient matrix of the linear layer immediately before head division by the number of heads H, sequentially inputs the divided division matrices to the communication channel for the h-th head, and the shared vector data d m Between,

number

[0012] The processing unit may have a function to combine the multi-headed attention vectors by dividing the number of columns in the coefficient matrix of the linear layer immediately after combining by the number of heads H, sequentially inputting the divided matrix into the h-th head communication channel, summing the partial sums divided for each head, and storing them within a predetermined processing unit.

[0013] In each of the above embodiments, the arithmetic processing unit has a network whose basic structure consists of a data reading daisy chain that sequentially reads data into the network and a data input daisy chain that inputs data to the arithmetic unit. The output of the data reading daisy chain is connected to the data input daisy chain, so that the read data is sequentially input in series to the arithmetic unit.

[0014] Furthermore, in each of the above embodiments, the arithmetic processing unit may propagate the arithmetic code sequentially and serially to the network along with the data.

[0015] In each of the above embodiments, the arithmetic processing unit may simulate multi-head processing by sequentially performing array transfer and sum-of-accumulate operations in order of the heads, in order to process multi-head column vector data without adding any hardware.

[0016] The arithmetic processing unit may have circuit elements that divide the network according to the number of channels or sequences and connect the data read daisy chain and the data input daisy chain, so as to be able to handle problems with different numbers of channels or sequences. [Effects of the Invention]

[0017] According to the above embodiment, with the exception of the process of adding partial sums when merging matrices, the network can perform almost all operations by sending data only to adjacent operation elements. Therefore, high mutually independent parallelism is achieved, making it possible to speed up the computation of attention-based network models. [Brief explanation of the drawing]

[0018] [Figure 1] This figure shows the general configuration of a processing unit according to one embodiment of the present invention. [Figure 2] This is an explanatory diagram illustrating the operation of calculating the dot product xm = [am·b0, am·b1, am·b2, ..., am·bM-1]T in one embodiment of the present invention. [Figure 3] This diagram illustrates the process of obtaining M N-dimensional column vectors by multiplying an M×M-dimensional matrix consisting of M M-dimensional column vectors xm and a matrix c=[c0,c1,c2,…,c(M-1)] consisting of M N-dimensional column vectors, in one embodiment of the present invention. [Figure 4] This is an explanatory diagram illustrating the operation of processing a linear layer in one embodiment of the present invention. [Modes for carrying out the invention]

[0019] Next, embodiments of the present invention will be described with reference to the drawings.

[0020] Figure 1 is a diagram showing the schematic configuration of an arithmetic processing unit according to one embodiment of the present invention. Referring to Figure 1, the arithmetic processing unit 10 has a plurality of arithmetic units 11. The plurality of arithmetic units 11 are connected to a network 14 that exchanges data via a first daisy chain 12 that propagates data to the right and a second daisy chain 13 that propagates data to the left. The first and second daisy chains 12 and 13 are connected by a first multiplexer 15, and it is also possible to directly input data to the first daisy chain 12 from an external source. This set of components is referred to as an arithmetic unit set 16.

[0021] The arithmetic processing unit 10 has H sets of independent arithmetic units 16 to perform partition matrix processing, i.e., multi-head processing, in H parallel. After the partition matrix processing is completed, instead of joining the matrices, an addition network 18 is arranged orthogonally to the first and second daisy chains 12 and 13 to sum the H partitioned partial sums when performing the dot product of the partitioned rows and columns. If the cost of implementing the arithmetic unit is high for this purpose, the processing may be carried out separately by a CPU (Central Processing Unit) or the like. Furthermore, the first and second daisy chains 12 and 13 are divided according to the number of dimensions of the vector to be processed, and a second multiplexer 19 is provided to appropriately propagate the data. The following describes how to perform dot product and linear layer calculations in this embodiment.

[0022] Figure 2 shows the set x of the dot product in one embodiment of the present invention. m =[a m ·b 0, a m ·b 1, a m ·b 2,…, a m ·b M-1 ] TIt is an explanatory diagram of an operation for obtaining 0, a 1, a 2,…, a M-1 and the second matrix b = [b 0, b 1, b 2,…, b M-1 , the product of the matrices a T b is used to calculate M M-dimensional column vectors x m = [a m ·b 0, a m ·b 1, a m ·b 2,…, a m ·b M-1 T , m = 0, 1, 2, ···, M - 1. The set of arithmetic units 16 of the arithmetic processing unit 10 has M arithmetic units 11 which are the minimum constituent units. Let the ID of the arithmetic unit 11 be m = 0, 1, 2, …, M - 1. One arithmetic unit 11 includes a scratch pad SRAM 20 inside and stores an N-dimensional array a m and b m inside the arithmetic unit 11. The arrays a m and b m are column vectors.

[0023] The product of the matrices x = a T b, that is, M M-dimensional column vectors x m = [a m ·b 0, a m ·b 1, a m ·b 2,…, a m ·b M-1 T When calculating, the set of arithmetic units 16 uses b​​m The elements of are passed from the arithmetic unit 11 to the second daisy chain 13, b0 = [b 00 ,b 01 ,b 02 ,…,b 0(N-1) ], b1=[b 10 ,b 11 ,b 12 ,…,b 1(N-1) ], b²=[b 20 ,b 21 ,b 22 ,…,b 2(N-1) ], ..., b M-1 =[b (M-1)0 ,b (M-1)1 ,b (M-1)2 ,…,b (M-1)(N-1) Each array is sent out sequentially as shown in ], transferred to the first daisy chain 12 by the first multiplexer 15, and the data b is sent sequentially from m=0 in the arithmetic unit 11. mn The multiply-accumulate operation is performed upon receiving the data. By creating array data b in this order, the product can be accumulated sequentially. The resulting N-dimensional column vector x m The data is stored again in the scratchpad SRAM 20. On the other hand, if the initial sequence length is M=2, the subsequent m=2 and beyond are routed by the second multiplexer 19 to the second daisy chain 13, allowing for the handling of a different sequence than the sequence length 2 using m=0,1. Those skilled in the art will understand that this operation corresponds to, for example, the dot product attention of query q and key k in a Transformer.

[0024] Figure 3 shows M M-dimensional column vectors x in one embodiment of the present invention. m A matrix consisting of an M×M-dimensional matrix and a matrix c=[c0,c1,c2, … ,c (M-1) This is an explanatory diagram of the operation to obtain M N-dimensional column vectors by multiplying the matrices of ]. In Figure 3, the operation unit set 16 is shown as an example when the first sequence length is 2. Referring to Figure 3, the matrix x = [x 0, x 1, x2,…, x M-1 ] and a matrix c=[c0,c1,c2, … ,c (M-1) This describes a computational method for generating M N-dimensional column vectors by multiplying matrices of ]. That is,

number

[0025] Figure 4 is an explanatory diagram of the operation of processing a linear layer in one embodiment of the present invention, and shows how to perform calculations on the linear layer in the structure shown in Figure 1. Referring to Figure 4, M channels or M sequences of L-dimensional vector data, i.e., M L-dimensional arrays d ml From this, M N-dimensional array z, represented by the following formula m

number

[0026] Figures 2 to 4 describe the case where partitioning matrix processing is not performed. When partitioning matrix processing is introduced, as shown in Figure 1, the arithmetic processing unit 10 has a set of arithmetic units 16 equal to the number of heads H, and the number of rows of the coefficient matrix of the linear layer immediately before head partitioning is divided and input sequentially to the communication channel for each head, and the array z expressed by the following formula between it and the shared operand vector data hm

number

[0027] Furthermore, when combining multi-head matrices, this can be implemented by dividing the coefficient matrix of the linear layer, which is used for calculations immediately after combining, in the column direction and sequentially inputting it into the communication channels for each head. The partial sums divided by the heads are then added together by the addition network 18 shown in Figure 1 and stored inside a predetermined calculation unit.

[0028] Since the arithmetic code can be executed by sequentially propagating it to the arithmetic unit 11 in order to execute the instruction, it only needs to be transmitted along the first daisy chain 12 and / or the second daisy chain 13 of the network 14.

[0029] In Figure 1, the number of arithmetic units 11 was increased according to the number of heads H. However, if the number of arithmetic resources is limited, it is also possible to simulate multi-head processing by only increasing the memory capacity of the scratchpad SRAM 20 and performing array transfer and sum-of-accumulate operations sequentially in order of the heads.

[0030] Furthermore, as illustrated in Figure 2 and its explanation, in order to address problems with different numbers of channels or sequences, the network 14 is divided by the second multiplexer 19 according to the number of channels or sequences, and the first daisy chain 12 for data reading and the second daisy chain 13 for input are joined at their respective short portions, thereby forming multiple arithmetic processing paths corresponding to short sequences. [Explanation of Symbols]

[0031] 10 Arithmetic Processing Unit 11 arithmetic units 12. First Daisy Chain 13. Second Daisy Chain 14 Networks 15. First Multiplexer 16 arithmetic unit set 18 Addition Network 19. Second Multiplexer 20 SRAM Scratchpad

Claims

1. The calculation for obtaining an \(M\times M\) - dimensional third matrix \(x = [x\) ０， a １， a ２，…， a Ｍ－１ and a second matrix \(b = [b\) ０， b １， b ２，…， b Ｍ－１ , which consists of \(M\) \(M\) - dimensional column vectors having dot - products of all combinations of column vectors belonging to the first matrix and the second matrix as elements, is obtained by the matrix product \(x=a\) ０， x １， x ２，…， x Ｍ－１ is obtained by the matrix product \(x = a\) Ｔ b\), when Inside each of the M arithmetic units, which are the smallest constituent units of the arithmetic processing unit, the first matrix a = [a ０， a １， a ２，…， a Ｍ－１ ] and the second matrix b = [ b ０， b １， b ２，…， b Ｍ－１ ] of the set of column vectors [a ｍ， b ｍ A set of ] is stored in the mth arithmetic unit, the M arithmetic units are connected in a series network, and an N-dimensional column vector b is generated from any of the arithmetic units. ｍ The array data is sent out and sequentially propagated to the adjacent processing units in the subsequent stages, and each column vector b of the second matrix ０， b １， b ２，…， b Ｍ－１ and a stored inside the processing unit ｍ The dot product of the M x M matrix, i.e., the M-dimensional column vector x that forms part of the array of the third M x M matrix. ｍ = [a ｍ ・b ０， a ｍ ・b １， a ｍ ・b ２，…， a ｍ ・b Ｍ－１ ] Ｔ The m-th arithmetic unit calculates this, and this M-dimensional column vector x ｍ The arithmetic processing unit, which does this by storing the mth arithmetic unit inside the mth arithmetic unit, where a ｍ , b ｍ and x ｍ This is a column vector that is stored as an array inside the arithmetic unit.

2. The column vector x belonging to the third matrix stored inside the m-th arithmetic unit ｍ And the pre-stored M x N dimension fourth matrix c = [c ０， c １， c ２，…， c Ｍ－１ ] and the matrix product y = [y ０， y １， y ２，…， y Ｍ－１ The column vector y, which forms this part of the data of the fifth matrix generated by ] = cx, is stored in the m-th operation unit. ｍ In other words [Number 7] When calculating, Using the network in which the aforementioned arithmetic units are connected in series, an N-dimensional column vector c stored inside each arithmetic unit ｍ The elements of c ｍ０ , c ｍ１ , c ｍ２ , …, c ｍ（Ｎ－１） The following are sent onto the network in this order, and the N-dimensional column vector y ｍ The arithmetic processing apparatus according to claim 1, which calculates and stores within each m-th arithmetic unit.

3. Column vectors d that make up an M x L ​​dimensional matrix ｍ = [d ｍ０， d ｍ１， d ｍ２，…， d ｍ（Ｌ－１） ] Ｔ When performing linear layer calculations for M channels, an N×L dimensional coefficient matrix U or an N dimensional bias vector V is sequentially input from an external source to the series-connected network and propagated, and the column vector d stored in advance in each m-th arithmetic unit is used. ｍ Between, [Number 8] Calculate the mth N-dimensional column vector z ｍ The arithmetic processing device according to claim 1 or 2, wherein the values ​​are stored within each arithmetic unit, where u and v represent elements of the coefficient matrix U and bias vector V, respectively.

4. When performing multi-head attention, the network has a number of heads H and a set of M processing units, the number of rows in the coefficient matrix of the linear layer immediately before head division is divided by the number of heads H, and the divided division matrices are sequentially input to the communication channel for the h-th head, and the shared vector data d ｍ Between, 【Number 9】 The arithmetic processing device according to claim 1 or 2, which performs multi-head operation by calculating the following.

5. The arithmetic processing unit according to claim 4, which, when performing multi-heading of attention, has the function of dividing the number of columns of the coefficient matrix of the linear layer immediately after coupling by the number of heads H in order to combine the multi-headed vectors, sequentially inputting the divided division matrices into the communication channels for the h-th head, summing the divided partial sums for each head and storing them inside a predetermined arithmetic unit.

6. The arithmetic processing apparatus according to claim 1 or 2, having a network whose basic structure includes a data reading daisy chain for sequentially reading data into the network and a data input daisy chain for inputting data to the arithmetic unit, wherein the output of the data reading daisy chain is connected to the data input daisy chain, and the read data is sequentially input in series to the arithmetic unit.

7. The arithmetic processing device according to claim 6, which sequentially propagates arithmetic codes along with data to the network.

8. The arithmetic processing apparatus according to claim 1 or 2, which simulates multi-head processing by sequentially performing array transfer and sum-of-accumulate operations in the order of the heads, in order to process multi-head column vector data without adding any hardware.

9. The arithmetic processing device according to claim 6, further comprising a circuit element that divides the network according to the number of channels or sequences and connects the data read daisy chain and the data input daisy chain, in order to address problems with different numbers of channels or sequences.

Images