Arithmetic processing device

Abstract

Provided is an arithmetic processing device comprising a plurality of arithmetic unit sets including M arithmetic units. Each of the arithmetic unit sets stores, in a SRAM or a register file in the m-th arithmetic unit, one set, [am, bm], of a column vector among a first matrix a = [a0, ..., aM -1] and a second matrix b = [b0, ..., bM -1] . The array data of the N-dimensional column vectors bm are sent out from each arithmetic unit in a first direction over a network connecting adjacent arithmetic units in series, and are sequentially propagated to the subsequent adjacent arithmetic unit, an M-dimensional column vector xm = [am・b0, ..., am・bM -1]T forming the dot product of each of a column vector b0, ..., bM -1 of the second matrix and am stored in the arithmetic unit, i.e., a portion of an array of an M×M-dimensional third matrix, is calculated by a data path integrated by an adder and a multiplier of the m-th arithmetic unit, and the M-dimensional column vector xm is stored in the SRAM or the register file of the m-th arithmetic unit.

Description

Processing Unit

[0001] The present disclosure relates to a computing system, and more particularly to a computing device that efficiently performs machine learning processing.

[0002] General purpose computing on a Graphical Processing Unit (GPU) (GPGPU) has dramatically improved the speed of machine learning.

[0003] On the other hand, machine learning network models that use not only simple convolution operations but also vector dot product operations known as attention mechanisms have become known to have many useful applications, such as automatic translation and image processing (see, for example, Patent Document 1, Non-Patent Documents 1 and 2).

[0004] Japanese Patent Application Laid-Open No. 2022-019422

[0005] A. Vaswani et al., “Attention is all you need”, 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA, Dec. 4-9, 2017. https: / / proceedings.neurips.cc / paper / 2017 / file / 3f5ee243547dee91fbd053c1c4a845aa-Paper.pdfA. Dudovskiy et al., “An image is worth 16x16 words: transformers for image recognition at scale”, The 9th International Conference on Learning Representations (ICLR 2021), Virtual Only, May 3-7, 2021. https: / / iclr.cc / virtual / 2021 / poster / 3013

[0006] Image processing arithmetic units are generally designed to be highly versatile, capable of performing many types of matrix operations, with specific processing performed by software libraries running on the unit. In this case, it is known that the versatility results in increased power consumption and a reduction in speed due to software processing. On the other hand, creating a completely dedicated circuit has the disadvantage of being unable to perform other calculations.

[0007] An object of the present disclosure is to provide a processing device that can speed up calculations of a network model based on an attention mechanism (hereinafter also referred to as "attention").

[0008] According to one aspect of the present disclosure, there is provided an arithmetic processing device including a plurality of arithmetic unit sets, each of the plurality of arithmetic unit sets including M arithmetic units (M is an integer of 2 or more) connected in a serial network, each of the M arithmetic units having at least one SRAM or register file, a multiplier and an adder, a port for selecting the SRAM or register file and outputting data to the multiplier, the SRAM or register file having an area for storing the data as an array, an input port for inputting data via the network from the arithmetic unit adjacent in a second direction, and an output port for outputting data to the network for transmission to the arithmetic unit adjacent in a first direction, a data path for inputting the data input from the input port and the data stored in the SRAM or register file to the multiplier and accumulating the multiplication result by the adder, and a data path for inputting the accumulation result as array data to the SRAM or register file, ０， a １， a ２，…， a Ｍ－１ ] and the second matrix b = [b ０， b １， b ２，…， b Ｍ－１ ], one set of column vectors [a ｍ， b ｍ] is stored in the SRAM or register file of the mth arithmetic unit, and an N-dimensional column vector b ｍ The array data is sent in the first direction through the network and propagated to the adjacent arithmetic unit in the subsequent stage, and each column vector b ０， b １， b ２，…， b Ｍ－１ and a stored in the arithmetic unit ｍ the dot product of the M-dimensional column vector x ｍ = [a ｍ ・b ０， a ｍ ・b １， a ｍ ・b ２，…， a ｍ ・b Ｍ－１ ] Ｔ is calculated by a data path that multiplies the multiplier and adder of the m-th arithmetic unit, and the M-dimensional column vector x ｍ is provided, wherein the above-mentioned arithmetic processing device stores the above-mentioned SRAM or register file of the m-th arithmetic unit.

[0009] In the above aspect, the arithmetic processing device may have a network having a first daisy chain that connects adjacent arithmetic units among the M arithmetic units in series in the first direction, and a second daisy chain that connects adjacent arithmetic units among the M arithmetic units in series in the second direction, and the end of the first daisy chain and the end of the second daisy chain may be connected to each other to have a function of folding back propagating data.

[0010] In the above aspect, the processing device may have circuit elements that divide the network according to the number of channels or the number of sequences, and connect the first daisy chain and the second daisy chain, so as to be able to deal with problems with different numbers of channels or sequences.

[0011] In the above aspect, the arithmetic processing device, in the one aspect, calculates the column vector x belonging to the third matrix stored inside the m-th arithmetic unit.ｍ and the previously stored fourth matrix c = [c ０， c １， c ２，…， c Ｍ－１ ] and the matrix product y = [y ０， y １， y ２，…， y Ｍ－１ ] = cx, which is the part of the data of the fifth matrix generated by ｍ That is, When calculating the N-dimensional column vector c ｍ The elements of c ｍ０ , c ｍ１ , c ｍ２ , …, c ｍ（Ｎ－１） and the N-dimensional column vector y ｍ may be calculated and stored inside each m-th arithmetic unit.

[0012] In the above-mentioned aspects, the arithmetic processing device calculates a column vector d ｍ = [d ｍ０， d ｍ１， d ｍ２，…， d ｍ（Ｌ－１） ] Ｔ When performing linear layer calculations for M channels for ｍ Between Calculate the m-th N-dimensional column vector z ｍ may be stored inside each arithmetic unit, where u and v represent elements of the coefficient matrix U and bias vector V, respectively.

[0013] In the above-mentioned aspects, when performing multi-head attention, the arithmetic processing device has a set of the networks and the M arithmetic units for the number H of heads, divides the number of rows of the coefficient matrix of the linear layer immediately before the head division by the number H of heads, sequentially inputs the divided partition matrices to the communication path for the h-th head, and shares the vector data d ｍ Between Multi-heading may be achieved by calculating:

[0014] In the above aspect, when the attention is multi-headed, the arithmetic processing device may have the function of dividing the number of columns of the coefficient matrix of the linear layer immediately after the combination by the number of heads H in order to combine the multi-headed vectors, sequentially inputting each of the divided partition matrices to the communication path for the hth head, adding up the partial sums divided for each head, and storing them inside a specified arithmetic unit.

[0015] In the above aspects, the arithmetic processing device may be configured to sequentially and serially propagate the operation code together with the data to the network.

[0016] In the above aspects, the processing device may simulate multi-head processing by dividing the transfer of arrays and the accumulation of product-sum operations in order of head order in order to process multi-headed column vector data without adding hardware.

[0017] In the above aspect, the processing device may have circuit elements that divide the network according to the number of channels or the number of sequences, and connect the data read daisy chain and the data input daisy chain, so as to be able to deal with problems involving different numbers of channels or sequences.

[0018] According to the above aspect, the network can perform almost all processing by sending data only to adjacent processing elements, except for the process of adding partial sums when combining matrices. Therefore, since high parallelism without mutual dependency is realized, it is possible to speed up the calculation of the attention mechanism-based network model.

[0019] 1 is a diagram illustrating a schematic configuration of an arithmetic processing device according to an embodiment; m = [a ｍ ・b ０， a ｍ ・b １， a ｍ ・b ２，…， a ｍ ・b Ｍ－１ ] Ｔ In one embodiment, M M-dimensional column vectors x ｍ and a matrix c = [c0, c1, c2, … , c (M-1) 10 is an explanatory diagram of an operation for obtaining M N-dimensional column vectors by multiplying the matrices of

[0041] . FIG. 11 is an explanatory diagram of an operation for performing processing of a linear layer in one embodiment.

[0020] Next, an embodiment will be described with reference to the drawings. Elements common to the drawings are designated by the same reference numerals, and detailed descriptions of those elements will not be repeated.

[0021] FIG. 1 is a diagram illustrating a schematic configuration of a processing device according to an embodiment. Referring to FIG. 1, the processing device 10 includes a plurality of processing unit sets 16. Each of the plurality of processing unit sets 16 includes a plurality of processing units 11. The plurality of processing units 11 are connected to a network 14 that exchanges data via a first daisy chain 12 that propagates data to the right and a second daisy chain 13 that propagates data to the left. The first and second daisy chains 12, 13 are connected to each other via a first multiplexer 15, and data can also be input directly to the first daisy chain 12 from outside. The first and second daisy chains 12, 13 are connected to each other at their respective ends and have a function of looping back the propagating data. That is, data propagating through the first daisy chain 12 is looped back to the second daisy chain 13, and data propagates through the second daisy chain 13.

[0022] The arithmetic processing device 10 has H independent sets of arithmetic units 16 for performing partitioned matrix processing, i.e., multi-head processing, in H parallel fashion. After completing partitioned matrix processing, instead of combining the matrices, an addition network 18 is arranged orthogonal to the first and second daisy chains 12, 13 to add up the H divided partial sums when performing inner products of the divided rows and columns. If the cost of implementing the arithmetic unit is high for this purpose, processing may be performed using a separate central processing unit (CPU) or the like. Furthermore, the first and second daisy chains 12, 13 are divided according to the number of dimensions of the vector to be processed, and a second multiplexer 19 is provided to appropriately propagate data. Below, a method for performing dot products and linear layer calculations in this embodiment will be described.

[0023] 2 is a diagram showing the configuration of one arithmetic unit shown in FIG. 1. Referring to FIG. 2 together with FIG. 1, the arithmetic unit set 16 shown in FIG. 1 includes M arithmetic units 11. Each of the arithmetic units 11 has at least one SRAM 20 or register file, a multiplier 21, and an adder 22, a first port 23 for inputting data to the multiplier 21 from a first daisy chain 12 connected to an adjacent arithmetic unit 11 in a first direction, and a second port 24 for outputting data from the SRAM 20 or register file to a second daisy chain 13 connected to an adjacent arithmetic unit 11 in a second direction. The SRAM 20 or register file stores a column vector a i , b i and x i The address where the data is stored is specified to store the column vector a i , b i and x ito the multiplier 21, a second read port 26 that outputs data to the second port 24 of the arithmetic unit 11, and a write port 28 that writes the data output from the adder 22 back to the SRAM 20 or the register file via a register. The arithmetic unit 11 further has a data path 29 that inputs the data input from its first port 23 and the data stored in the SRAM 20 or the register file to the multiplier 21 and accumulates the multiplication result with the adder 22, and a data path 30 that inputs the accumulation result as array data to the SRAM 20 or the register file. The SRAM will be described here as a scratchpad SRAM 20.

[0024] FIG. 3 illustrates a dot product set x m = [a ｍ ・b ０， a ｍ ・b １， a ｍ ・b ２，…， a ｍ ・b Ｍ－１ ] Ｔ 3 is an explanatory diagram of the operation of calculating the dot product, and shows a calculation method for performing the most basic dot product. Figure 3 shows one arithmetic unit set 16, and the other arithmetic unit sets 16 shown in Figure 1 have a similar configuration.

[0025] 3 illustrates an example in which the first sequence length of the arithmetic unit set 16 can be switched between 2 and M by the second multiplexer 19. When the sequence length is M, referring to FIG. 3 together with FIGS. 1 and 2, a first matrix a=[a ０， a １， a ２，…， a Ｍ－１ ] and the second matrix b = [b ０， b １， b ２，…， b Ｍ－１ ] to matrix product a Ｔ M M-dimensional column vectors x calculated by b ｍ = [a ｍ ・b ０， a ｍ ・b １， aｍ ・b ２，…， a ｍ ・b Ｍ－１ ] Ｔ , m=0, 1, 2, ..., M-1. The arithmetic unit set 16 of the arithmetic processing device 10 has M arithmetic units 11, which are the smallest structural units. The ID of the arithmetic unit 11 is m=0, 1, 2, ..., M-1. One arithmetic unit 11 includes an internal scratchpad SRAM 20, and stores an N-dimensional array a corresponding to the ID number m. m and b m is stored inside the arithmetic unit 11. m and b m is a column vector.

[0026] The product of the matrix x derived from the first matrix a and the second matrix b is x = a Ｔ b, that is, M M-dimensional column vectors x m = [a ｍ ・b ０， a ｍ ・b １， a ｍ ・b ２，…， a ｍ ・b Ｍ－１ ] Ｔ When calculating m The elements of b0 = [b 00 , b 01 , b 02 , ..., b 0(N-1) ], b1 = [b 10 , b 11 , b 12 , ..., b 1(N-1) ], b2 = [b 20 , b 21 , b 22 , ..., b 2(N-1) ], ..., b M-1 = [b (M-1)0 , b (M-1)1 , b (M-1)2 , ..., b(M-1)(N-1)] are sent out successively, and transferred to the first daisy chain 12 by the first multiplexer 15. The data b mnBy creating the array data b in this order, the products can be accumulated continuously. The resulting N-dimensional column vector x m The data is stored again in the scratchpad SRAM 20. On the other hand, if the length of the first sequence is M=2, the second multiplexer 19 selects a route to shortcut the second daisy chain 13 for subsequent sequences with m=2 and onward, enabling the handling of sequences different from the sequence length 2 using m=0, 1. In the attention mechanism used in the Transformer, for example, the arithmetic processing device 10 of this embodiment can calculate the inner product of the query q and the key k between any sequences required to calculate attention as a third matrix x, where the query q is the first matrix a and the key k is the second matrix b, and perform partial calculations to calculate the dot-product attention. According to this embodiment, highly parallel calculations can be performed by exchanging data between adjacent arithmetic units 11 without using a more versatile broadcast-type network.

[0027] FIG. 4 shows M M-dimensional column vectors x ｍ and a matrix c = [c0, c1, c2, … , c (M-1) 4 is an explanatory diagram of the operation of obtaining M N-dimensional column vectors by multiplying the matrices x=[x ０， x １， x ２，…， x Ｍ－１ ] and M N-dimensional array matrices c = [c0, c1, c2, … , c (M-1) ]. That is, the following calculation method is shown: When executing the above, the arithmetic unit set 16 arranges the set of arrays c stored in each arithmetic unit 11 in an order c different from that shown in FIG. 00 , c 10, c 20 , ..., c (M-1)0 , c 01 , c 11 , c 21 , ...c (M-1)1 , c 02 , c 12 , c(M-1)(N-1) are read from each arithmetic unit 11 by the second daisy chain 13, and data b mn , and executes a multiply-and-accumulate operation. The resulting N-dimensional data y m is stored again in the scratchpad SRAM 20. This operation can be applied to the weighted output calculation y = a v of v by the dot product attention a in the Transformer, for example. Here, for simplicity of explanation, partitioned matrix processing, i.e., multi-head processing, is not assumed.

[0028] 5 is a diagram for explaining the operation of linear layer processing in one embodiment, and shows a method for performing linear layer calculations in the structure shown in FIG. 5. Referring to FIG. 5, L-dimensional vector data for M channels or M sequences, i.e., M L-dimensional arrays d ml From the above, M N-dimensional arrays z m When calculating the matrix element u, the arithmetic unit set 16 receives the matrix element u from the first daisy chain 12 that propagates data from the outside in a rightward direction via the first multiplexer 15. ｎｌ , v ｎ u 00 , u 01 , u 02 , ..., u 0(L-1) , u 10 , u 11 , u 12 , …u 1(L-1) , u 20 , u 21 ,..., u(N-1)(L-1), v0, v1, v2,..., v (N-1) The data u ｎｌ , v ｎ , and executes a multiply-and-accumulate operation. The resulting N-dimensional data z mis stored again in the scratchpad SRAM 20. In the Transformer, for example, this operation corresponds to generating vectors of query q, key k, and value v from arbitrary vector data using a linear layer. Here, for simplicity of explanation, partition matrix processing is not assumed.

[0029] 3 to 5, the case where the partitioned matrix processing is not performed has been described. When the partitioned matrix processing is introduced, as shown in FIG. 1, the calculation unit set 16 for the number H of heads is provided on the calculation processing device 10, the number of rows of the coefficient matrix of the linear layer immediately before the head division is divided and input to the communication path for each head in sequence, and the array z expressed by the following formula is obtained between the shared vector data to be operated on. hm Here, the vector to be operated on in the linear layer can be calculated by providing a common copy in the calculation units involved in the division matrix processing, or it can be realized by providing a communication path shared in the head division direction or by providing a memory area common to the units involved in the division matrix processing.

[0030] Furthermore, when combining multi-head matrices, the coefficient matrix of the linear layer that is operated on immediately after the combination is divided in the column direction and input sequentially to the communication path for each head, and the partial sums divided for the heads are added up by the addition network 18 shown in Figure 1 and stored inside a specified calculation unit, which can be implemented.

[0031] To execute an instruction, an operation code can be transmitted to the operation units 11 by sequential propagation, and thus can be transmitted along the first daisy chain 12 and / or the second daisy chain 13 of the network 14 .

[0032] In FIG. 1, the number of arithmetic units 11 is increased according to the number of heads H, but if the number of arithmetic resources is limited, it is also possible to simulate multi-head processing by increasing only the storage capacity of the scratchpad SRAM 20 and dividing the transfer of arrays and the accumulation of product-sum operations in order of the heads and performing them sequentially.

[0033] Furthermore, as illustrated in Figure 3 and its explanation, in order to deal with problems involving different numbers of channels or sequences, the network 14 is divided by the second multiplexer 19 according to the number of channels or sequences, and the first daisy chain 12 for data reading and the second daisy chain 13 for input are connected at their respective short sections, thereby making it possible to form multiple calculation processing paths corresponding to short sequences.

[0034] Although the preferred embodiments have been described in detail above, the present disclosure is not limited to such specific embodiments, and various modifications and variations are possible within the scope of the invention described in the claims.

[0035] REFERENCE SIGNS LIST 10 Arithmetic processing device 11 Arithmetic unit 12 First daisy chain 13 Second daisy chain 14 Network 15 First multiplexer 16 Arithmetic unit set 18 Addition network 19 Second multiplexer 20 Scratchpad SRAM

Claims

1. An arithmetic processing device comprising a plurality of arithmetic unit sets, each of the plurality of arithmetic unit sets including M arithmetic units (M is an integer of 2 or more) connected in a serial network, each of the M arithmetic units having at least one SRAM or register file, a multiplier and an adder, a port for selecting the SRAM or register file and outputting data to the multiplier, the SRAM or register file having an area for storing the data as an array, an input port for inputting data from the arithmetic unit adjacent in a second direction via the network, and an output port for outputting data to the network for transmission to the arithmetic unit adjacent in a first direction, a data path for inputting the data input from the input port and the data stored in the SRAM or register file to the multiplier and accumulating the multiplication result by the adder, and a data path for inputting the accumulation result as array data to the SRAM or register file, each of the plurality of arithmetic unit sets further comprising: ０， a １， a ２，…， a Ｍ－１ ] and the second matrix b=[b ０， b １， b ２，…， b Ｍ－１ ], one set of column vectors [a ｍ， b ｍ ] is stored in the SRAM or register file of the mth arithmetic unit, and an N-dimensional column vector b ｍ The array data is sent in the first direction through the network and is propagated to the adjacent arithmetic units in the subsequent stages, so that each column vector b ０， b １， b ２，…， b Ｍ－１ and the a stored in the arithmetic unit ｍ That is, the M-dimensional column vector x ｍ = [a ｍ ・b ０， a ｍ ・b １， a ｍ ・b ２，…， a ｍ ・b Ｍ－１ ] Ｔ is calculated by a data path that multiplies the multiplier and the adder of the m-th arithmetic unit, and the M-dimensional column vector x ｍ the SRAM or register file of the mth arithmetic unit.

2. The arithmetic processing device according to claim 1, wherein the network has a first daisy chain that connects adjacent ones of the M arithmetic units in series in the first direction, and a second daisy chain that connects adjacent ones of the M arithmetic units in series in the second direction, and the end of the first daisy chain and the end of the second daisy chain are coupled to each other to have the function of folding back propagating data.

3. The processing device according to claim 2, further comprising a circuit element that divides the network in accordance with the number of channels or the number of sequences to accommodate problems with different numbers of channels or sequences, and connects the first daisy chain and the second daisy chain.

4. A column vector x belonging to the third matrix stored in the SRAM or register file of the mth arithmetic unit ｍ and the previously stored fourth matrix c = [c ０， c １， c ２，…， c Ｍ－１ ] to matrix multiplication y = [y ０， y １， y ２，…， y Ｍ－１ ]=cx, which is a part of the data of the fifth matrix generated by the above-mentioned column vector y ｍ That is When calculating the N-dimensional column vector c ｍ The elements of c ｍ０ , c ｍ１ , c ｍ２ , …, c ｍ（Ｎ－１） and the N-dimensional column vector y ｍ and storing it in the SRAM or register file of each m-th arithmetic unit.

5. Column vector d that constitutes an M × L dimensional matrix ｍ = [d ｍ０， d ｍ１， d ｍ２，…， d ｍ（Ｌ－１） ] Ｔ When performing linear layer calculations for M channels for ｍ Between Calculate the m-th N-dimensional column vector z ｍ 5. The processor according to claim 1, wherein u and v represent elements of a coefficient matrix U and a bias vector V, respectively.

6. When performing multi-head attention, a set of the network and the M arithmetic units for the number H of heads is provided, the number of rows of the coefficient matrix of the linear layer immediately before the head division is divided by the number H of heads, and the divided partition matrices are input sequentially to the communication path for the h-th head, and the shared vector data to be operated on d ｍ Between 5. The processor according to claim 1, wherein the multi-head configuration is achieved by calculating:

7. The arithmetic processing device according to claim 6, wherein when performing multi-head conversion of the attention, the arithmetic processing device has the function of dividing the number of columns of the coefficient matrix of the linear layer immediately after conversion by the number of heads H in order to combine the multi-headed vectors, sequentially inputting each of the divided partition matrices to the communication path for the h-th head, adding up the partial sums divided for each head, and storing them inside a specified arithmetic unit.

8. The processor according to claim 1, wherein operation codes are propagated serially along with data to said network.

9. A processing device as claimed in claim 1 or 4, which simulates multi-head processing by dividing the transfer of arrays and the accumulation of product-sum operations in order of heads in order to process multi-headed column vector data without adding hardware.

Images