Arithmetic processing device

Abstract

To calculate a network model at high speed based on an attention mechanism.SOLUTION: Provided is an arithmetic processing device comprising a plurality of arithmetic unit sets 16, each set including M (M is an integer equal to or greater than 2) arithmetic units 11 connected by a series network 14. Each unit stores one set [am,bm] of column vectors in a first matrix a and a second matrix b in an SRAM of an m-th arithmetic unit. Array data of an N-dimensional column vector bm is sent in a first direction from each arithmetic unit via the network, and sequentially propagated to an adjacent arithmetic unit in a next stage. Thereby, a dot product of each column vector b0 to bM-1 of the second matrix and am stored in the arithmetic unit, that is, an M-dimensional column vector xm is calculated by a data path integrated by a multiplier and an adder of an m-th arithmetic unit, and the M-dimensional column vector xm is stored in the SRAM of the m-th arithmetic unit.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a computing system, and more particularly to a computing device that efficiently performs machine learning processing. [Background technology]

[0002] The speed of machine learning has improved dramatically thanks to general purpose computing on Graphical Processing Units (GPUs).

[0003] On the other hand, many useful applications of machine learning network models, including those that use not only simple convolution operations but also vector inner product operations known as attention mechanisms, have become known, such as automatic translation and image processing (see, for example, Patent Document 1, Non-Patent Documents 1 and 2). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2022-019422 [Non-patent literature]

[0005] [Non-Patent Document 1] A. Vaswani et al., “Attention is all you need”, 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA, Dec. 4-9, 2017. https: / / proceedings.neurips.cc / paper / 2017 / file / 3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf [Non-Patent Document 2] A. Dudovskiy et al., “An image is worth 16x16 words: transformers for image recognition at scale”, The 9th International Conference on Learning Representations (ICLR 2021), Virtual Only, May 3-7, 2021. https: / / iclr.cc / virtual / 2021 / poster / 3013 Summary of the Invention [Problem to be solved by the invention]

[0006] Image processing calculation devices are generally designed to be highly versatile so that they can perform many types of matrix calculations, and specific processing is performed by software libraries running on the device. In this case, it is known that the versatility results in extra power consumption, and the inclusion of software processing reduces speed. On the other hand, creating a completely dedicated circuit has the disadvantage that it cannot perform other calculations.

[0007] An object of the present disclosure is to provide a processing device that can speed up calculations of a network model based on an attention mechanism (hereinafter also referred to as "attention"). [Means for solving the problem]

[0008] According to one aspect of the present disclosure, there is provided an arithmetic processing device having a plurality of arithmetic unit sets, each of the plurality of arithmetic unit sets including M arithmetic units (M is an integer equal to or greater than 2) connected in a serial network, each of the M arithmetic units having at least one SRAM or register file, a multiplier and an adder, a port for selecting the SRAM or register file and outputting data to the multiplier, the SRAM or register file having an area for storing the data as an array, an input port for inputting data from the arithmetic unit adjacent in a second direction via the network, and an output port for outputting data to the network for transmission to the arithmetic unit adjacent in a first direction, a data path for inputting the data input from the input port and the data stored in the SRAM or register file to the multiplier and accumulating the multiplication result by the adder, and a data path for inputting the accumulation result as array data to the SRAM or register file, Each of the plurality of sets of arithmetic units is a first matrix a=[a 0, a 1, a 2,…, a M-1 ] and the second matrix b=[b 0, b 1, b 2,…, b M-1 ], one set of column vectors [a m, b m ] is stored in the SRAM or register file of the mth arithmetic unit, and an N-dimensional column vector b m The array data is sent in the first direction through the network, and is propagated to the adjacent arithmetic units in the subsequent stages, so that each column vector b 0, b 1, b 2,…, b M-1 and the a stored in the arithmetic unit m The dot product of x and x is an M-dimensional column vector that is part of an array of a third matrix of dimension M × M. m =[a m b 0, a m b1, a m b 2,…, a m b M-1 ] T is calculated by a data path that multiplies the multiplier and adder of the m-th arithmetic unit, and the M-dimensional column vector x m The above-mentioned arithmetic processing device is provided, which stores the above in the SRAM or register file of the m-th arithmetic unit.

[0009] In the above aspect, the arithmetic processing device may have a network including a first daisy chain connecting adjacent arithmetic units among the M arithmetic units in series in the first direction, and a second daisy chain connecting adjacent arithmetic units among the M arithmetic units in series in the second direction, and the end of the first daisy chain and the end of the second daisy chain may be coupled to each other to have a function of folding back propagating data.

[0010] In the above aspect, the processing device may have circuit elements that divide the network according to the number of channels or the number of sequences and connect the first daisy chain and the second daisy chain to deal with problems involving different numbers of channels or sequences.

[0011] In the above aspect, the arithmetic processing device, in the one aspect, calculates a column vector x belonging to the third matrix stored inside the m-th arithmetic unit. m and the previously stored fourth matrix c=[c 0, c 1, c 2,…, c M-1 ] and matrix multiplication y=[y 0, y 1, y 2,…, y M-1 ]=cx, the column vector y m That is

number

[0012] In the above-mentioned aspect, the arithmetic processing device is configured to calculate a column vector d m =[d m0, d m1, d m2,…, d m(L-1) ] T When performing linear layer calculations for M channels on m Between

number

[0013] In the above-mentioned embodiment, when performing multi-head attention, the arithmetic processing device has a set of the above-mentioned networks and the above-mentioned M arithmetic units for the number H of heads, divides the number of rows of the coefficient matrix of the linear layer immediately before the head division by the number H of heads, sequentially inputs the divided partition matrices to the communication path for the h-th head, and outputs the shared vector data d m Between

number

[0014] In the above aspect, the arithmetic processing device may have a function of dividing the number of columns of the coefficient matrix of the linear layer immediately after combination by the number of heads H in order to combine the multi-headed vectors when performing multi-heading of the attention, sequentially inputting each of the divided partition matrices to the communication path for the h-th head, adding up the partial sums divided for each head, and storing them inside a specified arithmetic unit.

[0015] In the above aspect, the arithmetic processing device may be configured to sequentially and serially propagate the arithmetic code together with the data to the network.

[0016] In the above-mentioned aspects, the processing device may be adapted to simulate multi-head processing by sequentially transferring arrays and accumulating multiply-and-accumulate operations in head order in order to process multi-headed column vector data without adding additional hardware.

[0017] In the above aspect, the arithmetic processing device may have circuit elements that divide the network according to the number of channels or the number of sequences, and connect the data read daisy chain and the data input daisy chain, so as to be able to deal with problems involving different numbers of channels or sequences. Effect of the Invention

[0018] According to the above aspect, the network can perform almost all processing by sending data only to adjacent processing elements, except for the process of adding partial sums when combining matrices. Therefore, since a high degree of parallelism that is independent of each other is realized, it is possible to speed up the calculation of a network model based on an attention mechanism. [Brief description of the drawings]

[0019] [Figure 1] FIG. 1 is a diagram illustrating a schematic configuration of a processing device according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating the configuration of one arithmetic unit illustrated in FIG. [Diagram 3] FIG. 11 is an explanatory diagram of an operation for calculating a dot product xm=[am·b0, am·b1, am·b2, . . . , am·bM-1]T in one embodiment. [Figure 4] FIG. 11 is an explanatory diagram of an operation in one embodiment to obtain M N-dimensional column vectors by multiplying an M×M-dimensional matrix consisting of M M-dimensional column vectors xm by a matrix c=[c0, c1, c2, ..., c(M-1)] consisting of M N-dimensional column vectors. [Diagram 5] FIG. 11 is an explanatory diagram of an operation for processing a linear layer in one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] Next, an embodiment will be described with reference to the drawings. Note that elements common to the drawings are given the same reference numerals, and detailed descriptions of those elements will not be repeated.

[0021] FIG. 1 is a diagram showing a schematic configuration of a processing device according to an embodiment. Referring to FIG. 1, a processing device 10 has a plurality of processing unit sets 16. Each of the plurality of processing unit sets 16 has a plurality of processing units 11. The plurality of processing units 11 are connected to a network 14 that exchanges data by a first daisy chain 12 that propagates data in a rightward direction and a second daisy chain 13 that propagates data in a leftward direction. The first and second daisy chains 12 and 13 are connected to each other by a first multiplexer 15, and data can also be input directly to the first daisy chain 12 from the outside. The first and second daisy chains 12 and 13 are connected to each other at their respective ends, and have a function of folding back the propagating data. That is, data propagating through the first daisy chain 12 is folded back to the second daisy chain 13, and data propagates through the second daisy chain 13.

[0022] The arithmetic processing device 10 has H independent arithmetic unit sets 16 for performing partitioned matrix processing, i.e., multi-head processing, in H parallel. An addition network 18 for adding up the H divided partial sums when performing inner products of divided rows and columns instead of combining matrices after completing partitioned matrix processing is arranged in a manner orthogonal to the first and second daisy chains 12, 13. If the cost of implementing the arithmetic unit is high for this purpose, the processing may be performed by a separate CPU (Central Processing Unit) or the like. Furthermore, the first and second daisy chains 12, 13 are divided according to the number of dimensions of the vector to be processed, and a second multiplexer 19 is provided to appropriately propagate the data. Below, a method for performing dot products and linear layer calculations in this embodiment will be described.

[0023] 2 is a diagram showing the configuration of one arithmetic unit shown in FIG. 1. Referring to FIG. 2 together with FIG. 1, the arithmetic unit set 16 shown in FIG. 1 includes M arithmetic units 11. Each of the arithmetic units 11 has at least one SRAM 20 or a register file, a multiplier 21, and an adder 22, a first port 23 for inputting data to the multiplier 21 from a first daisy chain 12 connected to an adjacent arithmetic unit 11 in a first direction, and a second port 24 for outputting data from the SRAM 20 or the register file to a second daisy chain 13 connected to an adjacent arithmetic unit 11 in a second direction. The SRAM 20 or the register file stores a column vector a i , b i and x i Each of these is stored as an array, and the address where the data is stored is specified to obtain the column vector a i , b i and x ito the multiplier 21, a second read port 26 that outputs data to the second port 24 of the arithmetic unit 11, and a write port 28 that writes the data output from the adder 22 back to the SRAM 20 or the register file via a register. The arithmetic unit 11 further has a data path 29 that inputs the data input from its first port 23 and the data stored in the SRAM 20 or the register file to the multiplier 21 and accumulates the multiplication result by the adder 22, and a data path 30 that inputs the accumulation result as array data to the SRAM 20 or the register file. The SRAM will be described here as a scratch pad SRAM 20.

[0024] FIG. 3 illustrates a dot product set x m =[a m b 0, a m b 1, a m b 2,…, a m b M-1 ] T 3 is an explanatory diagram of the operation of calculating the dot product, and shows a calculation method for performing the most basic dot product. Fig. 3 shows one arithmetic unit set 16, and the other arithmetic unit sets 16 shown in Fig. 1 have the same configuration.

[0025] FIG. 3 illustrates an example in which the first sequence length of the arithmetic unit set 16 can be switched between 2 and M by the second multiplexer 19. When the sequence length is M, referring to FIG. 3 together with FIG. 1 and FIG. 2, a first matrix a=[a 0, a 1, a 2,…, a M-1 ] and the second matrix b=[b 0, b 1, b 2,…, b M-1 ] to matrix product a T M M-dimensional column vectors x calculated by b m =[a m b 0, am b 1, a m b 2,…, a m b M-1 ] T , m=0, 1, 2, . . . , M-1. The arithmetic unit set 16 of the arithmetic processing device 10 has M arithmetic units 11, which are the smallest constituent units. The IDs of the arithmetic units 11 are m=0, 1, 2, . . . , M-1. Each arithmetic unit 11 includes an internal scratch pad SRAM 20, and stores an N-dimensional array a corresponding to the ID number m. m and b m is stored inside the arithmetic unit 11. m and b m is a column vector.

[0026] The product of the first matrix a and the second matrix b is x=a T b, i.e., M M-dimensional column vectors x m =[a m b 0, a m b 1, a m b 2,…, a m b M-1 ] T In calculating m The element of b0=[b 00 ,b 01 ,b 02 ,…,b 0(N-1) ], b1=[b 10 ,b 11 ,b 12 ,…,b 1(N-1) ], b2=[b 20 ,b 21 ,b 22 ,…,b 2(N-1) ], …, b M-1 =[b (M-1)0 ,b (M-1)1 ,b (M-1)2 ,…,b (M-1)(N-1)], and the first multiplexer 15 transfers them to the first daisy chain 12, and the data b mn By creating the array data b in this order, the products can be accumulated continuously. The resulting N-dimensional column vector x m The data is stored in the scratch pad SRAM 20 again. On the other hand, when the length of the first sequence is M=2, the second multiplexer 19 selects a route so that subsequent m=2 and onwards are shortcutted to the second daisy chain 13, so that a different sequence different from the sequence of sequence length 2 using m=0,1 can be handled. In the attention mechanism used in Transformer, for example, the arithmetic processing device 10 of this embodiment can calculate the combination of the inner product of the query q and the key k between any sequences required to obtain attention as the third matrix x when the query q is the first matrix a and the key k is the second matrix b, and can perform partial calculation to obtain the dot product attention. According to this embodiment, it is possible to perform highly parallel calculations by data exchange between adjacent arithmetic units 11 without using a more versatile broadcast type network.

[0027] FIG. 4 illustrates, in one embodiment, M M-dimensional column vectors x m and a matrix c=[c0,c1,c2, … ,c (M-1) 4 is an explanatory diagram of an operation of obtaining M N-dimensional column vectors by multiplying matrices x=[x 0, x 1, x 2,…, x M-1 ] and M N-dimensional array matrices c=[c0,c1,c2, … ,c (M-1)] matrix multiplication to generate M N-dimensional column vectors. That is,

number

[0028] 5 is an explanatory diagram of the operation of processing the linear layer in one embodiment, and shows a method of performing linear layer calculation in the structure shown in FIG. 1. Referring to FIG. 5, L-dimensional vector data for M channels or M sequences, that is, M L-dimensional arrays d ml From the following formula, we create an N-dimensional array z m

number

[0029] 3 to 5, the case where the partitioned matrix processing is not performed has been described. When the partitioned matrix processing is introduced, as shown in FIG. 1, the calculation unit set 16 for the number of heads H is provided on the calculation processing device 10, the number of rows of the coefficient matrix of the linear layer immediately before the head division is divided and input to the communication path for each head in sequence, and the array z hm

number

[0030] In addition, when combining multi-head matrices, the coefficient matrix of the linear layer that performs the calculation immediately after the combination is divided in the column direction and inputted sequentially to the communication path for each head, and the partial sums divided for the heads are added up by the adding network 18 shown in Figure 1 and stored inside a specified calculation unit, which can be implemented.

[0031] To execute an instruction, an operation code may be transmitted to the arithmetic units 11 by sequential propagation, and thus may be transmitted along the first daisy chain 12 and / or the second daisy chain 13 of the network 14 .

[0032] In FIG. 1, the number of calculation units 11 is increased according to the number of heads H, but if the number of calculation resources is limited, it is also possible to simulate multi-head processing by increasing only the memory capacity of the scratchpad SRAM 20 and dividing the array transfer and the accumulation of product-sum operations in order of the heads and performing them sequentially.

[0033] In addition, as illustrated in Figure 3 and its explanation, in order to deal with problems involving different numbers of channels or sequences, the network 14 is divided by the second multiplexer 19 according to the number of channels or sequences, and the first daisy chain 12 for data reading and the second daisy chain 13 for input are connected at their respective short portions, making it possible to form multiple calculation processing paths corresponding to short sequences.

[0034] Although the preferred embodiments have been described in detail above, the present disclosure is not limited to such specific embodiments, and various modifications and variations are possible within the scope of the invention described in the claims. [Explanation of symbols]

[0035] 10 Processing unit 11. Calculation Unit 12 1st Daisy Chain 13 2nd Daisy Chain 14 Network 15 First Multiplexer 16 Calculation Unit Set 18 Addition network 19 Second Multiplexer 20 Scratchpad SRAM

Claims

1. A processor including a plurality of sets of arithmetic units, Each of the plurality of arithmetic unit sets includes M arithmetic units (M is an integer equal to or greater than 2) connected in a serial network; Each of the M arithmetic units has at least one SRAM or register file, a multiplier, and an adder, a port for selecting the SRAM or register file and outputting data to the multiplier, the SRAM or register file having an area for storing the data as an array, an input port for inputting data from the arithmetic unit adjacent in the second direction via the network, and an output port for outputting data to the network for transmission to the arithmetic unit adjacent in the first direction, a data path for inputting the data input from the input port and the data stored in the SRAM or register file to the multiplier and accumulating the multiplication result by the adder, and a data path for inputting the accumulation result as array data to the SRAM or register file, Each of the plurality of sets of arithmetic units has a first matrix a=[a ０， a １， a ２，…， a Ｍ－１ ] and the second matrix b=[b ０， b １， b ２，…， b Ｍ－１ ], one set of column vectors [a ｍ， b ｍ ] is stored in the SRAM or register file of the mth arithmetic unit, and an N-dimensional column vector b ｍ The array data is sent in the first direction through the network and is propagated to the adjacent arithmetic units in the subsequent stages, so that each column vector b ０， b １， b ２，…， b Ｍ－１ and the a stored in the arithmetic unit ｍ That is, the M-dimensional column vector x ｍ = [a ｍ ・b ０， a ｍ ・b １， a ｍ ・b ２，…， a ｍ ・b Ｍ－１ ] Ｔ is calculated by a data path that multiplies the multiplier and the adder of the m-th arithmetic unit, and the M-dimensional column vector x ｍ the SRAM or register file of the mth arithmetic unit.

2. 2. The arithmetic processing device according to claim 1, wherein the network has a first daisy chain that connects adjacent arithmetic units among the M arithmetic units in series in the first direction, and a second daisy chain that connects adjacent arithmetic units among the M arithmetic units in series in the second direction, and has a function of folding back data transmitted by coupling an end of the first daisy chain to an end of the second daisy chain.

3. 3. The processing device according to claim 2, further comprising a circuit element that divides the network in accordance with the number of channels or the number of sequences and connects the first daisy chain and the second daisy chain so as to be able to deal with problems involving different numbers of channels or sequences.

4. The column vector x belonging to the third matrix stored in the SRAM or register file of the m-th arithmetic unit ｍ and the previously stored fourth matrix c = [c ０， c １， c ２，…， c Ｍ－１ ] to matrix multiplication y = [y ０， y １， y ２，…， y Ｍ－１ ]=cx, which is a part of the data of the fifth matrix generated by the above-mentioned column vector y ｍ That is [0070] When calculating The N-dimensional column vector c stored in each arithmetic unit ｍ The elements of c ｍ０ , c ｍ１ , c ｍ２ , …, c ｍ（Ｎ－１） and the N-dimensional column vector y ｍ and storing it in the SRAM or register file of each m-th arithmetic unit.

5. A column vector d constituting an M×L dimensional matrix ｍ = [d ｍ０， d ｍ１， d ｍ２，…， d ｍ（Ｌ－１） ] Ｔ When performing linear layer calculations for M channels for ｍ Between [0080] Calculate the m-th N-dimensional column vector z ｍ 5. The processor according to claim 1, wherein u and v represent elements of a coefficient matrix U and a bias vector V, respectively.

6. When performing multi-head attention, a set of the network and the M number of arithmetic units is provided for the number H of heads, the number of rows of the coefficient matrix of the linear layer immediately before the head division is divided by the number H of heads, and the divided partition matrices are sequentially input to the communication path for the h-th head, and the shared vector data d ｍ Between [0097] 5. The processor according to claim 1, wherein the multi-head configuration is achieved by calculating:

7. The arithmetic processing device of claim 6 has the function of dividing the number of columns of the coefficient matrix of the linear layer immediately after combination by the number of heads H in order to combine the multi-headed vectors when performing multi-heading of the attention, sequentially inputting each of the divided partition matrices to the communication path for the h-th head, adding up the partial sums divided for each head, and storing them inside a specified arithmetic unit when performing multi-heading of the attention.

8. 2. The processor according to claim 1, wherein the operation codes are propagated serially along with the data to the network.

9. 5. The processor according to claim 1, further comprising: a multi-head processor for simulating multi-head processing by sequentially dividing the data transfer of arrays and accumulating multiplication and accumulation operations in order of the heads in order to process multi-headed column vector data without adding hardware.

Images