A method for accelerating forward propagation of LSTM based on systolic array

Abstract

The application relates to an LSTM forward propagation acceleration method based on a systolic array, and belongs to the technical field of neural network acceleration. In a traditional two-dimensional systolic array structure, each row of an input matrix X and each column of a weight matrix W are subjected to multiplication and summation operation, so that the data is highly multiplexed. However, a large number of large-scale matrix vector multiplication operations exist in a recurrent neural network, and the traditional systolic array structure with high data multiplexing rate cannot be used for operation and acceleration. Therefore, the application introduces a forward propagation acceleration method based on a systolic array, fully utilizes hardware resources, combines the systolic array structure, reduces the required bandwidth and memory access times of calculation, optimizes the calculation time of each PE in the systolic array, shortens the overall hardware working time, and maximizes the utilization of the systolic array to realize the acceleration effect of the LSTM.

Description

Technical Field

[0001] This invention belongs to the field of neural network acceleration technology, and relates to an LSTM forward propagation acceleration method based on a systolic array. Background Technology

[0002] Long Short-Term Memory (LSTM) networks are a special type of Recurrent Neural Network (RNN). An LSTM consists of three gates: an input gate, a forget gate, and an output gate, as well as a memory unit. These components work together to control the flow of information in a time series. The forget gate... t It determines which information is discarded from the cell state, taking into account the current input and the previous hidden state, and outputs a value between 0 and 1 via the sigmoid function; input gate i t This determines which new information will be stored in the cell state; memory cell C t By combining the output of the forget gate and the output of the input gate, a new cell state can be calculated. Some parts of the old state are forgotten, and new candidate values ​​are added; the output gate... t This determines how much information to read from the cell state for the output. This output will be used for the LSTM cell in the next time step and can be used for the network's predictions.

[0003] LSTM primarily controls the rate of information accumulation by introducing a gating mechanism, including selectively adding new information and selectively forgetting previously accumulated information. This improves the long-range dependency problem of recurrent neural networks and alleviates the gradient vanishing problem during long sequence training, and has been successfully applied in many fields such as speech recognition, image captioning, and natural language processing.

[0004] With the development of recurrent neural networks, not only are there more and more computational layers in recurrent neural networks, but also more and more weight parameters. Furthermore, bidirectional recurrent neural networks have emerged. This trend is not conducive to the hardware deployment of recurrent neural networks. Therefore, in recent years, various hardware structures that can accelerate the computation of recurrent neural networks have appeared. However, these structures cannot make full use of the computational characteristics of LSTM and cannot effectively accelerate the computation of LSTM, which also leads to a waste of hardware resources.

[0005] A systolic array (SRA) is a hardware architecture proposed by HTKung in 1982. It consists of numerous simple processing elements (PEs) arranged in a regular pattern. Its core concept is to allow data to flow within an array of processing elements, reducing the number of registers and memory accesses, resulting in a more regular structure, more uniform wiring, and increased frequency. In a traditional SRA structure, each row of the input matrix X and each column of the weight matrix W flows into the SRA array, the result is accumulated, and Y is output. This leads to two other SRA structures. These structures differ only in the type of data that is fixed and the data that flows. Essentially, one of the three variables W, X, and Y is fixed in a PE unit, while the other two are passed through the SRA array.

[0006] Since the most important computation in LSTM is matrix-vector multiplication, which is different from the data structure of traditional systolic arrays, it is impossible to use the traditional systolic array structure with high data reuse rate for computation acceleration. In order to take advantage of the characteristics of systolic arrays, such as high pipelining, low memory access frequency and low bandwidth utilization, it is necessary to modify the traditional systolic array. Summary of the Invention

[0007] In view of this, the purpose of this invention is to provide a method for accelerating LSTM forward propagation based on a systolic array that can make full use of hardware resources without occupying a large amount of hardware bandwidth.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] A method for accelerating LSTM forward propagation based on a pulsating array includes the following steps:

[0010] S1: Initialize the Long Short-Term Memory (LSTM) network and construct a systolic array structure, which includes m processing units (PEs).

[0011] S2: Store m-dimensional input data sequentially in m PEs, column by column;

[0012] S3: Using columns as units, the m data from each row of the weight matrix are sequentially passed to the corresponding positions in the k rows and n columns of the systolic array. The multiplication and accumulation calculation results in each PE are calculated and flowed to the right cycle by cycle. The results of the k rows of output are added together using the addition tree on the right side of the systolic array to obtain the output of m numbers.

[0013] S4: Repeat steps S2 and S3 to complete the matrix-vector multiplication operation in LSTM.

[0014] Furthermore, step S1, which involves initializing the Long Short-Term Memory (LSTM) network and constructing a systolic array, specifically includes:

[0015] Configure LSTM network parameters, including the number of nodes in the input layer, hidden layers, and output layer, and the stride size (step_size); initialize the input data x. t The dimension is m, and the weight matrix is ​​W. r×m ;

[0016] Construct a systolic array, dividing m processing units (PEs) into k rows and n columns. Each PE contains a multiplier, an adder, and register resources. Connect multiple adders to the right side of the k-row PE array to form an addition tree structure, thereby achieving step-by-step accumulation.

[0017] Furthermore, in step S2, the m input data are divided into n groups in order, with k data in each group, and each group of data is input into each column of the pulsation array in turn.

[0018] Further, in step S3, for the first row of the pulsating array, the calculation steps are as follows: The first n data differences of the first row in the weight matrix are entered into the array, and the multiplication and accumulation operations are performed sequentially. That is, the first multiply-accumulate calculates W(1,1)x1 in the first cycle, W(2,1)x1 in the second cycle, and W(r,1)x1 in the r-th cycle; the second multiply-accumulate does not calculate in the first cycle, but calculates W(1,1)x1+W(1,2)x2 in the second cycle, W(2,1)x1+W(2,2)x2 in the third cycle, and W(r,1)x1+W(r,2)x2 in the (r+1)-th cycle; the nth multiply-accumulate does not calculate in the first to (n-1)-th cycles, but calculates W(1,1)x1+W(1,2)x2+W(1,3)x3+…+W(1,n)x1 in the n-th cycle. n The (n+1)th period is calculated as W(2,1)x1+

[0019] W(2,2)x² + W(2,3)x³ + … + W(2,n)x n In the (r+n-1)th period, calculate W(r,1)x1 + W(r,2)x2 + W(r,3)x3 + ... +

[0020] W(r,n)x n For each row of the pulsating array, k results are obtained from the k PEs in the nth column of the nth period.

[0021] After [log2n] cycles, the addition operation is performed on the addition tree, and the result of the first row of the weight matrix W and the input data x is obtained.

[0022] After n+[log2n] cycles, the first row of the weight matrix W is output in each cycle, and the result of the product of the input data x is accumulated. The output is r numbers for r cycles until the matrix-vector multiplication is completed.

[0023] Furthermore, in step S4, after each column of PE completes the operation, the input data is immediately filled in and then the weight data is input, until all matrix-vector multiplication operations in the LSTM are completed.

[0024] Furthermore, two systolic arrays are constructed for the input gate, forget gate, output gate, and memory unit of the LSTM network, respectively.

[0025] The beneficial effects of this invention are as follows: This invention incorporates a systolic array into the LSTM network and changes the traditional connection method and control logic of the systolic array. PEs are arranged in an array configuration, and the computational data is accumulated sequentially from left to right. An addition tree is connected to the right side of the systolic array to achieve successive accumulation and output the computation result cycle by cycle. This invention fully utilizes hardware resources, reduces the bandwidth required for computation, and optimizes the computation time of each PE in the systolic array, thereby shortening the overall hardware operating time and maximizing the acceleration effect achieved by utilizing the systolic array.

[0026] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0027] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0028] Figure 1 This is a diagram of the pulsating array structure;

[0029] Figure 2 This is the data stream of one row in a pulsating array;

[0030] Figure 3 The data stream of one row in the pulsating array after one cycle;

[0031] Figure 4 The control logic diagram for a single PE. Detailed Implementation

[0032] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0033] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0034] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0035] This invention provides a method for accelerating LSTM forward propagation based on a systolic array that can fully utilize hardware resources without consuming a large amount of hardware bandwidth, comprising the following steps:

[0036] The calculation formula for LSTM is shown below. Most of the calculations involve matrix-vector multiplication, so optimizing matrix-vector multiplication is the most crucial step.

[0037] f t =σ(W f x t +U f h t-1 +b f )

[0038] i t =σ(W i x t +U i h t-1 +b i )

[0039] g t =tanh(W g x t +U g h t-1 +b g )

[0040] o t =σ(W o x t +U o h t-1 +b o )

[0041] C t =f t ⊙C t-1 +i t ⊙g t

[0042] h t =o t ⊙tanh(C t )

[0043] First, initialize the LSTM, configure the network parameters, including the number of nodes in the input layer, hidden layers, and output layer, and the stride (step_size); input data x t The dimension is m, and the weight matrix is ​​W. r×m The pulse array contains m processing elements (PEs), arranged in k rows and n columns, i.e., m = n × k. Based on this, a design is created... Figure 1 The pulsating array structure shown contains a multiplier, an adder, and register resources in each PE.

[0044] In this pulsating array, data is added sequentially and flows from left to right. First, m data points are stored in m PEs. Then, each row of m data points in the weight matrix is ​​passed to the corresponding position in the k rows and n columns of the pulsating array to complete the multiplication and accumulation operation. Finally, the results of the k rows of outputs are added together using an addition tree to obtain the accumulated output of m numbers.

[0045] Taking the first row of the pulsating array as an example, such as Figure 2 and Figure 3As shown: The first n data points in the first row of the weight matrix are entered into the array in sequence, and multiplication and accumulation operations are performed sequentially: the first multiply-accumulate calculates W(1,1)x1 in the first cycle, W(2,1)x1 in the second cycle, and W(r,1)x1 in the r-th cycle; the second multiply-accumulate does not calculate in the first cycle, but calculates W(1,1)x1+W(1,2)x2 in the second cycle, W(2,1)x1+W(2,2)x2 in the third cycle, and W(r,1)x1+W(r,2)x2 in the (r+1)-th cycle; the nth multiply-accumulate does not calculate in the first to (n-1)-th cycles, but calculates W(1,1)x1+W(1,2)x2+W(1,3)x3+…+W(1,n)x1 in the n-th cycle. n In the (n+1)th period, calculate W(2,1)x1 + W(2,2)x2 + W(2,3)x3 + ... + W(2,n)x n In the (r+n-1)th period, calculate W(r,1)x1 + W(r,2)x2 + W(r,3)x3 + ... + W(r,n)x n .

[0046] The operation is the same for each row in this systolic array, so k results are obtained from the k PEs in the nth column of the nth cycle. After [log2n] cycles, the addition operation is performed on the right-hand addition tree to obtain the cumulative result of the multiplication of the first row of the weight matrix W and the input data x. Each row of the weight matrix is ​​passed into the systolic array sequentially each cycle, so after n+[log2n] cycles, the cumulative result of the multiplication of the first row of the weight matrix W and the input data x is output in each cycle, outputting r numbers for r cycles, until the matrix-vector multiplication is completed.

[0047] If data is input for a second calculation after the pulse array calculation is completed, the utilization rate of a single PE can be obtained as follows:

[0048] V0 = r / (r+n)

[0049] The smaller r is, the higher the utilization rate V of PE. However, if the input data is filled immediately after each column of PE completes its calculation, followed by the input weight data, then PE can be fully utilized. In this case, the utilization rate of a single PE is:

[0050] V1=r×step_size / (r×step_size+n)

[0051] This greatly improves the utilization rate of PE.

[0052] Taking the first column PE as an example:

[0053] Originally, the first column of PE enters an idle state after r cycles of operation, waiting for the other PEs in the pulsating array to complete their operations, and then new input data is input before the weight matrix data is passed in for calculation, and this cycle continues.

[0054] The state control logic works as follows: After the first column of PEs completes its operation for r cycles, it does not need to wait for other PEs in the pulsating array to finish their operations. Instead, it directly stores the new input data and then passes it into the weight matrix data for calculation. Figure 4 As shown.

[0055] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by instructing related hardware through a program. The timing logic can be controlled by programming the pulsating array structure and the corresponding controller to complete the control of the operation process.

[0056] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for accelerating the forward propagation of LSTM based on a pulsating array, characterized in that: Includes the following steps: S1: Initialize the Long Short-Term Memory (LSTM) network and construct a systolic array structure, which includes m processing units (PEs). S2: Store m input data sequentially in m PEs (rows and n columns) in k rows and n columns; S3: In column-by-column order, the m data from each row of the weight matrix are sequentially passed to the corresponding positions in the k rows and n columns of the pulsating array. The multiplication and accumulation results in each PE are calculated and flowed to the right cycle by cycle. The results of the k rows of output are added together using the addition tree on the right side of the pulsating array to obtain the output of m numbers. S4: Repeat steps S2 and S3 to complete the matrix-vector multiplication operation in LSTM; Step S1, which involves initializing the Long Short-Term Memory (LSTM) network and constructing a systolic array, specifically includes: Configure LSTM network parameters, including the number of nodes in the input layer, hidden layers, and output layer, and the stride size (step_size); initialize the input data x. t The dimension is m, and the weight matrix is ​​W. r×m ; Construct a systolic array, dividing m processing units (PEs) into k rows and n columns. Each PE contains a multiplier, an adder, and register resources. Connect multiple adders to the right side of the k-row PE array to form an addition tree structure, thereby achieving step-by-step accumulation. In step S2, the m input data are divided into n groups in order, with k data in each group, and each group of data is input into each column of the pulsation array in turn. In step S3, for the first row of the pulsating array, the calculation steps are as follows: The first n data differences of the first row in the weight matrix are input into the array, and the multiplication and accumulation operations are performed sequentially. That is, the first multiply-accumulate calculates W(1,1)x1 in the first cycle, W(2,1)x1 in the second cycle, and W(r,1)x1 in the r-th cycle; the second multiply-accumulate does not calculate in the first cycle, but calculates W(1,1)x1+W(1,2)x2 in the second cycle, W(2,1)x1+W(2,2)x2 in the third cycle, and W(r,1)x1+W(r,2)x2 in the (r+1)-th cycle; the nth multiply-accumulate does not calculate in the first to (n-1)-th cycles, but calculates W(1,1)x1+W(1,2)x2+W(1,3)x3+…+W(1,n)x1 in the n-th cycle. n In the (n+1)th period, calculate W(2,1)x1 + W(2,2)x2 + W(2,3)x3 + ... + W(2,n)x n In the (r+n-1)th period, calculate W(r,1)x1 + W(r,2)x2 + W(r,3)x3 + ... + W(r,n)x n For each row of the pulsating array, k results are obtained from the k PEs in the nth column of the nth period. After [log2n] cycles, the addition operation is performed on the addition tree, and the result of the first row of the weight matrix W and the input data x is obtained. After n+[log2n] cycles, the first row of the weight matrix W is output in each cycle, and the result of the product of the input data x is accumulated. The r numbers are output for r cycles until the matrix-vector multiplication is completed. In step S4, after each column of PE is calculated, the input data is immediately filled in and then the weight data is entered for the next calculation.

2. The LSTM forward propagation acceleration method based on a pulsating array according to claim 1, characterized in that: Two systolic arrays are constructed for the input gate, forget gate, output gate, and memory unit of the LSTM network, respectively.

Images