Parameterized lstm acceleration system with parallel data access and design method thereof

Abstract

The application discloses a parameterized LSTM acceleration system with parallel data access and a design method thereof, and belongs to the technical field of deep neural network acceleration. Four parameterized row parallel operation modules are designed according to gate unit characteristics, two parallel pipelines are arranged in each module to process input vectors and hidden vectors respectively, and the operation results are sent to a parallel accumulation unit to be added with a bias vector; each pipeline comprises a weight matrix row parallel storage unit and a row parallel multiplication and accumulation module, the former splices one row of data from low to high for storage, and the latter reads one row of data, sends the data to each PE after hierarchical splitting for grouped parallel processing; the operation results of each module are sent to a nonlinear activation function operation module and a same element point multiplication operation module for processing. According to the parameterized parallel data access design of each module and a data path of the acceleration system in a normalization step, the application improves system flexibility and universality, reduces design difficulty, and improves operation efficiency.

Description

Technical Field

[0001] This invention belongs to the technical field of deep neural network acceleration, and relates to the design of FPGA-based LSTM acceleration systems, particularly a parameterized LSTM acceleration system with parallel data access and its design method. Background Technology

[0002] LSTM networks possess historical data memory and effectively address the problems of gradient explosion and vanishing gradients, making them particularly suitable for processing sequential signals. As a computationally intensive algorithm, it is typically accelerated using GPUs, ASICs, and FPGAs; considering the requirements of low power consumption and high performance, FPGAs are currently the most common implementation method. However, existing FPGA-based LSTM acceleration systems are mostly implemented on specific hardware platforms with fixed network models, lacking flexibility and versatility depending on the network structure and platform. Furthermore, the control logic used to implement on-chip parallel parameter access, computation data transmission, and distribution is complex, making system design challenging. Therefore, a parameterized LSTM network acceleration system and its design method are needed, featuring parallel data access, simple control logic for transmission and distribution, and flexible configuration through parameter settings. Summary of the Invention

[0003] The purpose of this invention is to address the aforementioned problems and shortcomings by proposing a parameterized LSTM acceleration system with parallel data access and its design method. This invention parameterizes each component module of the LSTM acceleration system, enabling flexible configuration for different network structures and hardware platforms. Simultaneously, by designing parallel data access storage units and standardized data transmission paths, it simplifies on-chip parameter access methods and the control logic for data transmission and allocation.

[0004] To achieve the above objectives, the technical solution adopted by the present invention is: a parameterized LSTM acceleration system with parallel data access, characterized in that it includes a parameterized input vector parallel access cache module Ppa_x. t Parallel access cache module for parameterized hidden vectors, Ppa_h t-1 The system consists of: a parameterized input gate parallel operation module (module1), a parameterized forget gate parallel operation module (module2), a parameterized update gate parallel operation module (module3), a parameterized output gate parallel operation module (module4), a nonlinear activation function operation module (NLAM), an element-wise multiplication operation module (EDAM), a parameterized intermediate state buffer module (P_Temp), and a parameterized output result buffer module (P_Res).

[0005] Modules 1 through 4 each include a row-parallel storage unit for the input weight matrix Lpm_Wx, a row-parallel storage unit for the hidden weight matrix Lpm_Wh, a bias vector storage unit P_B, a row-parallel group multiplication and accumulation module PU_MVx, a row-parallel group multiplication and accumulation module PU_MVh, and a parameterized parallel accumulation unit P_Adder; NLAM includes three sigmoid activation function units SI, SF, and SO, and one tanh activation function unit TG;

[0006] The Ppa_x t The output is transmitted to PU_MVx of module 1 to module 4, and Ppa_h t-1 The output of Lpm_Wx of module1 to module4 is transmitted to PU_MVh of this module, the output of Lpm_Wh is transmitted to PU_MVh of this module, and the outputs of PU_MVx, PU_MVh and P_B are transmitted to P_Adder of this module.

[0007] The output i of P_Adder of module1 to module4 t f t g t o t The signals are transmitted to SI, SF, TG, and SO of the NLAM, respectively; the output s_i of the NLAM t ,sf t t_g t ,s_o t and the output c of P_Temp t-1 Transmitted to the EDAM module; EDAM output c t and h t The data is transmitted to P_Temp and P_Res respectively.

[0008] Furthermore, the Ppa_x t and Ppa_h t-1 All of these can be parameterized, with a storage depth of 1 and a storage width of l. x *n bit and l h *n bits; where l x and l h are the dimensions of the input vector and the hidden vector, respectively, and n is the fixed-point data width.

[0009] Furthermore, P_Temp and P_Res, as well as Lpm_Wx, Lpm_Wh, and P_B of modules 1 to 4, can all be parameterized, and their storage depth is l.h The storage widths of Lpm_Wx and Lpm_Wh are l x *n bit and l h *n bits, the storage width of both P_Temp and P_Res modules is n bits; where l x and l h are the dimensions of the input vector and the hidden vector, respectively, and n is the fixed-point data width.

[0010] Furthermore, the processing units PE that perform row-parallel multiplication and accumulation in PU_MVx and PU_MVh of modules 1 to 4 are implemented in groups, and the number of groups and the number of PEs in each group can be parameterized. The number of groups are as follows: and The number of PEs in each group is g x =l x / m x and g h =l h / m h ; where l x and l h These are the dimensions of the input vector and the hidden vector, respectively.

[0011] Furthermore, SI, SF, SO, and TG in the NLAM can all be parameterized, with a data dimension of fixed-point data width n bits; and SI, SF, and SO perform the following operations.

[0012] sigmoid:

[0013] TG implements the following operations

[0014] tanh:

[0015] Furthermore, the EDAM can be parameterized, with a data dimension of fixed-point data width n_bit, and the following operations are performed.

[0016] c t =s_i t ⊙t_g t +s_f t ⊙c t-1

[0017] h t =s_o t ⊙tanh(c t )

[0018] Where s_i t s_f t t_g t ,s_ot These are the outputs of SI, SF, TG, and SO in the NLAM, respectively, c t-1 For the output of P_Temp, c t h represents the intermediate state value at the current moment. t Output the result for the current moment.

[0019] Furthermore, this invention also proposes a parameterized LSTM acceleration system with parallel data access and its design method, characterized by comprising the following steps:

[0020] (1) Based on the LSTM network parameters specified by the user, including the fixed-point data bit width n and the dimension l of the input vector. x And the dimension l of the hidden vector h The design dimensions of each module in the parameterized LSTM acceleration system are determined, and each module is instantiated and implemented.

[0021] (2) According to the storage feature requirements, the input weight matrix, hidden weight matrix and bias vector parameters of the input gate, forget gate, update gate and output gate of the LSTM network are stored in Lpm_Wx, Lpm_Wh and P_B of module1 to module4 respectively;

[0022] (3) Based on the storage feature requirements, the input vector at time t, the hidden vector at time t-1, and the intermediate state vector are respectively transmitted to Ppa_x. t Ppa_h t-1 In P_Temp; the Ppa_x t Output x t and Ppa_h t The output h t-1 The data is split into n bits of fixed-point data and transmitted to PU_MVx and PU_MVh of module1 to module4 respectively, and the address Addr is set to 0.

[0023] (4) Read a row of data in parallel from the corresponding addresses of Lpm_Wx and Lpm_Wh of module1 to module4 respectively, and split it into n-bit fixed-point data, and transmit it to PU_MVx and PU_MVh of the corresponding module respectively, and perform the grouping, multiplication and accumulation operation of a row of data in the four gates in parallel; and send the corresponding operation result and a data taken from the corresponding address of P_B ​​to P_Adder for accumulation operation;

[0024] (5) The output i of P_Adder in module1 to module4 t f t g t and ot The data is sent to the NLAM and EDAM respectively, and the relevant operations are performed in parallel; the output ct of the EDAM is used to update the address corresponding to P_Temp, and the output h of the EDAM is also sent to the NLAM. t Stored at the address corresponding to P_Res;

[0025] (6) Increment the address Addr of Lpm_Wx, Lpm_Wh, P_B of module1 to module4, and the address Addr of P_Temp and P_Res by 1, and repeat steps (4) and (5) until l h All rows of data have been processed.

[0026] (7) Based on the storage characteristic requirements, use the output h of P_Res. t Update the Ppa_h t-1 And update Ppa_x using the new input vector. t Repeat steps (3) to (6) to perform the calculation at the next time step.

[0027] Furthermore, the specific implementation method of step (1) is as follows:

[0028] (1.1) Based on the user description, extract the LSTM network parameter information, including the fixed-point data bit width n and the dimension l of the input vector. x The dimension l of the hidden vector h ;

[0029] (1.2) Based on the aforementioned feature requirements, determine the design dimensions of each component module of the parameterized LSTM acceleration system;

[0030] (1.3) Based on the design dimensions described in (1.2), each component module of the LSTM acceleration system is instantiated and implemented.

[0031] Furthermore, the specific implementation method of step (2) is as follows:

[0032] (2.1) Concatenate the elements of each row of the input weight matrices of the LSTM network's input gate, forget gate, update gate, and output gate from low to high to form a matrix with a bit width of l. x *n bits of data are stored in the corresponding address of Lpm_Wx in module1 to module4;

[0033] (2.2) Concatenate the elements of each row of the hidden weight matrices of the input gate, forget gate, update gate, and output gate of the LSTM network from low to high to form a matrix with a bit width of l. h The *nbit data is stored in the corresponding position in Lpm_Wh of module1 to module4;

[0034] (2.3) Store the bias vectors of the input gate, forget gate, update gate and output gate of the LSTM network into the corresponding positions in P_B of module1 to module4 respectively.

[0035] Furthermore, the specific implementation method of step (3) is as follows:

[0036] (3.1) Concatenate the elements of the input vector at time t from low to high to form a vector with a bit width of l. x *n bits of data are transmitted to Ppa_x t ;

[0037] (3.2) Concatenate the elements of the hidden vector at time t-1 from low to high to form a vector with a bit width of l. h *n bits of data are transmitted to Ppa_h t-1 ;

[0038] (3.3) Transmit each n-bit element of the intermediate state vector at time t-1 to P_Temp;

[0039] (3.4) From the Ppa_x t Parallel read bit width is l x *n bit input vector x t First, split it into m x The width of the unit digit is l x / m x *n bits of data are transmitted in parallel to m in PU_MVx of modules 1 to 4. x Group PE, then split each group of data into g x =l x / m x n bits of data are sent to each PE in parallel;

[0040] (3.5) While executing (3.4), from the Ppa_h t-1 The read bit width is l h The hidden vector h of *n bits t-1 First, split it into m h The width of the unit digit is l h / m h *n bits of data are transmitted in parallel to m in PU_MVh of modules 1 to 4. h Group PE, then split each group of data into g h =l h / m h n bits of data are sent to each PE in parallel;

[0041] (3.6) Initialize the address Addr to 0.

[0042] Furthermore, the specific implementation method of step (4) is as follows:

[0043] (4.1) Read the bit width l in parallel from the corresponding addresses of Lpm_Wx of module1 to module4 respectively. x A line of data containing *n bits is first split into m... x The width of the unit digit is l x / m x *n bits of data are sent in parallel to m in this module's PU_MVx. x Group PE, then split each group of data into g x =l x / m x n bits of data are sent to each PE in parallel;

[0044] (4.2) While executing (4.1), read the bit-width l from the corresponding addresses of Lpm_Wh in modules 1 to 4 in parallel. h A line of data containing *n bits is first split into m... h The width of the unit digit is l h / m h *n bits of data are sent in parallel to the m of PU_MVh in this module. h Group PE, then split each group of data into g h =l h / m h n bits of data are sent to each PE in parallel;

[0045] (4.3) m in PU_MVx of module1 to module4 x m in group PE and PU_MVh h Each PE group performs n-bit fixed-point multiplication in parallel, and accumulates the results within and between groups to obtain their respective n-bit operation results in parallel.

[0046] (4.4) The n-bit outputs of PU_MVx and PU_MV_x of modules 1 to 4, and an n-bit data retrieved from the corresponding address of P_B, are respectively sent to the P_Adder of this module for accumulation, and the n-bit outputs i of each module are obtained in parallel. t f t g t and o t .

[0047] Furthermore, the specific implementation method of step (5) is as follows:

[0048] (5.1) Convert the outputs i of module1 to module4 t f t g t and o t The SI, SF, TG, and SO bits of the NLAM are transmitted respectively, and the operation is performed in parallel to obtain the corresponding n-bit output s_i. t s_f t t_g t and s_o t ;

[0049] (5.2) Output s_i of the n bits of SI, SF, TG and SO in the NLAM. t s_f t t_g t and s_o t and the n-bit data c retrieved from the address corresponding to P_Temp t-1 The data is transmitted to the EDAM, where the calculation is completed, and the corresponding n-bit output c is obtained. t and h t ;

[0050] (5.3) Use the n-bit output c of the EDAM t Update the corresponding address in P_Temp and output the n bits of the EDAM as h. t Stored at the corresponding address of P_Res.

[0051] Furthermore, the specific implementation method of step (7) is as follows:

[0052] (7.1) The l in P_Res h n bits of data are concatenated from low to high to form a data structure with a width of l. h *n bits of data are transmitted to Ppa_h t-1 ;

[0053] (7.2) Concatenate the elements of the new input vector from low to high to form a vector with a bit width of l. x *n bits of data are transmitted to Ppa_x t ;

[0054] (7.3) Repeat steps (3) to (6) to perform the calculation at the next time step.

[0055] Compared with the prior art, the beneficial effects of the present invention are as follows: by parameterizing each component module of the LSTM acceleration system, the system can be flexibly configured for different network structures and hardware platforms, improving its versatility and flexibility; at the same time, by designing storage units that allow parallel data access and standardized data transmission paths, the on-chip parameter access method and the control logic for data transmission and allocation are simplified; and through parameterized grouped parallel operations, the system's computational efficiency is improved. Attached Figure Description

[0056] Figure 1 This is a structural block diagram of the acceleration system described in this invention.

[0057] Figure 2 The flowchart is as follows:

[0058] Figure 3 Schematic diagram of on-chip weight matrix splicing and combination storage

[0059] Figure 4 Schematic diagram for storing bias vectors and intermediate operation results

[0060] Figure 5 Schematic diagram of the parallel grouping multiplication and accumulation module for input rows Detailed Implementation

[0061] The embodiments of the present invention are described in detail below. These embodiments are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. A parameterized LSTM acceleration system design method with parallel data access according to the present invention will be described in detail below with reference to the accompanying drawings.

[0062] like Figure 1 As shown, the present invention discloses a parameterized LSTM acceleration system with parallel data access, including a parameterized input vector parallel access cache module Ppa_x. t Parallel access cache module for parameterized hidden vectors, Ppa_h t-1 The system consists of: a parameterized input gate parallel operation module (module1), a parameterized forget gate parallel operation module (module2), a parameterized update gate parallel operation module (module3), a parameterized output gate parallel operation module (module4), a nonlinear activation function operation module (NLAM), an element-wise multiplication operation module (EDAM), a parameterized intermediate state buffer module (P_Temp), and a parameterized output result buffer module (P_Res).

[0063] Modules 1 through 4 each include a row-parallel storage unit for the input weight matrix Lpm_Wx, a row-parallel storage unit for the hidden weight matrix Lpm_Wh, a bias vector storage unit P_B, a row-parallel group multiplication and accumulation module PU_MVx, a row-parallel group multiplication and accumulation module PU_MVh, and a parameterized parallel accumulation unit P_Adder; NLAM includes three sigmoid activation function units SI, SF, and SO, and one tanh activation function unit TG;

[0064] The Ppa_x t The output is transmitted to PU_MVx of module 1 to module 4, and Ppa_h t-1 The output of Lpm_Wx of module1 to module4 is transmitted to PU_MVh of this module, the output of Lpm_Wh is transmitted to PU_MVh of this module, and the outputs of PU_MVx, PU_MVh and P_B are transmitted to P_Adder of this module.

[0065] The output i of P_Adder of module1 to module4 t f t g t o t The signals are transmitted to SI, SF, TG, and SO of the NLAM, respectively; the output s_i of the NLAM t s_f t t_g t ,s_o t and the output c of P_Temp t-1 Transmitted to the EDAM module; EDAM output c t and h t The data is transmitted to P_Temp and P_Res respectively.

[0066] like Figure 2 As shown in the figure, this embodiment presents a design method for a parameterized LSTM acceleration system with parallel data access, which includes the following steps.

[0067] (1) Based on the LSTM network parameters specified by the user, including the fixed-point data bit width n and the dimension l of the input vector. x And the dimension l of the hidden vector h ,Sure Figure 1 The design dimensions of each module in the parameterized LSTM acceleration system are shown, and each module is instantiated and implemented.

[0068] (1.1) Extracting LSTM network parameter information. Assume the fixed-point data width is 16 bits and the input vector dimension is l. x =256, the dimension of the hidden vector is l h =256.

[0069] (1.2) Determine the design dimensions of each component module of the system. Based on the network parameters and characteristic requirements, Ppa_x can be obtained. t and Ppa_h t-1 The storage depth is 1, and the storage width is 256*16 bits; the storage width of Lpm_Wx and Lpm_Wh in modules 1 to 4 is 256*16 bits, and the storage width of P_Temp and P_Res is 16 bits, with a storage depth of 256; the number of PE groups in PU_MVx and PU_MVh in modules 1 to 4 is 16, and the number of PEs in each group is 16; the data dimension of SI, SF, SO, and TG in NLAM is 16 bits, and SI, SF, and SO implement the following operations.

[0070] sigmoid:

[0071] TG implements the following operations

[0072] tanh:

[0073] EDAM has a data dimension of 16 bits and performs the following operations.

[0074] c t =s_i t ⊙t_g t +s_f t ⊙c t-1

[0075] h t =s_o t ⊙tanh(c t )

[0076] (1.3) Based on the design dimensions described in (1.2), each component module of the LSTM acceleration system is instantiated and implemented.

[0077] (2) Store the input weight matrix, hidden weight matrix and bias vector parameters of the input gate, forget gate, update gate and output gate of the LSTM network into Lpm_Wx, Lpm_Wh and P_B of module1 to module4 respectively.

[0078] (2.1) Concatenate the elements of each row of the input weight matrices of the LSTM network's input gate, forget gate, update gate, and output gate from low to high to form a data with a bit width of 256*16=4096 bits, and store it in the corresponding address of Lpm_Wx in module1~module4, such as... Figure 3 As shown, each input weight matrix is ​​l h =256 lines, corresponding to memory addresses 0 to 255;

[0079] (2.2) Concatenate the elements of each row of the hidden weight matrices of the input gate, forget gate, update gate, and output gate of the LSTM network from low to high to form a matrix with a bit width of l. h The data, *n = 256 * 16 = 4096 bits, is stored in the corresponding positions in Lpm_Wh of module1 to module4, such as... Figure 3 As shown, each hidden weight matrix is ​​l h =256 lines, corresponding to memory addresses 0 to 255.

[0080] (2.3) Store the bias vectors of the LSTM network's input gate, forget gate, update gate, and output gate into the corresponding addresses in P_B of module1 to module4, such as... Figure 4 As shown, the data bit width is n = 16 bits, and the storage address is 0 to 255.

[0081] (3) Based on the storage feature requirements, the input vector at time t, the hidden vector at time t-1, and the intermediate state vector are transmitted to Ppa_x respectively. t Ppa_h t-1 In P_Temp; Ppa_x t Output x t and Ppa_h t The output h t-1 The data is split into 16-bit fixed-point data and transmitted to PU_MVx and PU_MVh of module1 to module4 respectively, and the address Addr is set to 0.

[0082] (3.1) Concatenate the elements of the input vector at time t from low to high to form a vector with a bit width of l. x The data, *n = 256 * 16 bits, with a storage depth of 1, is transmitted to Ppa_x. t .

[0083] (3.2) Concatenate the elements of the hidden vector at time t-1 from low to high to form a vector with a bit width of l. h The data, *n = 256 * 16 bits, with a storage depth of 1, is transmitted to Ppa_h. t-1 .

[0084] (3.3) Transmit each 16-bit element of the intermediate state vector at time t-1 to P_Temp, with a depth of l. h =256.

[0085] (3.4)Ppa_x t Parallel read bit width is l x *n = 256 * 16 bits input vector x t First, split it into m x =16 bits wide x / m x *n = 256 / 16 * 16 bits of data is transmitted in parallel to 16 PEs in PU_MVx of module1 to module4, and then each data group is split into g x =l x / m x = 16 16-bit data segments, sent in parallel to each PE.

[0086] (3.5) While executing (3.4), from Ppa_h t-1 The read bit width is l h *n = 256 * 16 bits of hidden vector h t-1 First, split it into m h =16 bits wide h / m h *n = 256 / 16 * 16 bits of data is transmitted in parallel to 16 PEs in PU_MVh of module1 to module4, and then each data group is split into g h =l h / m h = 16 16-bit data segments, sent in parallel to each PE.

[0087] (3.6) Initialize the address Addr to 0.

[0088] (4) Read a row of data in parallel from the corresponding addresses of Lpm_Wx and Lpm_Wh of module1 to module4 respectively, and split it into 16-bit fixed-point data, and transmit it to PU_MVx and PU_MVh of the corresponding module respectively, and perform the grouping, multiplication and accumulation operation of a row of data in the four gates in parallel; and send the corresponding operation result and a data taken from the corresponding address of P_B ​​to the P_Adder for accumulation operation.

[0089] (4.1) Read the bit width l from the corresponding addresses of Lpm_Wx in module1 to module4 in parallel. xA line of data, *n = 256 * 16 bits, is first split into m... x =16 bits wide x / m x *n = 256 / 16 * 16 bits of data are sent in parallel to 16 groups of PEs in this module PU_MVx, and then each group of data is split into g x =l x / m x = 16 16-bit data segments, sent in parallel to each PE.

[0090] (4.2) While executing (4.1), read the bit-width l from the corresponding addresses of Lpm_Wh in module1 to module4 in parallel. h A line of data, *n = 256 * 16 bits, is first split into m... h =16 bits wide h / m h *n = 256 / 16 * 16 bits of data, which are sent in parallel to the 16 PEs of PU_MVh in this module, and then each group of data is split into g h =l h / m h = 16 16-bit data segments, sent in parallel to each PE.

[0091] (4.3) The 16 PEs in PU_MVx and the 16 PEs in PU_MVh of module1 to module4 perform 16-bit fixed-point multiplication in parallel, and accumulate the results within and between groups to obtain their respective 16-bit operation results in parallel.

[0092] (4.4) The 16-bit outputs of PU_MVx and PU_MV_x of module1 to module4, as well as a 16-bit data retrieved from the corresponding address of P_B, are respectively sent to P_Adder of this module for accumulation, and the 16-bit output i of each module is obtained in parallel. t f t g t and o t .

[0093] (5) Convert the output i of P_Adder in module1 to module4 t f t g t and o t The data is sent to NLAM and EDAM respectively, and the relevant calculations are performed in parallel; the output c of EDAM is used. t Update the address corresponding to P_Temp and change the output h of EDAM. t Store at the address corresponding to P_Res.

[0094] (5.1) Convert the outputs i of module1 to module4 t f t g t and o t The signals are transmitted to SI, SF, TG, and SO of the NLAM respectively, and the operation is performed in parallel to obtain the corresponding n=16-bit output s_i. t s_f t t_g t and s_o t .

[0095] (5.2) Output s_i of the n bits of SI, SF, TG and SO in the NLAM. t s_f t t_g t and s_o t And the n=16bit data c retrieved from the address corresponding to P_Temp t-1 The data is transmitted to the EDAM, where the calculation is completed, and the corresponding n=16-bit output c is obtained. t and h t .

[0096] (5.3) Use the n=16-bit output c of the EDAM t Update the corresponding address in P_Temp and output h of the EDAM (n=16 bits). t Stored at the corresponding address of P_Res.

[0097] (6) Increment the address Addr of Lpm_Wx, Lpm_Wh, P_B, P_Temp and P_Res of module1 to module4 by 1, and repeat steps (4) and (5) until all 256 lines of data have been executed.

[0098] (7) Using the output h of P_Res t Update Ppa_h t-1 And update Ppa_x using the new input vector. t Repeat steps (3) to (6) to perform the calculation at the next time step.

[0099] (7.1) Concatenate the 256 16-bit data in P_Res from low to high to form a data structure with a bit width of l. h *n = 256 * 16 bits of data, transmitted to Ppa_h t-1 ;

[0100] (7.2) Concatenate the elements of the new input vector from low to high to form a vector with a bit width of l.x *n = 256 * 16 bits of data, transmitted to Ppa_x t ;

[0101] (7.3) Repeat steps (3) to (6) to perform the calculation at the next time step.

[0102] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A parameterized LSTM acceleration system with parallel data access, characterized in that, Includes the parameterized input vector parallel access cache module Ppa_x t Parallel access cache module for parameterized hidden vectors, Ppa_h t-1 The system consists of: a parameterized input gate parallel operation module (module1), a parameterized forget gate parallel operation module (module2), a parameterized update gate parallel operation module (module3), a parameterized output gate parallel operation module (module4), a nonlinear activation function operation module (NLAM), an element-wise multiplication operation module (EDAM), a parameterized intermediate state buffer module (P_Temp), and a parameterized output result buffer module (P_Res). Modules 1 through 4 each include a row-parallel storage unit for the input weight matrix Lpm_Wx, a row-parallel storage unit for the hidden weight matrix Lpm_Wh, a bias vector storage unit P_B, a row-parallel group multiplication and accumulation module PU_MVx, a row-parallel group multiplication and accumulation module PU_MVh, and a parameterized parallel accumulation unit P_Adder; NLAM includes three sigmoid activation function units SI, SF, and SO, and one tanh activation function unit TG. The Ppa_x t The output is transmitted to PU_MVx of module 1 to module 4, and Ppa_h t-1 The output of Lpm_Wx of module1 to module4 is transmitted to PU_MVh of this module, the output of Lpm_Wh is transmitted to PU_MVh of this module, and the outputs of PU_MVx, PU_MVh and P_B are transmitted to P_Adder of this module. The output i of P_Adder of module1 to module4 t f t g t o t The signals are transmitted to SI, SF, TG, and SO of the NLAM, respectively; the output s_i of the NLAM t s_f t t_g t ,s_o t and the output c of P_Temp t-1 Transmitted to the EDAM module; EDAM output c t and h t The data is transmitted to P_Temp and P_Res respectively.

2. A parameterized LSTM acceleration system with parallel data access as described in claim 1, characterized in that, The Ppa_x t and Ppa_h t-1 All of these can be parameterized, with a storage depth of 1 and a storage width of l. x *n bit and l h *nbit; where l x and l h are the dimensions of the input and hidden vectors, respectively, and n is the fixed-point data bit-width.

3. A parameterized LSTM acceleration system with parallel data access as described in claim 1, characterized in that, The P_Temp and P_Res, as well as Lpm_Wx, Lpm_Wh, and P_B of modules 1 to 4, can all be parameterized, and their storage depth is l. h The storage widths of Lpm_Wx and Lpm_Wh are l x *n bit and l h *n bits, the storage width of P_Temp and P_Res is n bits; where l x and l h are the dimensions of the input vector and the hidden vector, respectively, and n is the fixed-point data width.

4. A parameterized LSTM acceleration system with parallel data access as described in claim 1, characterized in that, The processing units PE in PU_MVx and PU_MVh of modules 1 to 4 that perform row-parallel multiplication and accumulation are implemented in groups, and the number of groups and the number of PEs in each group can be parameterized. The number of groups are as follows: and The number of PEs in each group is g x =l x / m x and g h =l h / m h ; where l x and l h These are the dimensions of the input vector and the hidden vector, respectively.

5. A parameterized LSTM acceleration system with parallel data access as described in claim 1, characterized in that, In the NLAM, SI, SF, SO, and TG can all be parameterized, with a data dimension of fixed-point data width n bits; and SI, SF, and SO perform the following operations. sigmoid: TG implements the following operations fishy:

6. A parameterized LSTM acceleration system with parallel data access as described in claim 1, characterized in that, The EDAM can be parameterized and its data dimension is a fixed-point data width of n bits, and the following operations are performed. c t =s-i t ⊙t_g t +s_f t ⊙c t-1 h t =s_o t ⊙tanh(c t ) Where s_i t s_f t t_g t ,s_o t These are the outputs of SI, SF, TG, and SO in the NLAM, respectively, c t-1 For the output of P_Temp, c t h represents the intermediate state value at the current moment. t Output the result for the current moment.

7. A design method for a parameterized LSTM accelerated system with parallel data access, characterized in that, The method, applied to a parameterized LSTM acceleration system with parallel data access as described in any one of claims 1-6, includes the following steps: (1) Based on the LSTM network parameters specified by the user, including the fixed-point data bit width n and the dimension l of the input vector. x And the dimension l of the hidden vector h The design dimensions of each module in the parameterized LSTM acceleration system are determined, and each module is instantiated and implemented. (2) Based on the storage characteristics requirements, the input weight matrix, hidden weight matrix and bias vector parameters of the input gate, forget gate, update gate and output gate of the LSTM network are stored in Lpm_Wx, Lpm_Wh and P_B of module1 to module4 respectively. (3) Based on the storage requirements, the input vector at time t, the hidden vector at time t-1, and the intermediate state vector are transmitted to Ppa_x respectively. t Ppa_h t-1 In P_Temp; the Ppa_x t Output x t and Ppa_h t The output h t-1 The data is split into n bits of fixed-point data and transmitted to PU_MVx and PU_MVh of module1 to module4 respectively, and the address Addr is set to 0. (4) Read a row of data in parallel from the corresponding addresses of Lpm_Wx and Lpm_Wh of module1 to module4 respectively, and split it into n-bit fixed-point data, and transmit it to PU_MVx and PU_MVh of the corresponding module respectively, and perform the grouping, multiplication and accumulation operation of a row of data in the four gates in parallel; and send the corresponding operation result and a data taken from the corresponding address of P_B ​​to P_Adder for accumulation operation; (5) The output i of P_Adder in module1 to module4 t f t g t and o t The data are sent to NLAM and EDAM respectively, and the relevant calculations are performed in parallel; the output c of the EDAM is used. t Update the address corresponding to P_Temp and change the output h of the EDAM. t Stored at the address corresponding to P_Res; (6) Increment the address Addr of Lpm_Wx, Lpm_Wh, P_B of module1 to module4, and the address Addr of P_Temp and P_Res by 1, and repeat steps (4) and (5) until l h All rows of data have been processed. (7) Based on the storage characteristics requirements, use the output h of the P_Res. t Update the Ppa_h t-1 And update Ppa_x using the new input vector. t Repeat steps (3) to (6) to perform the calculation at the next time step.

8. The design method for a parameterized LSTM accelerated system with parallel data access according to claim 7, characterized in that, The specific method for step (1) is as follows: (1.1) Based on the user description, extract the LSTM network parameter information, including the fixed-point data bit width n and the dimension l of the input vector. x The dimension l of the hidden vector h ; (1.2) Based on the characteristic requirements, determine the design dimensions of each component module of the parameterized LSTM acceleration system; (1.3) Based on the design dimensions described in (1.2), each component module of the LSTM acceleration system is instantiated and implemented.

9. The design method for a parameterized LSTM accelerated system with parallel data access according to claim 7, characterized in that, The specific method for step (2) is as follows: (2.1) Concatenate the elements of each row of the input weight matrices of the LSTM network's input gate, forget gate, update gate, and output gate from low to high to form a matrix with a bit width of l. x *n bits of data are stored in the corresponding positions in Lpm_Wx of module1 to module4; (2.2) Concatenate the elements of each row of the hidden weight matrices of the input gate, forget gate, update gate, and output gate of the LSTM network from low to high to form a matrix with a bit width of l. h The *n bits of data are stored in the corresponding positions in Lpm_Wh of module1 to module4 respectively; (2.3) Store the bias vectors of the input gate, forget gate, update gate and output gate of the LSTM network into the corresponding positions in P_B of module1 to module4 respectively.

10. The design method for a parameterized LSTM accelerated system with parallel data access according to claim 7, characterized in that, The specific method for step (3) is as follows: (3.1) Concatenate the elements of the input vector at time t from low to high to form a vector with a bit width of l. x *n bits of data are transmitted to Ppa_x t ; (3.2) Concatenate the elements of the hidden vector at time t-1 from low to high to form a vector with a bit width of l. h *n bits of data are transmitted to Ppa_h t-1 ; (3.3) Transmit each n-bit element of the intermediate state vector at time t-1 to P_Temp; (3.4) From the Ppa_x t Parallel read bit width is l x *n bit input vector x t First, split it into m x The width of the unit digit is l x / m x *n bits of data are transmitted in parallel to m in PU_MVx of modules 1 to 4. x Group PE, then split each group of data into g x =l x / m x n bits of data are sent to each PE in parallel; (3.5) While executing (3.4), from the Ppa_h t-1 The read bit width is l h The hidden vector h of *n bits t-1 First, split it into m h The width of the unit digit is l h / m h *n bits of data are transmitted in parallel to m in PU_MVh of modules 1 to 4. h Group PE, then split each group of data into g h =l h / m h n bits of data are sent to each PE in parallel; (3.6) Initialize the address Addr to 0.

11. The design method for a parameterized LSTM accelerated system with parallel data access according to claim 7, characterized in that, The specific method for step (4) is as follows: (4.1) Read the bit width l in parallel from the corresponding addresses of Lpm_Wx of module1 to module4 respectively. x A line of data containing *n bits is first split into m... x The width of the unit digit is l x / m x *n bits of data are sent in parallel to m in this module's PU_MVx. x Group PE, then split each group of data into g x =l x / m x n bits of data are sent to each PE in parallel; (4.2) While executing (4.1), read the bit-width l from the corresponding addresses of Lpm_Wh in modules 1 to 4 in parallel. h A line of data containing *n bits is first split into m... h The width of the unit digit is l h / m h *n bits of data are sent in parallel to the m of PU_MVh in this module. h Group PE, then split each group of data into g h =l h / m h n bits of data are sent to each PE in parallel; (4.3) m in PU_MVx of the module1 ~ module4 x PE and m in PU_MVh group h PE and m in PU_MVh group The PE group completes n bit fixed-point multiplication operation in parallel, and accumulates the operation results in the group and between groups, and obtains the respective n bit operation results in parallel. (4.4) The n-bit outputs of PU_MVx and PU_MV_x of modules 1 to 4, and an n-bit data retrieved from the corresponding address of P_B, are respectively sent to the P_Adder of this module for accumulation, and the n-bit outputs i of each module are obtained in parallel. t f t g t and o t .

12. The design method for a parameterized LSTM accelerated system with parallel data access according to claim 7, characterized in that, The specific method for step (5) is as follows: (5.1) The outputs i t , f t , g t , and o t of the module1~module4 are transmitted to the SI, SF, TG, and SO of the NLAM, respectively, and sigmoid and tanh operations are performed in parallel to obtain the corresponding n bit outputs s_i t , s_f t , t_g t , and s_o t ; (5.2) Output s_i of the n bits of SI, SF, TG and SO in the NLAM. t s_f t t_g t and s_o t and the n-bit data c retrieved from the address corresponding to P_Temp t-1 Transmitted to the EDAM to complete c t and h t The operation yields the corresponding n-bit output c. t and h t ; (5.3) Use the n-bit output c of the EDAM t Update the corresponding address in P_Temp and output the n bits of the EDAM as h. t Stored at the corresponding address of P_Res.

13. The design method for a parameterized LSTM accelerated system with parallel data access according to claim 7, characterized in that, The specific method for step (7) is as follows: (7.1) The l in P_Res h n bits of data are concatenated from low to high to form a data structure with a width of l. h *n bits of data are transmitted to Ppa_h t-1 ; (7.2) Concatenate the elements of the new input vector from low to high to form a vector with a bit width of l. x *n bits of data are transmitted to Ppa_x t ; (7.3) Repeat steps (3) to (6) to perform the calculation at the next time step.

Images