Neural network forward prediction hardware accelerator based on programmable logic device

Abstract

The neural network forward prediction hardware accelerator based on the programmable logic device can be applied to the technical field of neural network hardware acceleration. The accelerator comprises a multi-layer hardware accelerator module unit, and the hardware accelerator module unit comprises an input feature map cache module and a data processing module; the accelerator module units are connected in an assembly line structure mode; multi-channel feature maps can be processed in parallel; and the output data of the hardware accelerator module unit of each layer is stored in the input feature map cache module of the hardware accelerator module unit of the next layer for the operation of the next layer. By adopting an assembly line structure, the throughput rate of the hardware accelerator is improved; a multi-channel parallel processing mode is adopted, and the running speed of a hardware accelerator is increased; and on-chip storage is adopted, so that the off-chip access times are reduced, and the power consumption of the hardware accelerator is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of neural network hardware acceleration, and in particular to a neural network forward prediction hardware accelerator based on programmable logic devices. Background Art

[0002] In recent years, with the growth of data demand and the improvement of hardware computing performance, artificial intelligence has achieved a revival, especially in the field of deep learning. Deep learning can learn features with multiple levels of abstraction, so deep learning has excellent performance in solving complex and abstract learning problems, such as face recognition, autonomous driving, etc. However, as the problems become more complex and abstract, the models of deep learning networks become more complex, and the learning time of the models is also increasing.

[0003] The reasoning platform of neural network mainly includes CPU (central processing unit), GPU (graphics processing unit), ASIC (application-specific integrated circuit) and FPGA (programmable logic device), but CPU cannot meet the needs of high-performance parallel computing. Although GPU can do parallel computing, its reasoning speed is slow and its power consumption is large, so it cannot be applied to the edge. ASIC is not flexible enough, and has a long development cycle and high initial cost. Compared with CPU, GPU and ASIC platforms, FPGA has the advantages of high speed, low power consumption, low cost, reconfigurability and portability. Using FPGA to accelerate the calculation of the reasoning process of neural network can meet the complex reasoning environment and is more suitable for practical application development.

[0004] However, existing FPGA-based neural network hardware accelerators have a lot of circuit redundancy and low resource utilization. In addition, they frequently access off-chip storage, resulting in high power consumption and low speed of the accelerator. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] In order to solve the problems existing in the neural network hardware accelerator based on programmable logic devices in the prior art, an embodiment of the present invention provides a neural network forward prediction hardware accelerator based on programmable logic devices, which improves the throughput of the hardware accelerator by adopting a pipeline structure; improves the operating speed of the hardware accelerator by adopting a multi-channel parallel processing method; improves the data reuse rate by splicing and combining multiple sub-cache areas into a feature map cache area; and adopts on-chip storage to reduce the number of off-chip accesses and reduce the power consumption of the hardware accelerator.

[0007] (II) Technical solution

[0008] In view of the above problems, an embodiment of the present invention provides a neural network forward prediction hardware accelerator based on a programmable logic device.

[0009] According to a first aspect of the present invention, a neural network forward prediction hardware accelerator based on a programmable logic device is provided, comprising: a neural network forward prediction hardware accelerator based on a programmable logic device, characterized in that it comprises: a multi-layer hardware accelerator module unit, wherein the hardware accelerator module unit comprises an input feature map cache module and a data processing module; the accelerator module units are connected through a pipeline structure; multi-channel feature maps can be processed in parallel; and the output data of the hardware accelerator module unit of each layer is stored in the input feature map cache module of the hardware accelerator module unit of the next layer for use by the next layer of operation.

[0010] In some exemplary embodiments, the input feature map cache module is used to perform read and write operations on the input feature map; the data processing module includes a weight parameter cache module, a systolic array convolution calculation module, a normalization module, an activation function module and a pooling module, wherein the weight parameter cache module is an on-chip storage unit for storing convolution weights and bias parameters; the systolic array convolution calculation module is used to use the systolic array to calculate the convolution operation; the normalization module is used to normalize the data; the activation function module is used to operate on the normalized data using a nonlinear activation function; and the pooling module is used to complete the maximum pooling operation using the spliced ​​input feature map cache module to reduce the data dimension.

[0011] In some exemplary embodiments, the process of the input feature map cache module performing read and write operations on the input feature map specifically includes: dividing the input feature map cache module into multiple sub-cache areas; storing the input feature map in corresponding sub-cache areas according to rows and columns; selecting sub-cache areas corresponding to consecutive columns and reading out data of the sub-cache areas corresponding to different rows in parallel, and writing data of the sub-cache areas corresponding to the last idle column; and cyclically selecting consecutive columns of data of the input feature map to perform the above-mentioned read and write operations; wherein the data of the sub-cache areas corresponding to different rows are used to input the systolic array convolution calculation module to perform convolution calculation.

[0012] In some exemplary embodiments, the size of the sub-buffer area is related to the size of the convolution kernel, wherein the number of rows of the cache area is the same as the number of rows of the convolution kernel; the number of columns of the cache area is one more than the number of columns of the convolution kernel; and the timing of the read-out column data differs by one clock cycle.

[0013] In some exemplary embodiments, the process of storing the weight and bias parameters of the convolution in the weight parameter cache module specifically includes: storing the weight parameters and bias parameters related to the convolution calculation in an on-chip storage unit, wherein, for a convolution kernel of size N×N, N columns of weight parameters are respectively stored in N on-chip storage units; for a convolution kernel of size N×N, data W1, W2, W3...WN in N on-chip storage units are read out in parallel, wherein the read timing of W1, W2, W3...WN differs by one clock cycle; and N parameters stored in each on-chip storage unit are read cyclically, wherein N is a positive integer.

[0014] In some exemplary embodiments, the process of calculating the convolution operation by the systolic array convolution calculation module specifically includes: using each calculation unit in the systolic array to perform a multiplication operation on the input feature map data input at the current moment and the weight parameter; accumulating the calculation results of the calculation units at different moments and different positions to obtain the convolution result of the pixel.

[0015] In some exemplary embodiments, the process of the normalization module performing normalization operations on data specifically includes: concatenating three normalization parameters into one parameter and storing it in an on-chip storage unit; and reading out the normalization parameters stored in the on-chip storage unit and splitting them into three parallel normalization parameters for multiplication, addition, and subtraction operations.

[0016] In some exemplary embodiments, the activation function module performs a normalization operation on the data including: removing the linear correlation of the data and improving the generalization ability of the network.

[0017] In some exemplary embodiments, the pooling module uses a spliced ​​input feature map cache module to complete the process of the maximum pooling operation, which specifically includes: setting 4 sub-cache areas P1, P2, P3, and P4 in each layer of the input feature map cache module that can cache 4 columns of input feature map data; storing 2 consecutive columns of input feature map data in sub-cache areas P1 and P2 in sequence; reading the data in sub-cache areas P1 and P2 in parallel, and writing them into sub-cache area P3; calculating the maximum value of the 4 data in every 2 clock cycles as output, and writing them into sub-cache area P4; then, writing into sub-cache areas P1 and P2, and reading sub-cache areas P3 and P4 in parallel... until all feature maps are traversed.

[0018] In some exemplary embodiments, the neural network forward prediction hardware accelerator increases the data dimension through the nearest neighbor interpolation algorithm, specifically including: storing the input data in the input feature map cache module of each layer; when reading out, reading out the same address data every 2 clock cycles, and reading out the column cache area of ​​the same input data every two columns.

[0019] (III) Beneficial effects

[0020] It can be seen from the above technical solutions that the neural network forward prediction hardware accelerator based on programmable logic devices provided by the embodiments of the present invention has at least one of the following beneficial effects:

[0021] (1) Use a pipeline structure to improve the throughput of the hardware accelerator.

[0022] (2) Use multi-channel parallel processing to improve the operating speed of the hardware accelerator.

[0023] (3) Multiple sub-cache areas are spliced ​​and combined into a feature map cache area to improve the data reuse rate.

[0024] (4) On-chip storage is used to reduce the number of off-chip accesses and lower the power consumption of the hardware accelerator. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0026] Figure 1 A schematic diagram of the structure of a neural network forward prediction hardware accelerator based on a programmable logic device according to an embodiment of the present invention is shown;

[0027] Figure 2 A schematic diagram of the structure of a hardware accelerator module unit according to an embodiment of the present invention is shown;

[0028] Figure 3 A schematic diagram of a process flow of the input feature map cache module performing read and write operations on the input feature map according to an embodiment of the present invention is shown;

[0029] Figure 4 A schematic diagram schematically shows a schematic diagram of an input feature map cache module performing read and write operations on an input feature map according to an embodiment of the present invention;

[0030] Figure 5 A schematic diagram of a process of calculating convolution using a systolic array according to an embodiment of the present invention is shown;

[0031] Figure 6 Schematically showing the calculation results of different PE units in different clock cycles according to an embodiment of the present invention;

[0032] Figure 7 A schematic diagram of a maximum pooling process according to an embodiment of the present invention is shown; and

[0033] Figure 8A schematic diagram of an upsampling process of a neural network forward prediction hardware accelerator based on a programmable logic device according to an embodiment of the present invention is schematically shown.

[0034] Illustration Description:

[0035] 100-hardware accelerator module unit; 110-input feature map cache module; 120-data processing module; 121-weight parameter cache module; 122-systolic array convolution calculation module; 123-standardization module; 124-activation function module; 125-pooling module; S201-input feature map; S202-sub-cache area; S203-write operation; S204-read operation; S302-convolution kernel; S303-systolic array; S304-PE unit; S501-cache area; S502-pooling operation; S601-upsampling. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] Figure 1 A schematic diagram of the structure of a neural network forward prediction hardware accelerator based on a programmable logic device according to an embodiment of the present invention is shown schematically.

[0038] like Figure 1 As shown, a neural network forward prediction hardware accelerator based on a programmable logic device according to an embodiment of the present invention includes: a multi-layer hardware accelerator module unit 100, wherein the hardware accelerator module unit 100 includes an input feature map cache module 110 and a data processing module 120; the accelerator module units are connected through a pipeline structure so that multi-channel feature maps can be processed in parallel; and the output data of the hardware accelerator module unit 100 of each layer is stored in the input feature map cache module 110 of the hardware accelerator module unit 100 of the next layer for use in the next layer of operations.

[0039] Figure 2 The schematic diagram shows the structure of the hardware accelerator module unit 100 according to an embodiment of the present invention.

[0040] like Figure 2As shown, the hardware accelerator module unit 100 according to an embodiment of the present invention includes an input feature map cache module 110 and a data processing module 120, wherein the input feature map cache module 110 is used to perform read and write operations on the input feature map; the data processing module 120 includes a weight parameter cache module 121, a systolic array convolution calculation module 122, a normalization module 123, an activation function module 124 and a pooling module 125, wherein the weight parameter cache module 121 is an on-chip storage unit for storing convolution weights and bias parameters; the systolic array convolution calculation module 122 is used to use the systolic array to calculate the convolution operation; the normalization module 123 is used to normalize the data; the activation function module 124 is used to operate the normalized data using a nonlinear activation function; and the pooling module 125 is used to use the spliced ​​input feature map cache module 110 to complete the maximum pooling operation to reduce the data dimension.

[0041] In the embodiment of the present invention, the input feature map data of each layer of convolution is stored in the on-chip BRAM (RAM is Random Access Memory, that is, random access data storage. The interior of RAM is composed of a small memory unit (which can be regarded as a small grid). Each memory unit corresponds to an address number. When we need to read and write data, we access the corresponding memory unit address and quickly write or read the data. BRAM is Block RAM, that is, a dedicated RAM resource in FPGA, which is fixedly distributed in a specific position inside FPGA), which greatly reduces the number of off-chip data simulations and reduces power consumption; a pipeline structure is used to realize neural network forward prediction, and each layer of convolution input cache only needs to cache a small part of the input feature map data to start working, which reduces the on-chip cache resources and cache waiting time, and improves the circuit throughput; each layer of convolution is based on the number of input feature map channels, and the circuit is parallelized, which improves the parallelism and reduces the circuit running time.

[0042] Figure 3 The figure schematically shows a flow chart of the reading and writing operations of the input feature map by the input feature map cache module according to an embodiment of the present invention.

[0043] like Figure 3 As shown, the input feature map cache module 110 according to an embodiment of the present invention performs read and write operations on the input feature map, specifically including steps S110-S140.

[0044] In step S110, the input feature map cache module 110 is divided into a plurality of sub-cache areas.

[0045] In some exemplary embodiments, the size of the sub-buffer area is related to the size of the convolution kernel, wherein the number of rows of the cache area is the same as the number of rows of the convolution kernel; and the number of columns of the cache area is one more than the number of columns of the convolution kernel.

[0046] In step S120, the input feature map is stored in corresponding sub-buffer areas according to rows and columns.

[0047] According to the convolution kernel size N×N, the input feature map cache is divided into N×(N+1) sub-cache areas; the sub-cache areas use on-chip BRAM resources; the consecutive N+1 columns of the input feature map are stored in these sub-cache areas; when storing a column of input feature map data, according to its corresponding row, the (1, N+1, 2N+1...)th row is stored in the first sub-cache area H1, the (2, N+2, 2N+2...)th row is stored in the second sub-cache area H2, ..., the (N, 2N, 3N...)th row is stored in the Nth sub-cache area H N middle.

[0048] In step S130, sub-cache areas corresponding to consecutive columns are selected to read out data of the sub-cache areas corresponding to different rows in parallel, and data of the sub-cache areas corresponding to different rows are read out in parallel, and data of the sub-cache areas corresponding to the last idle column are written; wherein the data of the sub-cache areas corresponding to different rows are input into the systolic array convolution calculation module 122 for convolution calculation.

[0049] In some exemplary embodiments, sub-cache areas corresponding to several consecutive columns are selected to read out data of the sub-cache areas corresponding to different rows in parallel, specifically including: reading out data of the sub-cache areas corresponding to different rows in parallel in the order of input feature map columns, first reading out (1, 2, 3, ..., N) in parallel, then reading out (N+1, N+2, N+3, ...2N) in parallel, until reading out data of all columns and all rows stored in parallel.

[0050] In some exemplary embodiments, writing data to the sub-cache area corresponding to the last idle column specifically includes: selecting N+1 columns of the input feature map according to the convolution kernel size N×N and storing them in the input feature map cache area, first writing the N columns to the corresponding areas L1, L2, L3...L N Then read out the area L1, L2, L3...L N Columns, and write one column to area L N+1 ; After reading is completed, read out area L2, L3, L4...L N+1 The next column is written into the L1 column, and so on. In this way, the cache area corresponding to the input feature map of consecutive N columns is selected for reading in a loop, and a column is written into the free area at the same time. The read-out timings of the N columns of data read out in parallel differ by one clock cycle.

[0051] In step S140, several consecutive columns of data of the input feature map are cyclically selected to execute the above steps S120 and S130.

[0052] Since neural networks have a large number of parameters and feature map channels, a large amount of storage resources are required to cache these data during the hardware deployment of neural networks. The on-chip storage resources of programmable logic devices are limited, and data often need to be stored in off-chip storage resources, which increases access to off-chip resources and consumes more power consumption and time. To address this problem, an embodiment of the present invention uses row-based splicing and column-based cyclic reading and writing to read and write input feature maps. In this way, only a small amount of input feature map data needs to be stored in the on-chip storage resources before calculation can begin.

[0053] Figure 4 A schematic diagram schematically shows a schematic diagram of the input feature map cache module performing read and write operations on the input feature map according to an embodiment of the present invention.

[0054] like Figure 4 As shown, taking the 6×6 input feature map S201 and the 3×3 convolution kernel as an example, the input feature map S201 is read and written in a row-by-row and column-by-column manner. The write operation S203 on the input feature map S201 includes dividing the input feature map cache module 110 into 12 sub-cache areas S202 according to the convolution kernel size 3×3, corresponding to the four columns L1, L2, L3, and L4 and the three rows H1, H2, and H3, and cyclically writing the first column of the input feature map S201 data into the sub-cache areas H1L1, H2L1, and H3L1 according to its row position, cyclically writing the second column of the input feature map S201 data into the sub-cache areas H1L2, H2L2, and H3L2 according to its row position, and cyclically writing the third column of the input feature map S201 data into the sub-cache areas H1L2, H2L2, and H3L2 according to its row position. The sub-cache areas H1L3, H2L3, and H3L3 are written cyclically according to their row positions, the fourth column of input feature map S201 data is written cyclically into the sub-cache areas H1L4, H2L4, and H3L4 according to their row positions, the fifth column of input feature map S201 data is written cyclically into the sub-cache areas H1L1, H2L1, and H3L1 according to their row positions, covering the first column of input feature map S201 data, and the sixth column of input feature map S201 data is written cyclically into the sub-cache areas H1L2, H2L2, and H3L2 according to their row positions, covering the second column of input feature map S201 data.

[0055] A read operation S204 is performed on the input feature map S201. Specifically, when the read operation S204 has written three columns of input feature map S201 data and starts to write the fourth column of input feature map S201 data, the input feature map S201 data is read out for systolic array convolution calculation, and three groups of input feature map S201 data are read out in parallel according to the convolution kernel size 3×3. First, all the input feature map S201 data of the first, second and third columns are read out in parallel: the input feature map S201 data of the first, second and third columns of the first, second and third rows are read out in parallel in sequence using 3 clock cycles (clock cycle ①: A1_1, A2_1, A3_1; clock cycle ②: A1_2, A2_2, A3_2; clock cycle ③: A1_3, A2_3, A3_3), and then the input feature map S201 data of the first, second and third columns of the fourth, fifth and sixth rows are read out in parallel in sequence using 3 clock cycles (clock cycle ④: A4_1, A5_1, A6_1; clock cycle ⑤: A4_2, A5_2, A6_2; clock cycle ⑥: A4_3, A5_3, A6_3). After all the input feature map S201 data of the first, second and third columns are read out in parallel in the same manner; after all the input feature map S201 data of the second, third and fourth columns are read out in parallel in the same manner; after all the input feature map S201 data of the third, fourth and fifth columns are read out in parallel in the same manner; after all the input feature map S201 data of the third, fourth and fifth columns are read out in parallel in the same manner.

[0056] In an embodiment of the present invention, the storage unit for caching the input feature map S201 selects a BRAM resource on a programmable logic device chip.

[0057] In some exemplary embodiments, the process of storing the weight and bias parameters of the convolution in the weight parameter cache module 121 specifically includes: storing the weight parameters and bias parameters related to the convolution calculation in the on-chip storage unit, wherein, for a convolution kernel of size N×N, N columns of weight parameters are stored in N on-chip storage units respectively; for a convolution kernel of size N×N, data W1, W2, W3...W in the N on-chip storage units are read out in parallel. N , where W1, W2, W3...W N The readout timings differ by one clock cycle; N parameters stored in each on-chip storage unit are cyclically read, where N is a positive integer.

[0058] Since the on-chip memory has fast access speed and low access latency, the weights and bias parameters of the neural network are made into COE files (COE files are a commonly used memory initialization file format. COE files can contain initialization values ​​for memory contents and some metadata information. COE files mainly consist of two parts: the first part is header information, and the second part is memory initialization data) and stored in the on-chip memory. Since each channel operates in parallel, the weights and bias parameters need to be stored in different on-chip memories, so that the weight and bias parameter memory corresponding to each channel is small. Using distributed RAM, the bit width and depth can be configured at will, which can avoid wasting on-chip BRAM resources.

[0059] In some exemplary embodiments, the process of calculating the convolution operation of the systolic array convolution calculation module 122 specifically includes: using each calculation unit in the systolic array to perform a multiplication operation on the input feature map data input at the current moment and the weight parameter; accumulating the calculation results of the calculation units at different times and different positions to obtain the convolution result of the pixel. Optionally, the calculation unit is a PE calculation unit (process element, which is the smallest unit of the array and realizes one-dimensional multiplication and addition calculation), and each PE calculation unit uses an IP core to perform a multiplication operation on the input feature map data input at the current moment and the weight parameter. Optionally, accumulating the calculation results of the PE calculation units at different times and different positions to obtain the convolution result of a pixel includes: according to the convolution kernel size N×N, the PE array size of the systolic array convolution calculation module 122 is set to N×N; accumulating the results of different PE calculation units within a period to obtain the result of multiplication and addition, that is, the convolution result; after a delay of (N-1)×N+1 clock cycles from the input of the first set of input feature map data and weight parameters, the convolution result is continuously output.

[0060] In the embodiment of the present invention, a systolic array is used to calculate convolution, and data flows in the array of the operation unit, which reduces the number of data accesses and makes the circuit structure more regular and the wiring more uniform, which is conducive to improving the circuit frequency.

[0061] Figure 5 The figure schematically shows a process diagram of calculating convolution using a systolic array according to an embodiment of the present invention.

[0062] like Figure 5 As shown, according to the size of the convolution kernel S302 3×3, the size of the systolic array S303 is designed to be 3×3, the three groups of parallel input input feature map S201 data are delayed by one clock cycle each other, and the three groups of parallel input weight parameters are also delayed by one clock cycle each other, and each PE unit S304 multiplies the input feature map S201 data input at the current moment by the weight parameter, and the multiplier used is the IP core of the programmable logic device.

[0063] Figure 6 The schematic diagram shows the calculation results of different PE units in different clock cycles according to the embodiment of the present invention. The result of the convolution operation can be obtained by accumulating the calculation results of the corresponding PE units in the corresponding clock cycles.

[0064] In some exemplary embodiments, the process of normalizing the data by the normalization module 123 specifically includes: concatenating three normalization parameters into one parameter and storing it in an on-chip storage unit; and reading the normalization parameters stored in the on-chip storage unit and splitting them into three parallel normalization parameters for multiplication, addition, and subtraction operations. Optionally, after the normalization parameters stored in the on-chip BRAM are read out, they are split into three parallel normalization parameters for corresponding multiplication, addition, and subtraction operations using an IP core.

[0065] For example, the BN (Batch Normalization) layer of a neural network is calculated as:

[0066] (1)

[0067] Among them, mean, Var, gamma, and beta are four parameters saved after training. These four parameters are integrated into mean, , beta, and make them into COE files and store them in the on-chip memory, using distributed RAM. x is the input data and y is the standardized output data.

[0068] In some exemplary embodiments, the activation function module 124 performs a normalization operation on the data including: removing the linear correlation of the data and improving the generalization ability of the network.

[0069] For example, the activation function module 124 uses a nonlinear activation function ReLU function (full name is Rectified Linear Unit, Chinese name is linear rectification function, which is a commonly used activation function in neural networks) to operate on the normalized data. Optionally, when the input data is negative, the output is 0, otherwise, the output data is the corresponding input data.

[0070] In some exemplary embodiments, the pooling module 125 uses the concatenated input feature map cache module 110 to complete the maximum pooling operation process. The maximum pooling can extract the maximum data of a specific window and reduce the data dimension.

[0071] Figure 7 The diagram schematically shows a maximum pooling process according to an embodiment of the present invention.

[0072] like Figure 7As shown, the maximum pooling process according to the embodiment of the present invention specifically includes setting 4 cache areas S501: P1, P2, P3, P4 that can cache 4 columns of input feature map data in each layer of the input feature map cache module 110; storing 2 consecutive columns of input feature map data in cache areas P1 and P2 in sequence. According to the pooling operation S502, the data D1 and D2 in the cache areas P1 and P2 are read in parallel, and the maximum value O1 of the data read from the cache areas P1 and P2 in each clock cycle is compared, and the maximum values ​​of O1 and O2 in every 2 clock cycles are compared as the final output data Pout, and written to the cache area P3 at the same time; then write to the cache area P4, and then write to the cache areas P1 and P2, and read the cache areas P3 and P4 in parallel at the same time... until all feature maps are traversed.

[0073] Figure 8 The schematic diagram of the upsampling S601 process of the neural network forward prediction hardware accelerator based on a programmable logic device according to an embodiment of the present invention is schematically shown.

[0074] like Figure 8 As shown, the upsampling S601 process of the neural network forward prediction hardware accelerator based on programmable logic devices according to an embodiment of the present invention increases the data dimension through the nearest neighbor interpolation algorithm, specifically including: storing the input data in the input feature map S201 cache module 110 of each layer; when reading out, reading out the same address data every 2 clock cycles, and reading out the column cache area of ​​the same input data every two columns. Optionally, according to the number of channels C of each layer of the input feature map S201, select C circuits for parallel operation.

[0075] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A neural network forward prediction hardware accelerator based on programmable logic devices, characterized in that: include: Multi-layer hardware accelerator module unit, Wherein, the hardware accelerator module unit includes an input feature map cache module and a data processing module; The accelerator module units are connected via a pipeline structure; Multi-channel feature maps can be processed in parallel; and The output data of the hardware accelerator module unit of each layer is stored in the input feature map cache module of the hardware accelerator module unit of the next layer for use by the next layer of operations.

2. The neural network forward prediction hardware accelerator based on programmable logic devices according to claim 1, characterized in that: The input feature map cache module is used to perform read and write operations on the input feature map; The data processing module includes a weight parameter cache module, a systolic array convolution calculation module, a normalization module, an activation function module and a pooling module. Wherein, the weight parameter cache module is an on-chip storage unit for storing convolution weights and bias parameters; The systolic array convolution calculation module is used to calculate convolution operations using a systolic array; The standardization module is used to perform normalization operation on the data; The activation function module is used to operate the normalized data using a nonlinear activation function; and The pooling module is used to complete the maximum pooling operation using the spliced ​​input feature map cache module to reduce the data dimension.

3. The neural network forward prediction hardware accelerator based on programmable logic devices according to claim 2, characterized in that: The process of the input feature map cache module performing read and write operations on the input feature map specifically includes: Dividing the input feature map cache module into a plurality of sub-cache areas; The input feature map is stored in the corresponding sub-buffer area according to rows and columns; Select sub-buffer areas corresponding to several consecutive columns, read out data from the sub-buffer areas corresponding to different rows in parallel, and write data into the sub-buffer area corresponding to the last free column; and Loop through the input feature map to select consecutive columns of data and perform the above read and write operations; The data of the sub-cache areas corresponding to the different rows are used to be input into the systolic array convolution calculation module for convolution calculation.

4. The neural network forward prediction hardware accelerator based on programmable logic devices according to claim 3, characterized in that: The size of the sub-buffer area is related to the size of the convolution kernel, wherein the number of rows of the buffer area is the same as the number of rows of the convolution kernel; and the number of columns of the buffer area is one more than the number of columns of the convolution kernel; and The timing of the read column data differs by one clock cycle.

5. The neural network forward prediction hardware accelerator based on programmable logic devices according to claim 2, characterized in that: The process of storing the weight and bias parameters of the convolution in the weight parameter cache module specifically includes: The weight parameters and bias parameters related to the convolution calculation are stored in the on-chip storage unit, wherein for a convolution kernel of size N×N, N columns of weight parameters are stored in N on-chip storage units respectively; For a convolution kernel of size N×N, data W1, W2, W3, ... WN in N on-chip storage units are read out in parallel, where the readout timing of W1, W2, W3, ... WN differs by one clock cycle; The N parameters stored in each on-chip storage unit are read cyclically, where N is a positive integer.

6. The neural network forward prediction hardware accelerator based on programmable logic devices according to claim 2, characterized in that: The process of calculating the convolution operation by the systolic array convolution calculation module specifically includes: Using each computing unit in the systolic array to perform a multiplication operation on the input feature map data input at the current moment and the weight parameter; The calculation results of the calculation units at different times and positions are accumulated to obtain the convolution result of the pixel.

7. The neural network forward prediction hardware accelerator based on programmable logic devices according to claim 2, characterized in that: The process of the standardization module performing normalization operation on the data specifically includes: concatenating the three standardized parameters into one parameter and storing it in an on-chip memory unit; and The normalized parameters stored in the on-chip storage unit are read out and split into three parallel normalized parameters for multiplication, addition and subtraction operations.

8. The neural network forward prediction hardware accelerator based on programmable logic devices according to claim 2, characterized in that: The activation function module performs normalization operations on the data including: removing the linear correlation of the data and improving the generalization ability of the network.

9. The neural network forward prediction hardware accelerator based on programmable logic devices according to claim 2, characterized in that: The pooling module uses the spliced ​​input feature map cache module to complete the maximum pooling operation process, which specifically includes: The input feature map cache module is provided with 4 sub-cache areas P1, P2, P3, and P4 for caching 4 columns of input feature map data in each layer; Store two consecutive columns of input feature map data in sub-buffer areas P1 and P2 in sequence; Read the data in sub-cache areas P1 and P2 in parallel and write them into sub-cache area P3; Calculate the maximum value of the four data in every two clock cycles as output and write it into the sub-cache area P4; Next, write into the sub-buffer areas P1 and P2, and read in parallel the sub-buffer areas P3, P4, ... until all feature maps are traversed.

10. The neural network forward prediction hardware accelerator based on programmable logic devices according to claim 1, characterized in that: The neural network forward prediction hardware accelerator increases the data dimension by using the nearest neighbor interpolation algorithm, specifically including: Store the input data in the input feature map cache module of each layer; When reading, the same address data is read out every 2 clock cycles, and the column cache area of ​​the same input data is read out every two columns.

Images