A neural network accelerator based on FPGA for CNN_LSTM algorithm

Abstract

This invention claims protection for a CNN-LSTM algorithm neural network accelerator based on FPGA. The CNN hardware implementation includes a data input line buffer module, a convolution calculation module, a ReLU activation function module, an intermediate result buffer module, and a pooling calculation module. The LSTM hardware implementation includes an LSTM control module, a gate function calculation module, and a sigmoid activation function linear approximation module. The FC hardware implementation includes an FC control module, a fully connected layer calculation module, a ReLU activation function module, and a data output buffer. The purpose of this invention is to design a high-performance, low-power, and highly flexible CNN-LSTM neural network accelerator tailored to specific application scenarios. The innovation lies in the fact that, compared to traditional neural network accelerators, this invention uses a parallel pipelined design method to implement a CNN-LSTM algorithm neural network accelerator, which significantly improves the low power consumption and data throughput of the neural network accelerator. Furthermore, the parallel processing capabilities of the FPGA enable the algorithm to run at a faster speed.

Description

Technical Field

[0001] This invention belongs to the field of computer hardware acceleration technology, specifically involving the research on a parallel pipeline design method for a neural network accelerator based on the CNN_LSTM algorithm of FPGA. Background Technology

[0002] Currently, the rapid development of deep neural networks (DNNs) has yielded significant results in many artificial intelligence applications. Convolutional neural networks (CNNs) have made substantial contributions to object detection and image classification. Recurrent neural networks (RNNs) focus on time series problems and have shown great advantages in speech recognition and data prediction. With the diversification and complexity of application scenarios, research on time series prediction methods has deepened, revealing limitations of purely time series analysis methods. In recent years, a hybrid CNN-RNN deep learning model has emerged, integrating CNNs and RNNs into a single network framework. In these models, the CNN extracts network features, which are then fed into the RNN to establish sequence relationships. This is widely used in image captioning, stock trend prediction, lithium battery state of charge prediction, and video detection.

[0003] Currently, implementing DNNs on FPGAs has become a hot research direction. The unique architecture of FPGAs makes it possible to design high-performance, low-power deep learning accelerators. Research on CNN-RNN hybrid deep neural network accelerators is receiving increasing attention. Compared with traditional RNN networks, LSTM uses LSTM units to replace hidden layer nodes, thus solving the long-term dependency problem in traditional RNNs. CNNs have the ability to extract salient features from input data, while LSTMs have long short-term memory features. Therefore, the CNN-LSTM hybrid model can establish correlations for input data in both spatial and temporal dimensions. Based on the network characteristics of CNNs and LSTMs, a neural network prediction model based on the CNN-LSTM algorithm is established. Compared with CNNs and LSTMs alone, the prediction accuracy of this model is significantly improved. Since hardware accelerators for hybrid neural networks are relatively rare, this invention implements a CNN_LSTM neural network accelerator by cascading CNN convolutional neural network hardware and LSTM long short-term memory neural network hardware, and implements the CNN_LSTM algorithm hardware accelerator through FPGA.

[0004] Computational tasks can generally be categorized into software tasks and hardware tasks based on their form. Common hardware acceleration technologies include Application-Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), and Graphics Processing Units (GPUs). ASICs are integrated circuit chips designed and developed for specific applications, characterized by high performance, low power consumption, and small size. FPGAs simplify logic task design, have shorter design cycles than ASICs, and offer reconfigurability, providing flexibility and allowing users to dynamically reconfigure functional modules. GPUs are suitable for parallel computing of large amounts of data, featuring high bandwidth, high clock speed, and high parallelism. However, the high power consumption of GPUs limits their applications. Compared to GPUs, FPGAs have a significant advantage in terms of power consumption.

[0005] CN112949845A discloses an accelerator for deep convolutional neural networks based on FPGA. The accelerator includes a fast convolution operation module and a two-dimensional convolution memory interaction module. The fast convolution operation module combines lightweight fast multiplication with the Winograd algorithm to achieve fast convolution operations for deep convolutional neural networks. The two-dimensional convolution memory interaction module, during the fast convolution operation, uses a trapezoidal reuse memory interaction optimization strategy to perform data transmission between on-chip and off-chip operations for intermediate calculation results and weights, minimizing data interaction between on-chip and off-chip environments. This invention's accelerator improves the Winograd convolution algorithm with fast multiplication, further increasing the speed of Winograd convolution calculation. Furthermore, it proposes a trapezoidal reuse memory interaction optimization strategy, which reduces the latency of on-chip and off-chip memory interaction. Summary of the Invention

[0006] This invention aims to solve the problems of the prior art. It proposes a neural network accelerator based on FPGA for the CNN_LSTM algorithm. The accelerator includes a CNN convolution operation module and an LSTM computation module. The CNN convolution operation module is used to implement a two-dimensional convolutional neural network in hardware using a multiplier and adder tree structure to achieve the convolution operation of the CNN convolutional neural network. The LSTM computation module is used to implement an LSTM long short-term memory neural network in hardware using a multiply-accumulate structure and a linear approximation of the sigmoid activation function to achieve the operation of the LSTM neural network. The accelerator proposed in this invention improves the accelerator's throughput and reduces its power consumption through both hardware implementation of the CNN_LSTM algorithm and parallel pipelined data flow design.

[0007] The technical solution of the present invention is as follows:

[0008] A neural network accelerator based on FPGA for the CNN_LSTM algorithm, comprising:

[0009] CPU: Used to send input data to the input buffer module via the data transmission bus, and to send weights and bias parameters to the parameter RAM module;

[0010] Bus: Includes data bus and control bus. The data bus is responsible for data transmission between the CPU and FPGA; the control bus is responsible for the transmission of control signals and instructions between the CPU and FPGA.

[0011] Input / output buffer: Used to buffer the data to be processed from the CPU to the FPGA and the data to be processed from the FPGA to the CPU via the data bus;

[0012] The CNN convolution operation module is responsible for performing convolution, activation, and pooling logical operations on the data input from the input buffer module and the data from the CNN weight bias RAM.

[0013] LSTM operation module: responsible for performing multiplication and accumulation logical operations on the data output by the CNN convolution operation module and the data in the LSTM weight bias RAM;

[0014] FC (Functional Control) module: Responsible for performing multiplication and accumulation logical operations on the data output from the LSTM (Low-Speed ​​Matrix) module and the data in the FC weight bias RAM.

[0015] Furthermore, the CNN convolution operation module includes a data input line cache module, a convolution calculation module, a ReLU activation function module, an intermediate result cache module, and a pooling calculation module. The data input line cache module buffers the data in the input cache into a column the size of the convolution kernel, and inputs it column by column into the convolution calculation module for multiplication and accumulation calculation. The convolution calculation module extracts various features from the data. The convolution operation can be viewed as consisting of several multiplication and accumulation calculations, and the result of the convolution is input into the activation function module. The ReLU activation function module increases the nonlinear relationship between the layers of the neural network. The ReLU activation function is implemented by a simple comparator. The result of the activation function is input into the intermediate result cache module for the next layer's convolution calculation. The intermediate result cache module stores the intermediate calculation results of the accelerator. The cache's role is to store the data calculated by the previous layer. When the next layer of the neural network requests this data again, it directly retrieves it from the cache, omitting the data processing flow, thereby reducing resource consumption and improving response speed. The pooling calculation module further compresses the output feature data of the convolution layer, removes redundant feature information, prevents overfitting of the network, and makes the CNN model more fault-tolerant.

[0016] Furthermore, the LSTM operation module includes an LSTM control module, a gate function calculation module, and a Sigmoid activation function linear approximation module. The LSTM control module is used to control the reading of weight biases and the transmission of data stream in the LSTM module. The gate function calculation module is used to implement the multiply-accumulate operations in the LSTM module and calculate the results of each gate. The Sigmoid activation function linear approximation module is used to implement the hardware approximation of the Sigmoid activation function. By performing piecewise linear approximation of the Sigmoid activation function, nonlinear factors are added to the neural network model, thereby improving the neural network's expressive power.

[0017] Furthermore, the FC operation module includes an FC control module, a fully connected layer calculation module, a ReLU activation function module, and a data output buffer. The FC control module is used to control the reading of the weight bias and the transmission of the data stream in the FC module. The fully connected layer calculation module is used to perform vector inner product calculation of the fully connected layer through traditional multiply-accumulate operations. The ReLU activation function module is used to increase the nonlinear relationship between the layers of the neural network. The data output buffer is used to temporarily store the data sent to the CPU after the accelerator has completed the calculation.

[0018] Furthermore, a convolutional layer in a CNN's convolutional operation module contains several feature planes, each composed of neurons arranged in a rectangular pattern. Neurons on the same feature plane share weights, which are the convolutional kernels. The convolutional kernels are initialized in the form of random fractional matrices, and during network training, they learn appropriate weights. The convolutional layer is the core layer for building a convolutional neural network, generating most of the computational load in the network.

[0019]

[0020] in It is the output of the convolution calculation result through the activation function f. It represents the weight of the i-th row and j-th column of the l-th convolutional layer. It is input. is the bias term of the convolution, and f is the ReLU activation function;

[0021] Subsampling, also known as pooling, has two forms: mean subsampling and maximum subsampling.

[0022] f(x) = max(x1...x2) n )

[0023] Where f(x) is the result of max pooling calculation, x1...x n It is the input to the pooling layer, and max is the max pooling function.

[0024] Furthermore, to achieve a fully parallel design for convolution computation and realize the high performance and high concurrency of the accelerator, an input / output buffer module was designed using shift registers. The input buffer module pins include a clock signal clk, a reset signal rst_n, a data start receiving valid signal i_vaild, one 14-bit data input data_in, six 14-bit data outputs data_output, and an output valid signal o_vaild. When the system CPU sends a chip_en instruction to the neural network accelerator, the input buffer module starts receiving data, and the i_vaild signal becomes valid. When the input buffer is full, it starts passing data to the convolution operation module. At this time, i_vaild becomes invalid, o_vaild becomes valid, and the buffered data is output sequentially to the first convolution operation module for convolution operation.

[0025] Furthermore, the CNN convolution operation module includes a first CNN convolution operation module and a second CNN convolution operation module. The first CNN convolution operation module has pins including a clock signal clk, a reset signal rst_n, a convolution enable signal en_cnn1, six 14-bit data inputs data_input, 32 14-bit data outputs conv_result, and a convolution end signal end_cnn1. When the input buffer o_vaild is valid, the enable signal en_cnn1 for the first convolution operation becomes valid. After the multiplication and accumulation operations of the first convolution layer are completed, the result is output in parallel to the first ReLU activation function module. The ReLU activation module compares the input data with 0; if it is less than 0, the result is saved as 0 and output; if it is greater than 0, the input value is output. The second CNN convolution operation module... The NN convolution operation module pins include a clock signal clk, a reset signal rst_n, a convolution enable signal en_cnn2, 64 14-bit data inputs data_input, 32 14-bit data outputs conv_result, and a convolution end signal end_cnn2. When the intermediate buffer o_vald is valid, the enable signal en_cnn2 for the second convolution operation becomes valid. After the multiplication and accumulation operation of the second convolution layer is completed, the result is output in parallel to the second ReLU activation function module. The data processed by the second ReLU activation function is then input to the max pooling module through the intermediate buffer module for pooling operation. The pooling window size is 3×1. The max pooling operation module consists of two comparators and one register, which compare three input values ​​to obtain the maximum value.

[0026] Furthermore, the LSTM (Long Short-Term Memory) neural network computation module pins include a clock signal clk, a reset signal rst_n, an enable signal en, a data update signal update, a 14-bit data input x_in, a 448-bit data output h, and a computation end signal lstm_done. The LSTM computation module consists of a memory module, a multiplexer, a multiply-accumulate module, and an activation function approximation module. The memory module stores the weight bias parameters required for LSTM multiply-accumulate computation. The multiplexer selects which gate function's weight bias coefficients to input into the multiply-accumulate module during gate function computation. The multiply-accumulate module performs the LSTM multiply-accumulate computation. The activation function approximation module implements a hardware approximation of the Sigmoid activation function. By approximating the Sigmoid activation function piecewise linearly, nonlinear factors are added to the neural network model, improving the neural network's expressive power.

[0027] Furthermore, the FC fully connected layer computing module pins include a clock signal clk, a reset signal rst_n, an enable signal en, a 448-bit data input in, a 14-bit data output out, and a computing end signal done. The FC computing module consists of a multiply-accumulate module and an activation function approximation module.

[0028] The advantages and beneficial effects of this invention are as follows:

[0029] The main innovation of this invention is the hardware implementation of the entire CNN_LSTM accelerator and the use of parallel pipelined data stream transmission and computation to improve the accelerator's throughput and reduce power consumption. The layers of the neural network are connected serially, while the computation within each layer is implemented in parallel.

[0030] This invention combines the advantages of FPGA's fast parallel computing, high performance, low power consumption, and high flexibility to design a CNN-LSTM accelerator using a parallel pipeline structure. This design can reduce the power consumption of neural network hardware, significantly improve the computational efficiency of the algorithm, and accelerate the algorithm's operation speed. Attached Figure Description

[0031] Figure 1 This is a structural diagram of a preferred embodiment of the present invention;

[0032] Figure 2 This is a structural diagram of the input line caching module;

[0033] Figure 3 This is a structural diagram of the CNN convolutional computation module;

[0034] Figure 4 This is a structural diagram of the PE calculation unit;

[0035] Figure 5 This is a structural diagram of a PE array computing unit that uses data flow sharing;

[0036] Figure 6 This is a structural diagram of the LSTM and FC modules;

[0037] Figure 7 This is a time-series simulation result of the CNN-LSTM accelerator. Detailed Implementation

[0038] The technical solutions of the present invention will be clearly and thoroughly described below with reference to the accompanying drawings. The described embodiments are merely some embodiments of the present invention.

[0039] The technical solution of the present invention to solve the above-mentioned technical problems is:

[0040] like Figure 1 As shown, this invention provides a neural network accelerator for the CNN_LSTM algorithm based on FPGA, comprising:

[0041] CPU: System control function, used to send input data to the input buffer module through the data transmission bus, and to send weights and bias parameters to the parameter RAM module;

[0042] Bus: The data bus is responsible for data transmission between the CPU and the FPGA; the control bus is responsible for the transmission of control signals and instructions between the CPU and the FPGA.

[0043] Input / output buffer: Responsible for buffering the data to be processed from the CPU to the FPGA and the processed data from the FPGA to the CPU via the data bus;

[0044] CNN Convolution Operation Module: Responsible for performing logical operations such as convolution, activation, and pooling on the data input from the input buffer module and the data from the CNN weight bias RAM;

[0045] LSTM operation module: responsible for performing multiplication and accumulation logical operations on the data output by the CNN convolution operation module and the data in the LSTM weight bias RAM;

[0046] FC (Functional Control) module: Responsible for performing multiplication and accumulation logical operations on the data output from the LSTM (Low-Speed ​​Matrix) module and the data in the FC weight bias RAM;

[0047] This invention aims to propose a parallel pipelined design method for a neural network accelerator based on the CNN_LSTM algorithm using FPGA. By accelerating the complex computational part of the CNN_LSTM neural network prediction process through FPGA hardware processing units, compared with performing the computation in the CPU, FPGA has higher performance and lower power consumption, and can improve the computing speed.

[0048] The technical solution of the present invention is as follows:

[0049] A parallel pipelined design method for a neural network accelerator based on FPGA-based CNN_LSTM algorithm, characterized by the following steps:

[0050] A. Use TensorFlow to build a CNN_LSTM neural network prediction model and train the neural network parameters;

[0051] B. Determine the computational units in the CNN_LSTM neural network prediction model and design the accelerator.

[0052] C. Design the accelerator framework and generate the accelerator IP core;

[0053] D. Bind the physical constraint pins of the FPGA resource board in the Vivado software environment, generate a bitstream file and burn it to the development board, and write the IP core and hardware device driver.

[0054] A fully parallel pipeline design method for an FPGA-based CNN_LSTM neural network accelerator includes the following modules: weight bias storage area, convolutional layer, pooling layer, LSTM layer, and fully connected layer;

[0055] The weight bias storage area is used to store the weight parameters and bias parameters during convolution and LSTM operations;

[0056] The convolutional layer is used to perform convolution operations on the input feature data and the weight parameters of the weight bias storage area, and add the corresponding bias parameters.

[0057] The pooling layer is used to perform max pooling on the convolutional data;

[0058] The LSTM layer is used to store information about the data after convolution and pooling for a long time, and to extract relevant rows of the data in the time dimension for prediction.

[0059] The fully connected layer is used to perform a "classification" operation on the data processed by the LSTM layer, thereby increasing the non-linear expressive power of the model.

[0060] The convolutional layer includes an input line buffer, a PE calculation unit, and an activation function unit;

[0061] The input line buffer is used to buffer the data read from the input buffer channel and then input the data into the PE calculation unit;

[0062] The PE calculation unit is used to perform multiplication and accumulation operations on the data, and then send it to the activation function unit;

[0063] The activation function unit is used to process the data using a linear rectification function and output the result to the next module through the intermediate cache module.

[0064] The LSTM layer includes an input buffer, a gate function calculation unit, a Sigmoid activation function calculation unit, a tanh activation function unit, and an output buffer.

[0065] The LSTM layer input buffer is used to buffer the data output from the pooling layer, and then input the data into the gate function calculation unit;

[0066] The gate function calculation unit includes an input gate calculation module, a forget gate calculation module, a memory gate calculation module, and an output gate calculation module. The data in the LSTM input buffer is input to the gate function calculation unit and simultaneously fed to the input gate calculation module, forget gate calculation module, memory gate calculation module, and output gate calculation module for multiplication and accumulation operations with the LSTM weight bias. The result is output to the Sigmoid activation function calculation unit and the tanh activation function unit.

[0067] The Sigmoid activation function calculation unit is used to add nonlinear factors to the neural network model, improving the neural network's expressive power. The outputs from the input gate calculation module, forget gate calculation module, and output gate calculation module are input into the Sigmoid activation function calculation unit, and the memory gate output is input into the tanh activation function unit. The results are then output to the LSTM output buffer, resulting in a fully connected layer.

[0068] The fully connected layer includes a fully connected layer computation unit and an activation function computation unit;

[0069] The fully connected layer computing unit is used for multiplication and accumulation operations on data;

[0070] The activation function calculation unit adds a ReLU activation function to the fully connected layer to increase nonlinearity.

[0071] Furthermore, the PE calculation unit and the gate function calculation unit include a multiplication unit and an addition unit. The multiplication unit is used to perform multiplication operations on the data read from the input row buffer and the weight buffer respectively. The addition unit is used to accumulate the results of the multiplication and add the bias parameter before passing them to the activation function unit.

[0072] Furthermore, the activation function of the convolutional layer is the ReLU function, and its function expression is:

[0073] f(x) = max(0,x)

[0074] Furthermore, the activation functions of the LSTM layer are the Sigmoid function and the tanh function, with the following function expressions:

[0075]

[0076]

[0077] Furthermore, the convolutional layer is equipped with a pipeline strategy unit, which overlaps different loop level operations within the convolutional layer to form an efficient loop pipeline, and divides the data in the row buffer into blocks according to the array block instruction.

[0078] Furthermore, the CNN-LSTM accelerator connects the input buffer module, CNN convolution calculation module, pooling calculation module, LSTM calculation module, and FC fully connected layer calculation module in series, and makes data flow through each module in the form of a data stream, achieving a parallel pipeline structure.

[0079] like Figure 2 The diagram shows the line buffer structure for convolution input, which is composed of shift registers. When the system starts working, I read data from the input storage and buffer the input data into the data array needed by my computation unit through the shift registers. Then, it is passed to the multiply-accumulate unit of the convolution computation module for operation.

[0080] like Figure 3 The diagram shows the structure of the convolution module. Each time the accelerator module starts computation, this invention reads 15 14-bit arrays from the input buffer and places them into the input line buffer, forming four 2x3 line computation blocks. The first convolutional layer of this invention is set to a size of 2x3x32, with 2x3 kernels, for a total of 32 kernels. Therefore, this invention adopts a data sharing approach, simultaneously inputting data into the 32 convolutional kernels and performing the first convolution calculation with 192 weight values. The multiplied and accumulated data is then added to the bias coefficients corresponding to each convolutional kernel via an adder, and the result is output to the ReLU activation function for calculation. The output of the first convolutional layer is processed through a data buffer module, which has the same structure as the input line buffer, combining the data into an array for the second convolutional layer. The second convolutional layer of this invention is 2x32x32 in size, with 2x32 kernels, for a total of 32 kernels. Through data sharing, the data is broadcast to the 32 2x32 convolutional kernels for the second convolution calculation. The second layer of this invention contains a total of 2048 weight values. When the output of the first layer is input into the second layer convolutional unit, it is multiplied and accumulated with the 2048 weight values. Then, the output of each convolutional kernel after completing the multiplication and accumulation calculation is added to the bias coefficient by an adder and then input into the ReLU module for activation function calculation.

[0081] The data after convolution calculation is passed through a data buffer module and arranged into a 3x32 data array. The pooling layer of this invention is a 3x1 max pooling structure, consisting of two comparators and a register. At each time step, the max pooling layer simultaneously inputs three data points for comparison and outputs the maximum value. After 32 clock cycles, it outputs 32x1 results as the result of one convolution calculation, which are then used as input for subsequent LSTM calculations.

[0082] like Figure 6 The diagram shows the structure of the LSTM and FC layers. Xt represents the input of the CNN layer's computational output to the LSTM. The memory stores the weight coefficients required for the LSTM gate computations. There are a total of four gates, each 32x32 in this invention, containing weight coefficients for the forward pass and the directional pass, resulting in a total of 8192 weight coefficients. The results from the input gate, forget gate, and output gate are calculated using matrix dot multiplication and then output through the Sigmoid activation function. The memory gate's computational result is output through the Tanh activation function. In this embodiment of the invention, the forward computation hardware processing unit on the FPGA platform employs a piecewise linear approximation to implement the Sigmoid activation function. Since the Sigmoid(x) function is symmetric, when x is less than 0, it is calculated using 1-Sigmoid(-x). When x is greater than 0, this invention approximates the Sigmoid function into six segments: [0, 1), [1, 1.5), [1.5, 2.2), [2.2, 2.6), [2.6, 3.2), and [3.2, 5). When x is less than -5, the function approximates to 0; similarly, when x is greater than 5, the function approximates to 1. The result from the activation function is input through a buffer to a multiply-accumulate module, then to the ht module, then to the FC module, and finally output to the output buffer.

[0083] like Figure 7 As shown, clk and rst_n are the clock input and reset signals of the accelerator system, respectively; en_chip is the accelerator input enable signal; data_in[13:0] is the input data signal; data_out[13:0] is the output data signal; and done is the accelerator computation completion signal. By observing the simulation test platform, it is found that the FPGA-based CNN_LSTM algorithm neural network accelerator has good prediction effect and high computation efficiency.

[0084] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0085] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0086] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0087] The above embodiments should be understood as illustrative only and not as limiting the scope of protection of the present invention. After reading the description of the present invention, those skilled in the art can make various alterations or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.

Claims

1. A neural network accelerator based on FPGA using the CNN_LSTM algorithm, characterized in that, include: CPU: Used to send input data to the input buffer module via the data transmission bus, and to send weights and bias parameters to the parameter RAM module; Bus: Includes data bus and control bus. The data bus is responsible for data transmission between the CPU and FPGA; the control bus is responsible for the transmission of control signals and instructions between the CPU and FPGA. Input / output buffer: Used to buffer the data to be processed from the CPU to the FPGA and the data to be processed from the FPGA to the CPU via the data bus; The CNN convolution operation module is responsible for performing convolution, activation, and pooling logical operations on the data input from the input buffer module and the data from the CNN weight bias RAM. LSTM operation module: responsible for performing multiplication and accumulation logical operations on the data output by the CNN convolution operation module and the data in the LSTM weight bias RAM; FC (Functional Control) module: Responsible for performing multiplication and accumulation logical operations on the data output from the LSTM (Low-Speed ​​Matrix) module and the data in the FC weight bias RAM; To achieve a fully parallel design for convolution computation and to enable high performance and high concurrency of the accelerator, an input / output buffer module was designed using shift registers. The input buffer module pins include a clock signal clk, a reset signal rst_n, a data start signal i_vaild, one 14-bit data input data_in, six 14-bit data outputs data_output, and an output valid signal o_vaild. When the system CPU sends a chip_en instruction to the neural network accelerator, the input buffer module begins receiving data, and the i_vaild signal becomes valid. When the input buffer is full, data begins to be passed to the convolution operation module. At this time, i_vaild becomes invalid, and o_vaild becomes valid. The buffered data is then output sequentially to the first convolution operation module for convolution operation.

2. The neural network accelerator based on FPGA for the CNN_LSTM algorithm according to claim 1, characterized in that, The CNN convolution operation module includes a data input line cache module, a convolution calculation module, a ReLU activation function module, an intermediate result cache module, and a pooling calculation module. The data input line cache module buffers the data in the input cache into a column the size of the convolution kernel, and inputs it column-by-column into the convolution calculation module for multiplication and accumulation calculations. The convolution calculation module extracts various features from the data; the convolution operation consists of several multiplication and accumulation calculations, and the result is input into the activation function module. The ReLU activation function module adds non-linear relationships between layers of the neural network. The ReLU activation function is implemented using a simple comparator, and the result is input into the intermediate result cache module for the next layer's convolution calculation. The intermediate result cache module stores the intermediate calculation results of the accelerator. The cache's function is to store the data calculated in the previous layer, so that when the next layer of the neural network requests this data again, it can be retrieved directly from the cache, omitting the data processing steps, thereby reducing resource consumption and improving response speed. The pooling calculation module further compresses the output feature data of the convolutional layer, removing redundant feature information, preventing overfitting of the network, and giving the CNN model higher fault tolerance.

3. The neural network accelerator based on FPGA for the CNN_LSTM algorithm according to claim 1, characterized in that, The LSTM operation module includes an LSTM control module, a gate function calculation module, and a Sigmoid activation function linear approximation module. The LSTM control module is used to control the reading of weight biases and the transmission of data flow in the LSTM module. The gate function calculation module is used to implement the multiply-accumulate operations in the LSTM module and calculate the results of each gate. The Sigmoid activation function linear approximation module is used to implement the hardware approximation of the Sigmoid activation function. By performing piecewise linear approximation of the Sigmoid activation function, nonlinear factors are added to the neural network model, thereby improving the neural network's expressive power.

4. A neural network accelerator based on FPGA for the CNN_LSTM algorithm according to claim 1, characterized in that, The FC (Full-Connected) computation module includes an FC control module, a fully connected layer computation module, a ReLU activation function module, and a data output buffer. The FC control module is used to control the reading of the weight biases of the FC module and the transmission of the data stream. The fully connected layer computation module is used to perform vector inner product calculation of the fully connected layer through traditional multiply-accumulate operations. The ReLU activation function module is used to increase the nonlinear relationship between the layers of the neural network, and the data output buffer is used to temporarily store the data sent to the CPU after the accelerator has completed the computation.

5. A neural network accelerator based on FPGA for the CNN_LSTM algorithm according to claim 2, characterized in that, In a convolutional layer of a CNN, there are several feature planes. Each feature plane consists of neurons arranged in a rectangular pattern. Neurons in the same feature plane share weights, which are called the convolutional kernel. The convolutional kernel is initialized in the form of a random fractional matrix. During the training process of the network, the convolutional kernel will learn to obtain reasonable weights. The convolutional layer is the core layer for building a convolutional neural network, and it generates most of the computation in the network. in It is the output of the convolution calculation result through the activation function f. It represents the weight of the i-th row and j-th column of the l-th convolutional layer. It is input. is the bias term of the convolution, and f is the ReLU activation function; Subsampling, also known as pooling, has two forms: mean subsampling and maximum subsampling. f1(x)=max(x1...x n ) Where f1(x) is the result of max pooling calculation, x1...x n It is the input to the pooling layer, and max is the max pooling function.

6. A neural network accelerator based on FPGA for the CNN_LSTM algorithm according to claim 1, characterized in that, The CNN convolution operation module includes a first CNN convolution operation module and a second CNN convolution operation module. The first CNN convolution operation module has pins including a clock signal clk, a reset signal rst_n, a convolution enable signal en_cnn1, six 14-bit data inputs data_input, 32 14-bit data outputs conv_result, and a convolution end signal end_cnn1. When the input buffer o_vaild is valid, the enable signal en_cnn1 for the first convolution operation becomes valid. After the multiplication and accumulation operations of the first convolution layer are completed, the result is output in parallel to the first ReLU activation function module. The ReLU activation module compares the input data with 0; if the result is less than 0, it saves the result as 0 and outputs it; otherwise, it outputs the input value. The second CNN convolution operation module... The product operation module pins include a clock signal clk, a reset signal rst_n, a convolution enable signal en_cnn2, 64 14-bit data inputs data_input, 32 14-bit data outputs conv_result, and a convolution end signal end_cnn2. When the intermediate buffer o_vald is valid, the enable signal en_cnn2 for the second convolution operation becomes valid. After the multiplication and accumulation operation of the second convolution layer is completed, the result is output in parallel to the second ReLU activation function module. The data processed by the second ReLU activation function is then input to the max pooling module through the intermediate buffer module for pooling operation. The pooling window size is 3×1. The max pooling operation module consists of two comparators and one register, which compare three input values ​​to obtain the maximum value.

7. A neural network accelerator based on FPGA for the CNN_LSTM algorithm according to claim 3, characterized in that, The LSTM (Long Short-Term Memory) neural network computation module has pins including a clock signal clk, a reset signal rst_n, an enable signal en, a data update signal update, a 14-bit data input x_in, a 448-bit data output h, and a computation end signal lstm_done. The LSTM computation module consists of a memory module, a multiplexer, a multiply-accumulate module, and an activation function approximation module. The memory module stores the weight bias parameters required for LSTM multiply-accumulate computation. The multiplexer selects which gate function's weight bias coefficients to input into the multiply-accumulate module during gate function computation. The multiply-accumulate module performs the LSTM multiply-accumulate computation. The activation function approximation module implements a hardware approximation of the Sigmoid activation function. By approximating the Sigmoid activation function piecewise linearly, nonlinear factors are added to the neural network model, improving the neural network's expressive power.

8. A neural network accelerator based on FPGA for CNN_LSTM algorithm according to claim 3, characterized in that, The FC fully connected layer computing module has a clock signal clk, a reset signal rst_n, an enable signal en, a 448-bit data input in, a 14-bit data output out, and a computing end signal done. The FC computing module consists of a multiply-accumulate module and an activation function approximation module.

Images