A parallel CNN-LSTM architecture implementation system based on FPGA

Abstract

The present invention discloses a parallel CNN-LSTM architecture implementation system based on FPGA, including: a control module, an operation module and a storage module; the control module is used to generate a control data stream, and the control data stream includes an enable signal, a reset signal of different units and a synchronization signal; the operation module includes a parallel CNN-LSTM architecture network, which is used for data calculation, storage, splicing and output; the storage module is implemented based on BRAM on FPGA, and BRAM is divided into four parts, which respectively store network input data, network output data, trained weight data and function values ​​for activation function fitting. The present invention will adopt a modular design concept, innovatively deploy a parallel CNN-LSTM network on an FPGA platform, and design a flexible and universal deep learning implementation framework. The modules can modify parameters according to different tasks and flexibly interconnect, reducing the high amount of calculations brought about by the increase in the depth of the network structure.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and more specifically, to a parallel CNN-LSTM architecture implementation system based on FPGA. Background Art

[0002] Deep learning technology is booming, and more mature deep learning architectures such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs) have been applied to many fields. Different deep learning frameworks have their own advantages and problems they are good at solving. For example, CNN is good at extracting image features and processing these feature maps, and classifying images by training and using the weights in each layer; Long short-term memory (LSTM) neural network, as a special RNN, can effectively solve the gradient vanishing problem in RNN, and can easily learn long-term dependency information. It works better when modeling long sequence problems, but it is not effective in processing related signals. Many current studies have found that CNN can be connected with LSTM, and the advantages of the structures of the two networks can be used to extract relevant information and time series information of the data, thereby improving the data processing effect. This CNN-LSTM network architecture has been applied in many fields such as image recognition and speech recognition.

[0003] However, in the specific implementation process, the existing technology mostly connects CNN and LSTM in series, generally connecting the LSTM network after CNN. This connection method may cause the loss of timing information during the data processing process of the previous CNN, which is not conducive to the subsequent LSTM operation. In addition, as the calculation depth increases, the calculation complexity will increase significantly. This also causes the current CNN-LSTM research to remain at the theoretical verification level and lacks more reliable hardware implementation ideas. Summary of the invention

[0004] In response to at least one defect or improvement need of the prior art, the present invention provides a parallel CNN-LSTM architecture implementation system based on FPGA, adopts a modular design concept, innovatively deploys a parallel CNN-LSTM network on an FPGA platform, and designs a flexible and general deep learning implementation framework. The modules can modify parameters according to different tasks and are flexibly interconnected, reducing the high amount of computation brought about by the increase in the depth of the network structure.

[0005] In order to achieve the above object, the embodiments of the present invention provide the following technical solutions:

[0006] A parallel CNN-LSTM architecture implementation system based on FPGA, including:

[0007] Control module, operation module and storage module;

[0008] The control module is used to generate a control data stream, which includes an enable signal, a reset signal of different units and a synchronization signal;

[0009] The operation module includes a parallel CNN-LSTM architecture network for data calculation, storage, splicing and output;

[0010] The storage module is implemented based on the BRAM on the FPGA, and the BRAM is divided into four parts, which respectively store network input data, network output data, trained weight data, and function values ​​for activation function fitting;

[0011] Wherein, the parallel CNN-LSTM architecture network includes:

[0012] CNN part, LSTM part and output part;

[0013] The CNN part and the LSTM part receive network input data simultaneously, concatenate their output data after completing their respective operations, and obtain network output data through the output part.

[0014] As a preferred solution of the present invention, the CNN part includes several convolutional layers, activation function layers and pooling layers;

[0015] The LSTM part includes a plurality of LSTM units; the LSTM unit includes a Tanh function and a Sigmoid function as an activation function layer;

[0016] The output part includes a fully connected layer, an activation function layer and a Softmax layer which are connected in sequence.

[0017] As a preferred solution of the present invention, in the CNN part, BRAM is used to store the convolution area by address, and the convolution area is read in sequence when operating the convolution;

[0018] In each convolution operation, the convolution region is stored in two BRAMs;

[0019] The convolutional layer is Convolution units are used for calculation, and all convolution units work simultaneously;

[0020] in, , , Represents the width, height, and depth of the input data. , , Indicates the width, height, and number of convolution kernels;

[0021] The convolution area refers to the convolution kernel sliding to the right on the feature maps of all channels. In this process, all data in the convolution window are convolved.

[0022] As a preferred solution of the present invention, the pooling layer reads data from the output register of the convolutional layer to generate The pooling units are pooled in parallel, and finally the pooling results are written into the output register of this layer;

[0023] in, , , Respectively represent the width, height, and depth of the input data. Indicates the size of the matrix region selected during pooling.

[0024] As a preferred solution of the present invention, each LSTM unit input includes the output of the previous unit as well as , the input of the entire network , corresponding to the weights and biases of the four gates , and bias; bias represents bias, represents the weight corresponding to the network input, Represents the weight corresponding to the output of the previous unit;

[0025] The LSTM unit adopts a four-stage pipeline structure;

[0026] The first stage pipeline includes two matrix splicing modules to and Perform matrix splicing, and Perform matrix splicing, two 1-4 multiplexers are used to select the input, weight and bias required by different gates, multiply the corresponding items of input and weight and add them to the bias to obtain the data before passing through the Tanh and Sigmoid functions , , and and store it in cache;

[0027] The second stage pipeline takes out the four data of the first stage from the cache, passes them through the corresponding functions respectively, and obtains the output of the four gate data. , , , ;

[0028] The third stage performs the Hadamard product operation and stores the second-stage data in the cache to synchronize the pipeline;

[0029] The fourth level pipeline completes the accumulation operation and divides the data into as well as Output as the current unit.

[0030] As a preferred solution of the present invention, the activation function layer of the CNN part and the output part is one of a Sigmoid function, a Tanh function or a Relu function.

[0031] As a preferred solution of the present invention, the operation mode of the Sigmoid function is as follows:

[0032] For the input data of the Sigmoid function, two BRAMs are constructed to store positive and negative data respectively. During operation, the positive and negative inputs are first determined to correspond to the two BRAMs, and then the size of the data exponent bit and the binary code '10010' is determined.

[0033] When the positive BRAM part is selected to store, and the data exponent is greater than '10010', the output value of the Sigmoid function is 1; when the negative BRAM part is selected to store, and the data exponent is greater than '10010', the output value of the Sigmoid function is -1;

[0034] When the data exponent bit is less than '10010', the last 15 bits of the 16-bit floating point number of the Sigmoid function input data are input into BRAM as the address code, and BRAM outputs the data corresponding to the address code as the output value of the Sigmoid function; the data exponent bit is 2 to 6 bits of the input signal.

[0035] As a preferred solution of the present invention, the operation method of the Tanh function is as follows:

[0036] For the input data of the Tanh function, two BRAMs are constructed to store the positive and negative data respectively. During the operation, the positive and negative inputs are first determined to correspond to the two BRAMs, and then the size of the data exponent bit and the binary code '10010' is determined;

[0037] When the BRAM part for storing positive numbers is selected and the data exponent bit is greater than '10010', the output value of the Tanh function is 1;

[0038] When the BRAM part storing positive numbers is selected, and the data exponent bit is less than '10010', the last 15 bits of the 16-bit floating point number of the input data of the Tanh function are input into the BRAM as the address code, and the BRAM outputs the data corresponding to the address code, and the first bit of the binary code in the data output by the BRAM is replaced with 0 as the output value of the Tanh function;

[0039] When the BRAM part for storing negative numbers is selected and the data exponent is greater than '10010', the output value of the Tanh function is -1;

[0040] When the BRAM part storing negative numbers is selected, and the data exponent bit is less than '10010', the last 15 bits of the 16-bit floating point number of the input data of the Tanh function are input into the BRAM as the address code, and the BRAM outputs the data corresponding to the address code as the output value of the Tanh function;

[0041] The data index bits are bits 2 to 6 of the input signal.

[0042] As a preferred solution of the present invention, the operation mode of the Relu function is as follows:

[0043] When the input data of the Relu function is less than 0, the output of the Relu function is 0; when the input data of the Relu function is greater than 0, the output of the Relu function is the same as the input.

[0044] As a preferred solution of the present invention, the Softmax layer is expanded through a 7-term Taylor series, and the float16 corresponding data of the 7 coefficients used are sequentially stored in the coefficient register. After each calculation is completed, the register is shifted right by 16 bits. When the register is full of 0s, the calculation is completed. Two buffers are used to store the cumulative multiplication results to obtain the coefficient of the x exponential term, and the cumulative results are used to obtain the fitting results. After the operation is completed, the data in the cumulative buffer is output, that is, the exponential function corresponding to the input value is obtained.

[0045] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0046] (1) The FPGA-based parallel CNN-LSTM architecture implementation system provided by the present invention adopts a modular design concept and innovatively deploys the parallel CNN-LSTM network on the FPGA platform. A flexible and universal deep learning implementation framework is designed. The parameters of each module can be modified according to different tasks and flexibly interconnected, which reduces the high amount of computation caused by the increase in the depth of the network structure and avoids the loss of timing features that may be caused when CNN processes data. At the same time, the parallel network structure is more conducive to FPGA implementation.

[0047] (2) The FPGA-based parallel CNN-LSTM architecture implementation system provided by the present invention innovatively designs a multi-channel high-parallelism convolution layer and pooling layer implementation method, efficiently utilizes the FPGA on-chip storage resources, and effectively improves the computing speed.

[0048] (3) The FPGA-based parallel CNN-LSTM architecture implementation system provided by the present invention innovatively designs a 16-bit floating-point activation function fitting method based on a lookup table, which effectively overcomes the high computational complexity and high resource consumption problems of the traditional polynomial fitting method, while improving the computing speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0050] Figure 1 A diagram of the parallel CNN-LSTM network architecture provided for an embodiment of the present invention.

[0051] Figure 2 A framework diagram of the FPGA-based parallel CNN-LSTM implementation system provided in an embodiment of the present invention.

[0052] Figure 3 A schematic diagram of the convolutional layer design provided in an embodiment of the present invention.

[0053] Figure 4 A schematic diagram of a convolution unit according to an embodiment of the present invention.

[0054] Figure 5 A fully parallel pooling design diagram provided for an embodiment of the present invention.

[0055] Figure 6 This is an average pooling design diagram provided by an embodiment of the present invention.

[0056] Figure 7 LSTM Cell design diagram provided for an embodiment of the present invention.

[0057] Figure 8 The overall design architecture diagram of LSTM provided for the embodiment of the present invention.

[0058] Fig. 9 A schematic diagram of an LSTM implementation method based on a four-stage pipeline structure provided in an embodiment of the present invention.

[0059] Fig.10 A flowchart of a Sigmoid function implementation based on a lookup table provided in an embodiment of the present invention.

[0060] Fig.11 A flow chart of the Tanh function implementation based on a lookup table provided in an embodiment of the present invention.

[0061] Fig.12 A schematic diagram of an exponential function fitting method provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0063] The terms "first", "second", "third", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0064] As the first embodiment of this embodiment, a parallel CNN-LSTM architecture implementation system based on FPGA is provided. This embodiment deploys a parallel CNN-LSTM network on FPGA, so the parallel CNN-LSTM network of this embodiment is first introduced.

[0065] Figure 1 The parallel CNN-LSTM network architecture diagram of this embodiment is shown, including the CNN part ( Figure 1 Middle right), LSTM part ( Figure 1 center left) and the output section ( Figure 1 The CNN part and the LSTM part receive network input data at the same time, and concatenate their output data after completing their respective operations, and obtain network output data after passing through the fully connected layer, activation function layer and Softmax layer. Among them, the CNN part includes several convolutional layers, activation functions and a pooling layer, and the LSTM part includes several LSTM units.

[0066] Figure 2 The system structure diagram of the parallel CNN-LSTM network of this embodiment when deployed on FPGA is shown, including a control module, an operation module and a storage module. The function of the control module is to generate a control data stream, which includes an enable signal, a reset signal of different units and a synchronization signal. Among them, the synchronization signal calculates the running time of different units to ensure that the units complete the data operation of this unit when exchanging data between them. The control data stream is transmitted between the two modules to ensure the normal operation of the entire architecture.

[0067] The operation module is responsible for the calculation, storage, splicing and output of data. The operation module transmits the operation data stream, including the address information of BRAM and the calculation results of each unit. In order to improve the calculation speed and resource utilization, the storage module in this embodiment is implemented based on BRAM on FPGA. BRAM is divided into four parts, which respectively store network input data, network output data, trained weight data and function values ​​for activation function fitting. During the calculation process, BARM outputs the data to the buffer, and the module calls the data in the buffer for calculation.

[0068] For the convolutional layer in the CNN part, such as Figure 3 As shown in Figure 1, this is the implementation principle of the convolutional layer. , , Represents the width, height, and depth (number of channels) of the input data. , , Represents the width, height, and number of convolution kernels, and the input data size is Description, the convolution kernel size is given by Indicates. The data in the convolution layer is stored as 16-bit floating point numbers. When the feature map of each channel is convolved, the convolution window needs to slide to the right on the feature map. , swipe down times, a total of single channel convolution operation.

[0069] In this embodiment, the convolution kernel slides to the right on the feature maps of all channels. In this process, all data in the convolution window are defined as "convolution area". This embodiment uses BRAM to store convolution areas by address and reads them in sequence when operating convolution. In order to prevent insufficient BRAM storage space when the amount of data is too large, the convolution area is stored in two BRAMs respectively. In order to improve the running speed, this embodiment chooses the strategy of exchanging area for speed and generates Convolutional units (CUs) are used to compute the data stored in the two convolutional regions. All these convolutional units work simultaneously to achieve highly parallel convolution of multi-channel feature maps.

[0070] The operation principle of the convolution module is as follows Figure 4 As shown in the figure, the bottom layer of this module is a common multiplier and adder. After all convolution units are completed, the output is stored in BRAM to form the convolution result of a single convolution kernel on a single input data. The number of clock signals required to implement the network convolution operation with current parameters under this model is ,in Indicates the number of clock signals required for a convolution unit to complete the convolution operation, which is determined by the size of the convolution kernel.

[0071] When designing the convolution layer, this embodiment takes into account the high depth of the convolution layer and the large computational complexity, and adopts a high-parallelism multi-channel convolution design idea. During convolution, the BRAM on the FPGA chip is used in advance to sequentially extract the multi-channel input data used for the convolution operation, that is, the convolution area, and multiple convolution units are generated in parallel to simultaneously calculate the data in the convolution area during convolution. This high-parallelism convolution implementation method can effectively reduce the calculation time.

[0072] The pooling layer calculates the local area of ​​the feature map to output a pixel, thereby reducing the size of the feature map. Common pooling methods include average pooling and maximum pooling. Average pooling calculates the average value of the local field as output, while maximum pooling selects the maximum value of the local field as output.

[0073] In this embodiment, the input parameters are set as follows: , use one The rectangular area "scans" the values ​​in the neighborhood of the feature map, calculates the average or maximum value and outputs it to the next layer. Usually the pooling layer is connected after the convolution layer. The pooling layer reads data from the output register of the convolution layer to generate The pooling units are pooled in parallel, and finally the pooling results are written into the output register of this layer. The architecture is as follows Figure 5 shown.

[0074] Taking average pooling as an example, the pooling unit consists of an adder and a multiplier. The hardware structure is as follows: Figure 6 As shown, the adder adopts a serial structure. The result of the previous addition is used as the input of the next adder. After the accumulation is completed, a multiplication (average) operation is performed on the result to obtain the output and write the data into the output register.

[0075] Similar to the design of the convolution layer, this embodiment generates multiple pooling units in parallel for parallel pooling when designing the pooling layer, and the pooling result can be obtained by consuming only one pooling operation time.

[0076] This embodiment adopts the pipeline architecture to design the LSTM computing unit (LSTM Cell) and the overall LSTM network structure based on the LSTM classic architecture. Figure 7 As shown in the figure, assuming that at time t, a Cell calculates the output values ​​of four gates in parallel, and after the forget gate is calculated, it is directly used as input and combined with the state of the previous Cell Multiply, multiply the data of the input gate and the update gate at the same time, and add the two values ​​to calculate the current state of the Cell , the value is stored in the cache, and the next Cell reads it out for calculation, and the value is obtained by the Tanh activation function And read out the calculated output gate data from the cache, and multiply the two to get the output of the current Cell .

[0077] Multiple LSTM cells form an LSTM network, such as Fig. 9 As shown, in two adjacent cells, the latter cell changes the state of the previous cell. Output As a driving signal, it ensures that data is calculated immediately after transmission, saving clock resources to the greatest extent.

[0078] Since LSTM requires the previous module data to be calculated as subsequent input between layers and within layers, parallel design is not applicable. This embodiment is based on the characteristics of LSTM, such as Fig. 9 As shown, the LSTM implementation method based on four-stage pipeline designed in this embodiment is given to improve the hardware operation efficiency. Fig. 9 It can be seen that each LSTM input includes the output of the previous unit as well as , network input, corresponding to the weights and biases of the four gates: , , bias. Where bias means bias, represents the weight corresponding to the network input, Represents the weight corresponding to the output of the previous unit.

[0079] In the first stage of the pipeline, two matrix splicing modules are included to and X, and Matrix splicing is performed for subsequent operations. Two 1-4 multiplexers are used to select the input, weight and bias required by different gates. The corresponding items of input and weight are multiplied and added to the bias to obtain the data before passing through the Tanh and Sigmoid functions ( , , , ) and store it in the cache. The second-level pipeline takes out the four data of the first level from the cache, passes them through the corresponding functions respectively, and obtains the output of the four gate data. , , , The third level mainly completes the Hadamard product operation, because this process requires the output of the current unit The value after the Tanh function, so the second-level data is stored in the cache to synchronize the pipeline, the fourth-level pipeline completes the accumulation operation and divides the data into as well as Output as the current unit.

[0080] In this embodiment, activation functions are needed in many places, such as the activation function layer of the CNN part, the Sigmoid function and Tanh function in the LSTM unit, and the activation function layer of the output part. For the activation function in this embodiment, a 16-bit floating-point activation function fitting based on a lookup table (LUT) is used.

[0081] This embodiment uses a lookup table method to replace the polynomial approximation method, and uses the BRAM resources in the FPGA to store the corresponding activation function values ​​of all possible values ​​of 16-bit floating-point numbers in [-8, 8] according to the characteristics of different activation functions. The two-level mapping from input to address and address to function output is designed. The activation function implementation method only takes one clock cycle to obtain the activation function value, and the number of accurate digits can reach 5 decimal places. The following is the design method of the three types of activation functions used in this embodiment, namely Sigmoid function, Tanh function and Relu function.

[0082] ① Design of Sigmoid function lookup table.

[0083] When the input is greater than 8 or less than -8, this embodiment determines that the output of the Sigmoid function is 1 or 0. When the input range is [-8,8], the data output range is [0,1], and the data on both sides are symmetrical around (0, 0.5). Two BRAMs are constructed to store the positive and negative data respectively. During the operation, the positive and negative inputs are first determined to correspond to the two BRAMs. Then, the size of the data exponent bit (that is, bits 2 to 6 of the input signal) and the binary code '10010' are determined. Then, the last 15 bits of the 16-bit floating point number are input into the BRAM as the address code. The BRAM outputs the data stored corresponding to the address code as the corresponding activation function value. At this time, the time from input to output only consumes one clock cycle used by the BRAM output. The specific implementation process is as follows: Fig.10 As shown, the details are as follows.

[0084] For the input data of the Sigmoid function, two BRAMs are constructed to store the positive and negative data respectively. During the operation, the positive and negative of the input are first determined to correspond to the two BRAMs, and then the size of the data exponent bit and the binary code '10010' is determined.

[0085] When the positive BRAM part is selected to store and the data exponent is greater than '10010', the output value of the Sigmoid function is 1. When the negative BRAM part is selected to store and the data exponent is greater than '10010', the output value of the Sigmoid function is -1.

[0086] When the data exponent bit is less than '10010', the last 15 bits of the 16-bit floating point number of the Sigmoid function input data are input into the BRAM as the address code, and the BRAM outputs the data corresponding to the address code as the output value of the Sigmoid function. The data exponent bit is 2 to 6 bits of the input signal.

[0087] ②Tanh function lookup table design.

[0088] The implementation of the Tanh activation function is similar to the Sigmoid function. The difference is that since the Tanh function is symmetric about the origin, only the positive or negative LUT needs to be constructed. Assuming that the data LUT for the negative part is constructed, when the input is positive, the address code is the same as the negative part. When outputting, only the first bit needs to be replaced with '0'. The implementation method is as follows: Fig.11 The details are as follows.

[0089] For the input data of the Tanh function, two BRAMs are constructed to store the positive and negative data respectively. During the operation, the positive and negative of the input are first determined to correspond to the two BRAMs, and then the size of the data exponent bit and the binary code '10010' is determined.

[0090] When the BRAM portion for storing positive numbers is selected and the data exponent bit is greater than '10010', the output value of the Tanh function is 1.

[0091] When the BRAM part for storing positive numbers is selected, and the data exponent bit is less than '10010', the last 15 bits of the 16-bit floating point number of the Tanh function input data are input into the BRAM as the address code, and the BRAM outputs the data corresponding to the address code. The first bit of the binary code in the data output by the BRAM is replaced with 0 as the output value of the Tanh function.

[0092] When the BRAM part for storing negative numbers is selected and the data exponent is greater than '10010', the output value of the Tanh function is -1.

[0093] When the BRAM part for storing negative numbers is selected and the data exponent is less than '10010', the last 15 bits of the 16-bit floating point number of the Tanh function input data are input into the BRAM as the address code, and the BRAM outputs the data corresponding to the address code as the output value of the Tanh function.

[0094] ③Relu function.

[0095] According to its expression, the Relu activation function does not need to build a special LUT. It only needs to set the part less than 0 to zero and keep the part greater than zero.

[0096] The Softmax function is mainly located in the last layer of the deep learning network and is used to classify data. Its output is related to the number and size of the input, so it is not suitable to design a lookup table. This embodiment takes into account that the network has undergone several rounds of pooling and full connection, and the amount of data is small when it reaches the Softmax layer, so a polynomial fitting method is chosen to implement the Softmax function. The fitting focus of Softmax is how to fit the exponential function. This embodiment selects 7 Taylor series expansions, and the formula is as follows:

[0097]

[0098] The 7 coefficients used, namely 1, 1 / 2, 1 / 3, 1 / 4, 1 / 5, and 1 / 6, are stored in the coefficient register in order. After each calculation, the register is shifted right by 16 bits (corresponding to a 16-bit floating point number). When the register is full of 0s, the calculation is completed. Two buffers are used to store the multiplication results (to obtain the coefficient of the x exponential term) and the accumulation results (to obtain the fitting results). After the calculation is completed, the data in the accumulation buffer is output to obtain the exponential function corresponding to the input value. The implementation structure is as follows: Fig.12 shown.

[0099] The parallel CNN-LSTM implementation architecture based on FPGA in this embodiment has the following characteristics:

[0100] 1. An innovative FPGA-based parallel CNN-LSTM implementation framework is proposed. Compared with the traditional serial CNN-LSTM implementation framework, the present invention reduces the high amount of computation caused by the increase in network structure depth; avoids the loss of timing features that may be caused when CNN processes data; and the parallel network structure is more conducive to FPGA implementation;

[0101] 2. Innovatively designed multi-channel high-parallel convolutional layer and pooling layer implementation methods, which efficiently utilized FPGA on-chip storage resources and effectively improved the computing speed;

[0102] 3. An innovative 16-bit floating-point activation function fitting method based on a lookup table is designed, which effectively overcomes the high computational complexity and high resource consumption problems of the traditional polynomial fitting method, while improving the computing speed.

[0103] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0104] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A parallel CNN-LSTM architecture implementation system based on FPGA, comprising: Control module, operation module and storage module; The control module is used to generate a control data stream, which includes an enable signal, a reset signal of different units and a synchronization signal; The operation module includes a parallel CNN-LSTM architecture network for data calculation, storage, splicing and output; The storage module is implemented based on the BRAM on the FPGA. The BRAM is divided into four parts, which respectively store network input data, network output data, trained weight data, and function values ​​for activation function fitting; the activation function is implemented by fitting a 16-bit floating-point activation function based on a lookup table, and the BRAM resources in the FPGA are used to store the corresponding activation function values ​​of the 16-bit floating-point values ​​in [-8, 8] according to the characteristics of different activation functions; Wherein, the parallel CNN-LSTM architecture network includes: CNN part, LSTM part and output part; The CNN part and the LSTM part simultaneously receive network input data, respectively complete their own operations and splice their own output data, and obtain network output data through the output part; The CNN part includes several convolutional layers, activation function layers and pooling layers; The LSTM part includes a plurality of LSTM units; the LSTM unit includes a Tanh function and a Sigmoid function as an activation function layer; The output part includes a fully connected layer, an activation function layer and a Softmax layer connected in sequence; In the CNN part, BRAM is used to store the convolution area by address, and the convolution area is read in sequence when operating the convolution; In each convolution operation, the convolution region is stored in two BRAMs; The convolutional layer is Convolution units are used for calculation, and all convolution units work simultaneously; in, , Indicates the width and depth of the input data. Represents the width of the convolution kernel; The convolution area refers to the convolution kernel sliding to the right on the feature maps of all channels. In this process, all the data in the convolution window; The pooling layer reads data from the output register of the convolutional layer and generates The pooling units are pooled in parallel, and finally the pooling results are written into the output register of this layer; in, , , Respectively represent the width, height, and depth of the input data. Indicates the size of the matrix area selected during pooling; Each LSTM unit input includes the output of the previous unit as well as , the input of the entire network , corresponding to the weights and biases of the four gates , and bias; bias represents bias, represents the weight corresponding to the network input, Represents the weight corresponding to the output of the previous unit; The LSTM unit adopts a four-stage pipeline structure; The first stage pipeline includes two matrix splicing modules to and Perform matrix splicing, and Perform matrix splicing, two 1-4 multiplexers are used to select the input, weight and bias required by different gates, multiply the corresponding items of input and weight and add them to the bias to obtain the data before passing through the Tanh and Sigmoid functions , , and and store it in cache; The second stage pipeline takes out the four data of the first stage from the cache, passes them through the corresponding functions respectively, and obtains the output of the four gate data. , , , ; The third stage performs the Hadamard product operation and stores the second-stage data in the cache to synchronize the pipeline; The fourth level pipeline completes the accumulation operation and divides the data into as well as Output as the current unit.

2. The FPGA-based parallel CNN-LSTM architecture implementation system according to claim 1, characterized in that: The activation function layer of the CNN part and the output part is one of a Sigmoid function, a Tanh function or a Relu function.

3. The FPGA-based parallel CNN-LSTM architecture implementation system as claimed in claim 2, characterized in that: The Sigmoid function operates as follows: For the input data of the Sigmoid function, two BRAMs are constructed to store positive and negative data respectively. During operation, the positive and negative inputs are first determined to correspond to the two BRAMs, and then the size of the data exponent bit and the binary code '10010' is determined. When the positive BRAM part is selected to store, and the data exponent is greater than '10010', the output value of the Sigmoid function is 1; when the negative BRAM part is selected to store, and the data exponent is greater than '10010', the output value of the Sigmoid function is -1; When the data exponent bit is less than '10010', the last 15 bits of the 16-bit floating point number of the Sigmoid function input data are input into BRAM as the address code, and BRAM outputs the data corresponding to the address code as the output value of the Sigmoid function; the data exponent bit is 2 to 6 bits of the input signal.

4. The FPGA-based parallel CNN-LSTM architecture implementation system as claimed in claim 2, characterized in that: The Tanh function is calculated as follows: For the input data of the Tanh function, two BRAMs are constructed to store the positive and negative data respectively. During the operation, the positive and negative inputs are first determined to correspond to the two BRAMs, and then the size of the data exponent bit and the binary code '10010' is determined; When the positive BRAM portion is selected for storage and the data exponent is greater than '10010', the output value of the Tanh function is 1; When the BRAM part storing positive numbers is selected, and the data exponent bit is less than '10010', the last 15 bits of the 16-bit floating point number of the input data of the Tanh function are input into the BRAM as the address code, and the BRAM outputs the data corresponding to the address code, and the first bit of the binary code in the data output by the BRAM is replaced with 0 as the output value of the Tanh function; When the BRAM part for storing negative numbers is selected and the data exponent is greater than '10010', the output value of the Tanh function is -1; When the BRAM part for storing negative numbers is selected, and the data exponent bit is less than '10010', the last 15 bits of the 16-bit floating point number of the input data of the Tanh function are input into the BRAM as the address code, and the BRAM outputs the data corresponding to the address code as the output value of the Tanh function; The data index bits are bits 2 to 6 of the input signal.

5. The FPGA-based parallel CNN-LSTM architecture implementation system as claimed in claim 2, characterized in that: The operation mode of the Relu function is as follows: When the input data of the Relu function is less than 0, the output of the Relu function is 0; when the input data of the Relu function is greater than 0, the output of the Relu function is the same as the input.

6. The FPGA-based parallel CNN-LSTM architecture implementation system according to claim 1, characterized in that: The Softmax layer is expanded through 7 Taylor series, and the float16 corresponding data of the 7 coefficients used are sequentially stored in the coefficient register. After each calculation is completed, the register is shifted right by 16 bits. When the register is full of 0s, the calculation is completed. Two buffers are used to store the cumulative multiplication results to obtain the coefficient of the x exponential term. The cumulative results are used to obtain the fitting results. After the operation is completed, the data in the cumulative buffer is output, that is, the exponential function corresponding to the input value is obtained.

Images