Hardware accelerator supporting pulse Transform calculation and acceleration method thereof

Abstract

The invention discloses a hardware accelerator supporting pulse Transform calculation and an acceleration method of the hardware accelerator. The accelerator comprises a control module, a PE calculation unit array, a matrix processing array, a full-connection core and a storage module. The PE calculation unit array is used for realizing tasks of pulse convolution calculation, pulse distribution and membrane potential maximum pooling calculation; the matrix processing array is used for realizing an operation task of a pulse matrix based on a bit AND and bit OR operation mode; and the full-connection core is used for realizing a pulse full-connection calculation task. According to the invention, acceleration support of a pulse self-attention mechanism and a Transform architecture is realized through the PE calculation unit array, the matrix processing array and the full-connection kernel, and meanwhile, the calculation energy consumption of the accelerator is reduced by adopting a bit AND and bit OR operation mode, so that the deployment universality of the accelerator is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of neural networks, and in particular, to a hardware accelerator for supporting pulsed Transformer computing and an acceleration method thereof. Background Art

[0002] In traditional deep learning methods, Transformer is a neural network architecture based on the self-attention mechanism, first proposed by Vaswani et al. in 2017, mainly used for natural language processing (NLP) tasks. Different from traditional recurrent neural networks (RNNs) and convolutional neural networks (CNNs), Transformer can more efficiently process long-range dependencies and has become a widely used basic model in modern deep learning. Spiking neural networks (SNNs) are the third generation of neural networks with spiking neurons as computing units, which can simulate the encoding and processing process of human brain information. Different from traditional artificial neural networks (ANNs), SNNs use discrete spike signals to transmit information, and their neurons have memory functions, so they have higher biological realism and real-time processing capabilities. Currently, SNNs have been widely studied and applied to various high-performance computing tasks.

[0003] It is worth noting that existing research has combined spiking neural networks with Transformer. This processing method has the characteristics of low power consumption, high energy efficiency, and the ability to process spatio-temporal information of spiking neural networks, as well as the powerful expressive ability of Transformer. However, the existing Transformer accelerator designs lack sufficient flexibility and cannot support both the pulsed self-attention mechanism and the Transformer architecture at the same time, making it difficult to effectively accelerate the pulsed Transformer computing process. Summary of the Invention

[0004] The present invention provides a hardware accelerator for supporting pulsed Transformer computing and an acceleration method thereof, which are used to solve the technical problem that the existing Transformer accelerator designs lack sufficient flexibility and cannot support both the pulsed self-attention mechanism and the Transformer architecture at the same time, making it difficult to effectively accelerate the pulsed Transformer computing process.

[0005] The present invention provides a hardware accelerator for supporting pulsed Transformer computing, and the hardware accelerator includes: a control module, a PE computing unit array, a matrix processing array, a fully connected core, and a storage module;

[0006] The control module is respectively connected to the PE computing unit array, the matrix processing array, and the fully connected core, and is used to perform operation control on the PE computing unit array, the matrix processing array, and the fully connected core based on the calculation level of the current pulsed Transformer neural network;

[0007] The PE computing unit array is used to implement the tasks of pulsed convolution calculation, pulsed firing, and membrane potential max pooling calculation of the pulsed Transformer neural network;

[0008] The matrix processing array is used to implement the operation task of the pulsed matrix of the pulsed Transformer neural network based on the operation methods of bitwise AND and bitwise OR;

[0009] The fully connected core is used to implement the calculation task of pulsed full connection of the pulsed Transformer neural network;

[0010] The storage module is connected to the control module and is used to store the interaction data between the control module and the off-chip memory.

[0011] Further, the control module includes: a neural network module controller, a PE matrix calculation controller, a convolution calculation controller, a max pooling controller, a fully connected calculation controller, and a storage controller;

[0012] The neural network module controller is respectively connected to the PE matrix calculation controller, the convolution calculation controller, the max pooling controller, and the fully connected calculation controller, and is used to perform scheduling control on the PE matrix calculation controller, the convolution calculation controller, the max pooling controller, and the fully connected calculation controller based on the calculation level of the current pulsed Transformer neural network;

[0013] The neural network module controller is connected to the storage module through the storage controller, and the storage controller is connected to the off-chip memory.

[0014] Further, the storage module includes a bias memory for storing bias data, a weight memory for storing weight data, a pulse memory for storing pulse data, a neuron membrane potential memory for storing neuron membrane potential, and a pulse matrix memory for storing pulse matrices.

[0015] Further, the convolution calculation controller is used to read the required bias data, weight data, and pulse input data;

[0016] The PE calculation unit array is connected to the convolution calculation controller, and is configured to receive the bias data, weight data, and pulse input data sent by the convolution calculation controller; process the bias data, weight data, and pulse input data required for multiple time steps in parallel, calculate the calculation results corresponding to multiple time steps; sequentially input the calculation results into the pulse neurons according to the time step order to update the membrane potential according to the calculation results. If the updated membrane potential is greater than the threshold, a pulse is emitted and the membrane potential is reset at the same time; if the updated membrane potential is not greater than the threshold, no pulse is emitted and the membrane potential is attenuated at the same time, so as to obtain the first pulse matrix and membrane potential data corresponding to multiple time steps.

[0017] Further, a max pooling controller is configured to read the membrane potential data;

[0018] The PE calculation unit array is connected to the max pooling controller, and is configured to receive the membrane potential data sent by the max pooling controller and screen out the maximum membrane potential from the membrane potential data.

[0019] Further, the PE matrix calculation controller is configured to read the first pulse matrix; wherein, the first pulse matrix includes a Q pulse matrix, a K pulse matrix, and a V pulse matrix;

[0020] The matrix processing array is connected to the PE matrix calculation controller, and is configured to receive the first pulse matrix sent by the PE matrix calculation controller; perform bitwise AND operation on the pulse sequences with the width of the sequence length in the Q pulse matrix and the pulse sequences of the K pulse matrix corresponding to the embedding dimension of the Q pulse matrix in parallel until all the pulse sequences corresponding to the embedding dimensions in the Q pulse matrix are calculated, to obtain a bitwise AND matrix; perform bitwise OR operation on each pulse sequence in the bitwise AND matrix in parallel to obtain a bitwise OR result matrix; perform bitwise AND operation on the pulse sequences in the bitwise OR result matrix and the pulse sequences of the V pulse matrix corresponding to the embedding dimension of the bitwise OR result matrix in parallel until all the pulse sequences corresponding to the embedding dimensions in the bitwise OR result matrix are calculated, to obtain a second pulse matrix, and send the second pulse matrix to the off-chip memory.

[0021] Further, the storage controller is connected to the off-chip memory and is configured to read the second pulse matrix from the off-chip memory;

[0022] The PE computing unit array is connected to the storage controller, and is configured to receive the second pulse matrix sent by the storage controller; perform a convolution operation based on the bias data, weight data, and the second pulse matrix to obtain a third pulse matrix; use the third pulse matrix as the new second pulse matrix, and combine the bias data, weight data, and the new second pulse matrix; until the number of iterations of the convolution operation reaches a preset number, send the third pulse matrix obtained in the last iteration to the storage controller, so that the storage controller stores the third pulse matrix in the off-chip memory.

[0023] Further, a fully-connected computing controller is configured to read the required bias data, weight data, and the third pulse matrix;

[0024] The fully-connected core is respectively connected to the fully-connected computing controller and the storage controller, and is configured to receive the bias data, weight data, and the third pulse matrix sent by the fully-connected computing controller, and perform a fully-connected operation based on the bias data, weight data, and the third pulse matrix, and output a fully-connected result.

[0025] Further, the PE computing unit array is composed of multiple PEs for simulating pulse neurons; each PE is composed of four convolution units and one pooling unit;

[0026] The matrix processing array is composed of four multi-bit AND computing units and four multi-bit OR computing units.

[0027] The present invention also provides an acceleration method for a hardware accelerator supporting pulse Transformer computing, and the method includes:

[0028] Implement the tasks of pulse convolution calculation, pulse firing, and membrane potential max pooling calculation of the pulse Transformer neural network through the PE computing unit array;

[0029] Implement the operation task of the pulse matrix of the pulse Transformer neural network through the matrix processing array based on the operation modes of AND and OR;

[0030] Implement the calculation task of pulse fully-connected of the pulse Transformer neural network through the fully-connected core.

[0031] It can be seen from the above technical solutions that the present invention has the following advantages:

[0032] The present invention provides a hardware accelerator for supporting pulsed Transformer calculations and an acceleration method thereof. The accelerator includes: a control module, a PE computing unit array, a matrix processing array, a fully connected core, and a storage module; the control module is respectively connected to the PE computing unit array, the matrix processing array, and the fully connected core, and is used to perform operation control on the PE computing unit array, the matrix processing array, and the fully connected core based on the current calculation level of the pulsed Transformer neural network; the PE computing unit array is used to implement the tasks of pulsed convolution calculation, pulsed firing, and membrane potential max pooling calculation of the pulsed Transformer neural network; the matrix processing array is used to implement the operation task of the pulsed matrix of the pulsed Transformer neural network based on the operation modes of bitwise AND and bitwise OR; the fully connected core is used to implement the calculation task of pulsed full connection of the pulsed Transformer neural network; the storage module is connected to the control module and is used to store the interaction data between the control module and the off-chip memory.

[0033] In the present invention, the pulsed self-attention mechanism and the acceleration support for the Transformer architecture are realized through the PE computing unit array, the matrix processing array, and the fully connected core. At the same time, the operation modes of bitwise AND and bitwise OR are adopted to implement the operations of pulsed matrix multiplication and accumulation in self-attention, so as to reduce the computing energy consumption of the accelerator. The deployment versatility of the accelerator is improved, and further, the technical problem that the existing Transformer accelerator design lacks sufficient flexibility and cannot support both the pulsed self-attention mechanism and the Transformer architecture at the same time, thus making it difficult to effectively accelerate the pulsed Transformer calculation process is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained according to these drawings without creative efforts.

[0035] Figure 1 A connection schematic diagram of a hardware accelerator for supporting pulsed Transformer calculations provided by an embodiment of the present invention;

[0036] Figure 2 A structural block diagram of a hardware accelerator for supporting pulsed Transformer calculations provided by an embodiment of the present invention;

[0037] Figure 3 A calculation flow chart of the pulsed self-attention mechanism provided by an embodiment of the present invention;

[0038] Figure 4 A flowchart showing the steps of an acceleration method for a hardware accelerator supporting pulsed Transformer calculations provided by an embodiment of the present invention. Detailed implementation manners

[0039] An embodiment of the present invention provides a hardware accelerator supporting pulsed Transformer calculations and an acceleration method thereof, which are used to solve the technical problem that the existing Transformer accelerator design lacks sufficient flexibility and cannot support both the pulsed self-attention mechanism and the Transformer architecture at the same time, thus making it difficult to effectively accelerate the pulsed Transformer calculation process.

[0040] To make the objectives, features, and advantages of the present invention more obvious and understandable, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the embodiments described below are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

[0041] Self-attention mechanism: Refers to a mechanism used to model sequence data in deep learning.

[0042] SPS: Spiking Patch Splitting in pulsed Transformer, which divides the input data into multiple small blocks.

[0043] SDSA: Self-attention mechanism driven by pulses in pulsed Transformer.

[0044] MLP: Multilayer perceptron, referring to the feedforward neural network part in pulsed Transformer in this technical solution.

[0045] Classification Head: Classification module in pulsed Transformer, composed of pulsed fully connected calculations, which completes the classification of the self-attention results.

[0046] Pulsed convolution calculation: A convolution calculation method in pulsed neural networks.

[0047] Pulsed fully connected calculation: A fully connected calculation method in pulsed neural networks.

[0048] Please refer to Figure 1 and Figure 2, an embodiment of a hardware accelerator supporting pulsed Transformer calculation provided by the first aspect of the present invention includes: a control module, a PE computing unit array (PE Array), a matrix processing array (MatrixPE), a fully connected core (FC Core), and a storage module;

[0049] The control module is respectively connected to the PE computing unit array, the matrix processing array, and the fully connected core, and is used to perform operation control on the matrix processing array, the PE computing unit array, and the fully connected core based on the calculation level of the current pulsed Transformer neural network;

[0050] The PE computing unit array is used to implement the tasks of pulsed convolution calculation, pulsed firing, and membrane potential max pooling calculation of the pulsed Transformer neural network;

[0051] The matrix processing array is used to implement the operation task of the pulsed matrix of the pulsed Transformer neural network based on the operation modes of bitwise AND and bitwise OR;

[0052] The fully connected core is used to implement the calculation task of pulsed fully connected of the pulsed Transformer neural network;

[0053] The storage module is connected to the control module and is used to store the interaction data between the control module and off-chip memory.

[0054] It should be noted that the existing pulsed Transformer neural network usually includes an SPS module, an SDSA module, an MLP module, and a classification head module. Among them, the SPS (Spiking Patch Splitting) module is responsible for pulsed convolution calculation and fixed-point max pooling calculation; the SDSA module is responsible for pulsed convolution calculation and pulsed matrix operation; the MLP module is responsible for pulsed convolution calculation; the Classification head (classification head) module is responsible for pulsed fully connected calculation.

[0055] Correspondingly, the PE computing unit array provided by the present invention can complete pulsed convolution calculation, pulsed firing, and fixed-point max pooling calculation, the matrix processing array can complete pulsed matrix calculation, and the fully connected core can complete pulsed fully connected calculation. Therefore, the hardware accelerator provided by the present invention completes the calculation acceleration of the SPS module and the MLP module in the pulsed Transformer neural network through the PE computing unit array, the fully connected core completes the calculation acceleration of the Classification head module, and the matrix processing array and the PE computing unit array jointly complete the calculation acceleration of the SDSA module.

[0056] Meanwhile, considering that in existing accelerators, the calculation of the pulsed self-attention mechanism usually adopts matrix dot multiplication, column-by-column accumulation operation, and matrix transformation operation, resulting in the problem that the calculation of the pulsed self-attention mechanism is not easy to be deployed on hardware, and thus it is difficult to achieve hardware acceleration quickly; the present invention uses bitwise AND and bitwise OR operations to replace the operations of pulsed matrix multiplication and accumulation in self-attention, thereby improving the convenience and generality of hardware acceleration.

[0057] Furthermore, the control module includes: a neural network module controller (spikformer controller), a PE matrix calculation controller (Matrix_PE_Controller), a convolution calculation controller (Conv_controller), a max pooling controller (Max_Pooling_Controller), a fully connected calculation controller (FC_Controller), and a storage controller (Memory Controller);

[0058] The neural network module controller is used to control the operation of the entire hardware accelerator architecture; among them, the neural network module controller is respectively connected to the PE matrix calculation controller, the convolution calculation controller, the max pooling controller, and the fully connected calculation controller, and is used to schedule and control the PE matrix calculation controller, the convolution calculation controller, the max pooling controller, and the fully connected calculation controller based on the calculation level of the current pulsed Transformer neural network.

[0059] The neural network module controller is connected to the storage module through the storage controller, and the storage controller is connected to the off-chip memory. Therefore, the neural network module controller can read or store the required data from the storage module or the off-chip memory through the storage controller.

[0060] Furthermore, the storage module includes a bias memory (Bias Memory) for storing bias data, a weight memory (Weight Buffer) for storing weight data, a spike memory (SpikeBuffer) for storing spike data, a neuron membrane potential memory (Neuron State Buffer) for storing neuron membrane potential, and a spike matrix memory (Spike Matrix Buffer) for storing spike matrices.

[0061] It should be noted that during the calculation process, the bias memory can initialize the bias data; the weight memory, the spike memory, and the spike matrix memory usually store the relevant data read from the off-chip memory, while the neuron membrane potential memory is used to store the membrane potential data calculated by the PE calculation unit array.

[0062] Furthermore, the PE computing unit array is composed of multiple PEs for simulating spiking neurons; each PE is composed of four convolutional units (Conv unit) and one pooling unit (Pooling unit); the matrix processing array is composed of four multi-bit AND computing units (MUL Core) and four multi-bit OR computing units (SUM Core).

[0063] It should be noted that the existing hardware acceleration architectures for spiking neural networks are generally not time-step parallel operations, that is, first complete the multiplication of spike data and weights and neuron calculations in one time step, and then start the multiplication of spike data and weights and neuron calculations in the next time step, which greatly prolongs the calculation time of the accelerator.

[0064] Correspondingly, the present invention adopts time-step parallel technology to save the calculation and processing time of the accelerator. For example, the PE computing unit array can be composed of 64 PEs in an 8x8 arrangement. Each PE simulates a spiking neuron and includes 4 convolutional units and one pooling unit, which can complete the functions of weight accumulation, max pooling, and spike firing in convolutional calculations. The 4 convolutional units can respectively execute convolutional operations such as multiplying spike data and weights in 4 time steps, that is, it supports a maximum of 4-time-step parallelism; the AND and OR operations of the matrix processing array also adopt time-step parallelism. It is composed of four multi-bit AND computing units and four multi-bit OR computing units, that is, it can complete the AND and OR calculations of 4 time steps simultaneously.

[0065] In addition, in the present invention, the PE uses the LIF neuron as the spiking neuron. The behavior of the LIF neuron can be briefly described as follows: 1) When receiving a stimulus, the membrane potential accumulates, that is, the result of multiplying the spike data and weights in the current time step is used as the neuron input (receiving a stimulus), and the membrane potential accumulates the result of multiplying the spike data and weights. 2) Judge spike firing. In each time step, the neuron will judge whether the current membrane potential is greater than the threshold. If it is greater, it will fire a spike, that is, output 1, otherwise it will not fire, that is, output 0. 3) Update the membrane potential. If a spike fires in the current time step, the membrane potential will be reset, usually reset to 0. If no spike fires, the membrane potential will decay. In this solution, for the convenience of hardware deployment, the membrane potential decay is to divide the membrane potential by two (shift the memory storing the membrane potential one bit to the right). After the three steps in the current time step are completed, the calculation of the next time step starts, and these three steps are repeated. The result of multiplying the spike data and weights in the next time step is used as the new neuron input.

[0066] Furthermore, the convolutional calculation controller is used to read the required bias data, weight data, and pulse input data;

[0067] The PE computing unit array is connected to the convolution computing controller, and is used to receive the bias data, weight data, and pulse input data sent by the convolution computing controller; parallelly process the bias data, weight data, and pulse input data required for multiple time steps, calculate the calculation results corresponding to multiple time steps; sequentially input the calculation results into the pulse neurons in the order of time steps to update the membrane potential according to the calculation results. If the updated membrane potential is greater than the threshold, a pulse is emitted and the membrane potential is reset at the same time; if the updated membrane potential is not greater than the threshold, no pulse is emitted and the membrane potential is decayed at the same time, so as to obtain the first pulse matrix and membrane potential data corresponding to multiple time steps.

[0068] It can be understood that during convolution calculation, the neural network module controller first reads the required weight and pulse input data from off-chip memory through the storage controller and stores them in the corresponding memory; then, the convolution computing controller reads the weight data from the weight memory, reads the pulse input data from the pulse memory, reads the bias data from the bias memory, and transmits them to the PE computing unit array.

[0069] Next, the PE computing unit array starts convolution calculation, parallelly processes the bias data, weight data, and pulse input data required for multiple time steps, calculates the calculation results corresponding to multiple time steps; sequentially inputs the calculation results into the pulse neurons in the order of time steps to update the membrane potential according to the calculation results. If the updated membrane potential is greater than the threshold, a pulse of 1 is emitted and the membrane potential is reset at the same time; if the updated membrane potential is not greater than the threshold, no pulse is emitted and 0 is output, and the membrane potential is decayed at the same time, so as to obtain the first pulse matrix and membrane potential data corresponding to multiple time steps; finally, the convolution computing controller writes the first pulse matrix into off-chip memory and writes the membrane potential data back to the neuron membrane potential memory.

[0070] Furthermore, the max pooling controller is used to read the membrane potential data; the PE computing unit array is connected to the max pooling controller and is used to receive the membrane potential data sent by the max pooling controller and screen out the maximum membrane potential from the membrane potential data.

[0071] Furthermore, the PE matrix computing controller is used to read the first pulse matrix; among them, the first pulse matrix includes a Q pulse matrix, a K pulse matrix, and a V pulse matrix;

[0072] The matrix processing array is connected to the PE matrix calculation controller and is used to receive the first pulse matrix sent by the PE matrix calculation controller; perform bitwise AND operation on the pulse sequence with the width of the sequence length in the Q pulse matrix and the pulse sequence of the K pulse matrix corresponding to the embedding dimension of the corresponding Q pulse matrix in parallel until all the pulse sequences corresponding to the embedding dimensions in the Q pulse matrix are calculated, obtaining a bitwise AND matrix; perform bitwise OR operation on each pulse sequence in the bitwise AND matrix in parallel, obtaining a bitwise OR result matrix; perform bitwise AND operation on the pulse sequence in the bitwise OR result matrix and the pulse sequence of the V pulse matrix corresponding to the embedding dimension of the corresponding bitwise OR result matrix in parallel until all the pulse sequences corresponding to the embedding dimensions in the bitwise OR result matrix are calculated, obtaining a second pulse matrix.

[0073] Further, the storage controller is connected to the off-chip memory and is used to read the second pulse matrix from the off-chip memory.

[0074] The PE calculation unit array is connected to the storage controller and is used to receive the second pulse matrix sent by the storage controller; perform convolution operation based on the bias data, weight data, and the second pulse matrix, obtaining a third pulse matrix; use the third pulse matrix as the new second pulse matrix and combine the bias data, weight data, and the new second pulse matrix; until the number of iterative convolution operations reaches a preset number, send the third pulse matrix obtained from the last iteration to the storage controller, so that the storage controller stores the third pulse matrix in the off-chip memory.

[0075] Further, the fully connected calculation controller is used to read the required bias data, weight data, and the third pulse matrix.

[0076] The fully connected core is respectively connected to the fully connected calculation controller and the storage controller and is used to receive the bias data, weight data, and the third pulse matrix sent by the fully connected calculation controller and perform a fully connected operation based on the bias data, weight data, and the third pulse matrix, outputting a fully connected result.

[0077] It should be noted that when the PE calculation unit array performs convolution calculation, each PE only needs the weight data of the size of one convolution kernel at a time. When the fully connected core performs fully connected calculation, it needs more weight data. At the same time, after the fully connected core finishes the calculation, it only needs to write the result back to the off-chip memory finally.

[0078] It is worth mentioning that in the calculation of the existing pulse self-attention mechanism, matrix dot multiplication, column-by-column accumulation operation and matrix transformation operation are usually adopted, resulting in relatively large calculation delay and calculation energy consumption of the accelerator. Based on this, the present invention designs a pulse self-attention optimized calculation mechanism. In the design architecture of the hardware accelerator, the existing matrix dot multiplication and accumulation operations are replaced by the operation modes of multi-bit bitwise AND and multi-bit bitwise OR through the matrix processing array, so as to optimize the existing pulse self-attention calculation mechanism, make it easy to be deployed on the hardware, and thus reduce the calculation delay and calculation energy consumption of the accelerator.

[0079] Exemplarily, please refer to Figure 3 , Figure 3 which shows the flowcharts of the traditional self-attention mechanism calculation mechanism, the existing pulse self-attention calculation mechanism and the pulse self-attention optimized calculation mechanism provided by the present invention.

[0080] In Figure 3 , the traditional self-attention mechanism calculation mechanism adopts calculations such as multiplication, division, and square root extraction of real numbers, resulting in relatively large calculation consumption. In the existing pulse self-attention calculation mechanism, mainly operations such as multiplication and accumulation of pulse matrices are performed. However, the elements of the pulse matrix are 0 or 1, and its composition is very sparse, resulting in relatively large calculation consumption. In contrast, the present invention designs a pulse self-attention optimized calculation mechanism, replaces matrix multiplication with bitwise AND operation, and replaces the operation of column-by-column accumulation and inputting into a pulse neuron to detect the threshold for firing with bitwise OR operation. This method greatly reduces the calculation consumption and calculation delay, and at the same time, the bitwise AND and bitwise OR operations can be easily deployed on the hardware.

[0081] Please refer to Figure 3 , and the following further describes the calculation process of the pulse self-attention optimized calculation mechanism provided by the present invention; among them, Figure 3 the linear layer and the pulse neuron in

[0082] 1) First, perform pulse convolution calculation: The neural network module controller reads the pulse input data from off-chip memory, and performs convolution calculation through the PE calculation unit array to obtain three three-dimensional pulse matrices (Q pulse matrix, K pulse matrix, and V pulse matrix). For example, the pulse matrix can be a 4 x 256 x 64 pulse matrix, with 4 time steps, an embedding dimension of 256, and a sequence length of 64; and a sequence length of 64 can be considered as the one-dimensional unfolding of an 8 x 8 two-dimensional image.

[0083] 2) Then, perform a bitwise AND operation on the three three-dimensional pulse matrices output by the PE computing unit array through the matrix processing array: The matrix processing array will perform a bitwise AND operation on a pulse sequence with a sequence length width in the Q matrix and the pulse sequence of the K matrix. For example, both the Q matrix and the K matrix are 4 x 256 x 64 pulse matrices. Each time, a 64-bit pulse sequence in the Q matrix and a 64-bit pulse sequence in the K matrix corresponding to the embedding dimension are subjected to a bitwise AND calculation until all the pulse data in all embedding dimensions of the Q matrix are calculated. For example, a 4 x 256 x 64 pulse matrix will perform 256 64-bit wide bitwise AND calculations to obtain a bitwise AND matrix. It should be noted that during the entire bitwise AND operation process, time step parallelism is adopted. For example, each time a bitwise AND operation is performed, four time steps are calculating simultaneously, that is, for a 4 x 256 x 64 pulse matrix, a total of 4 groups of 64 pulse sequences participate in the bitwise AND operation each time a bitwise AND operation is performed.

[0084] 3) Next, perform a bitwise OR operation on the bitwise AND matrix through the matrix processing array: In the traditional pulse self-attention mechanism, it is necessary to accumulate the results of multiplying the Q matrix and the K matrix column by column and then input them into the pulse neuron. Since the accumulated result is no longer a pulse matrix and the elements in the matrix are integers, not just 0 or 1, it is necessary to first use the elements in the integer matrix as the input of the pulse neuron membrane potential and judge through pulse firing in the neuron, that is, judge whether the value of each element is greater than the threshold. If it is greater, output 1; if it is less, output 0. In this way, each element is binarized, and the integer matrix is converted into a pulse matrix for output. For example, both the Q matrix and the K matrix are 4 x 256 x 64 pulse matrices, and the multiplication result is also a 4 x 256 x 64 pulse matrix. Column-by-column addition requires adding 6 data in a 64-bit pulse sequence with a sequence length of 64 to obtain a 4 x 256 x 1 integer matrix, and then inputting the integer matrix into the neuron to obtain a 4 x 256 x 1 pulse matrix. In practical applications, the threshold of the neuron is usually set to a relatively small value (less than 1). As long as one bit of data in the accumulated sequence of the pulse matrix is 1, the neuron will fire a pulse. Based on this, the present invention directly uses a bitwise OR operation to replace the existing step of accumulating the pulse matrix column by column and inputting it into the neuron. For example, the above bitwise AND result is a 4 x 256 x 64 pulse matrix, and the matrix processing array of the present invention directly performs a bitwise OR operation on the 64-bit pulse sequence. If one bit in the 64 bits is 1, the output result is 1, thereby obtaining a bitwise OR result matrix. It should be noted that during the entire bitwise OR operation process, time step parallelism is adopted. For example, each time a bitwise OR operation is performed, four time steps are calculating simultaneously, that is, for a 4 x 256 x 64 pulse matrix, a total of 4 groups of 64 pulse sequences participate in the bitwise OR operation each time a bitwise OR operation is performed.

[0085] 4) Next, perform a bitwise AND operation on the bitwise OR result matrix and the V matrix through the matrix processing array: It can be understood that the bitwise AND operation here is similar to the bitwise AND operation process between the Q matrix and the K matrix, which will not be elaborated here, and the bitwise AND result obtained from this calculation process will be written into the off-chip memory.

[0086] 5) Finally, input the bitwise AND result into the linear layer: Use the bitwise AND result as the input of the PE computing unit array again. The PE computing unit array performs convolution calculations. After multiple convolution calculations, write the obtained third pulse matrix back to the off-chip memory.

[0087] The present invention provides a hardware accelerator supporting pulsed Transformer calculations, which has the following advantages:

[0088] 1. Build a hardware accelerator supporting pulsed Transformer calculations, design a PE computing unit array, a matrix processing array, and a fully connected core to achieve the acceleration of the pulsed Transformer neural network architecture. At the same time, adopt the self-attention mechanism of pulses to solve the problem of excessive consumption in existing Transformer calculations.

[0089] 2. Design a pulsed self-attention calculation mechanism that is easy to deploy on hardware, that is, use bitwise AND and bitwise OR operations through the matrix processing array to replace the existing multiplication and column-by-column accumulation operations of the pulsed matrix respectively, reducing the computing energy consumption and computing latency, and solving the problem that the existing self-attention mechanism is not easy to be deployed on hardware due to calculations such as multiplication and column-by-column accumulation.

[0090] 3. Propose the Transformer time-step parallel computing technology, and adopt multi-time-step parallelism in the operation processing of the PE computing unit array and the matrix processing array, which greatly reduces the computing latency of the pulsed Transformer, thereby solving the problem of excessive latency caused by multi-time-step operations during the hardware calculation of the Transformer based on the pulsed neural network.

[0091] Please refer to Figure 4 , the second aspect of the present invention also provides an acceleration method for a hardware accelerator supporting pulsed Transformer calculations, and the method includes:

[0092] Step 101, implement the tasks of pulsed convolution calculation, pulsed firing, and membrane potential max pooling calculation of the pulsed Transformer neural network through the PE computing unit array;

[0093] It can be understood that the hardware accelerator can call the PE computing unit array, matrix processing array, and fully connected kernel according to the calculation level of the current pulsed Transformer neural network. Among them, the first pulsed matrix obtained by convolutional calculation through the PE computing unit array is used as the input of the matrix processing array; while the second pulsed matrix obtained after matrix operation through the matrix processing array needs to be input into the PE computing unit array for multiple convolutional calculations to obtain the third pulsed matrix.

[0094] Step 102, implement the operation task of the pulsed matrix of the pulsed Transformer neural network through the matrix processing array based on the bitwise AND and bitwise OR operation methods;

[0095] It should be noted that the matrix processing array uses bitwise AND and bitwise OR operations to respectively replace operations such as pulsed matrix multiplication and column-by-column addition in the pulsed self-attention mechanism calculation of the existing pulsed Transformer, solving the problem that it is not easy to deploy the accelerator to hardware due to calculations such as multiplication and column-by-column accumulation in the existing self-attention mechanism.

[0096] Step 103, implement the calculation task of the pulsed fully connected layer of the pulsed Transformer neural network through the fully connected kernel. It should be noted that the fully connected operation is implemented on the third pulsed matrix through the fully connected kernel, and the fully connected result is output.

[0097] The acceleration method of the hardware accelerator supporting pulsed Transformer calculation provided by the present invention realizes the acceleration support for the pulsed Transformer architecture and self-attention mechanism through the PE computing unit array, matrix processing array, and fully connected kernel. At the same time, bitwise AND and bitwise OR operation methods are used to implement the operations of pulsed matrix multiplication and accumulation in self-attention to reduce the computational energy consumption of the accelerator and improve the deployment versatility of the accelerator.

[0098] Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated here.

[0099] In several embodiments provided in the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there can be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces, and the indirect couplings or communication connections of devices or units can be in electrical, mechanical, or other forms.

[0100] The unit described as a separation component may or may not be physically separated. The component shown as a unit may or may not be a physical unit, that is, it may be located in one place or may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0101] In addition, each functional unit in various embodiments of the present invention may be integrated in a processing unit, may exist separately as individual physical units, or two or more units may be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0102] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0103] As mentioned above, the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of various embodiments of the present invention.

Claims

1. A hardware accelerator supporting pulse Transformer calculation, characterized in that: The hardware accelerator includes: a control module, a PE computing unit array, a matrix processing array, a fully connected core and a storage module; The control module is connected to the PE computing unit array, the matrix processing array and the fully connected core respectively, and is used to perform computational control on the PE computing unit array, the matrix processing array and the fully connected core based on the computational level of the current pulse Transformer neural network; The PE computing unit array is used to implement the tasks of pulse convolution calculation, pulse emission and membrane potential maximum pooling calculation of the pulse Transformer neural network; The matrix processing array is used to implement the pulse matrix operation task of the pulse Transformer neural network based on the bitwise AND and bitwise OR operation modes; The fully connected core is used to implement the pulse fully connected computing task of the pulse Transformer neural network; The storage module is connected to the control module and is used to store the interaction data between the control module and the off-chip memory.

2. The hardware accelerator according to claim 1, characterized in that: The control module includes: a neural network module controller, a PE matrix calculation controller, a convolution calculation controller, a maximum pooling controller, a fully connected calculation controller and a storage controller; The neural network module controller is respectively connected to the PE matrix calculation controller, the convolution calculation controller, the maximum pooling controller and the fully connected calculation controller, and is used to schedule and control the PE matrix calculation controller, the convolution calculation controller, the maximum pooling controller and the fully connected calculation controller based on the calculation level of the current pulse Transformer neural network; The neural network module controller is connected to the storage module through the storage controller, and the storage controller is connected to the off-chip memory.

3. The hardware accelerator according to claim 2, characterized in that: The storage module includes a bias memory for storing bias data, a weight memory for storing weight data, a pulse memory for storing pulse data, a neuron membrane potential memory for storing neuron membrane potential, and a pulse matrix memory for storing a pulse matrix.

4. The hardware accelerator according to claim 2, characterized in that: The convolution calculation controller is used to read the required bias data, weight data and pulse input data; The PE calculation unit array is connected to the convolution calculation controller and is used to receive bias data, weight data and pulse input data sent by the convolution calculation controller; The bias data, weight data and pulse input data required for multiple time steps are processed in parallel to obtain calculation results corresponding to the multiple time steps; the calculation results are sequentially input into the pulse neurons in the order of the time steps to update the membrane potential according to the calculation results; if the updated membrane potential is greater than a threshold, a pulse is emitted and the membrane potential is reset at the same time; if the updated membrane potential is not greater than the threshold, a pulse is not emitted and the membrane potential is attenuated at the same time, thereby obtaining a first pulse matrix and membrane potential data corresponding to the multiple time steps.

5. The hardware accelerator according to claim 2, characterized in that: Max pooling controller, used to read membrane potential data; The PE calculation unit array is connected to the maximum pooling controller, and is used to receive the membrane potential data sent by the maximum pooling controller, and screen out the maximum membrane potential from the membrane potential data.

6. The hardware accelerator according to claim 4, characterized in that: The PE matrix calculation controller is used to read a first pulse matrix; wherein the first pulse matrix includes a Q pulse matrix, a K pulse matrix and a V pulse matrix; The matrix processing array is connected to the PE matrix calculation controller, and is used to receive the first pulse matrix sent by the PE matrix calculation controller; in parallel, bitwise AND operations are performed on the pulse sequences with wide sequence lengths in the Q pulse matrix and the pulse sequences of the K pulse matrix corresponding to the embedded dimension of the Q pulse matrix, until the pulse sequences corresponding to all embedded dimensions in the Q pulse matrix are completed, and a bitwise AND matrix is ​​obtained; in parallel, bitwise OR operations are performed on each pulse sequence in the bitwise AND matrix, and a bitwise OR result matrix is ​​obtained; in parallel, bitwise AND operations are performed on the pulse sequences in the bitwise OR result matrix and the pulse sequences of the V pulse matrix corresponding to the embedded dimension of the bitwise OR result matrix, until the pulse sequences corresponding to all embedded dimensions in the bitwise OR result matrix are completed, and a second pulse matrix is ​​obtained, and the second pulse matrix is ​​sent to an off-chip memory.

7. The hardware accelerator according to claim 2, characterized in that: The storage controller is connected to the off-chip memory and is used to read the second pulse matrix from the off-chip memory; The PE calculation unit array is connected to the storage controller and is used to receive a second pulse matrix sent by the storage controller; Performing a convolution operation based on the bias data, the weight data and the second pulse matrix to obtain a third pulse matrix; Using the third pulse matrix as a new second pulse matrix, combining the bias data, the weight data and the new second pulse matrix; Until the number of iterative convolution operations reaches a preset number, the third pulse matrix obtained by the last iteration is sent to the storage controller, so that the storage controller stores the third pulse matrix in the off-chip memory.

8. The hardware accelerator according to claim 6, characterized in that: A fully connected computing controller for reading required bias data, weight data and a third pulse matrix; The fully-connected core is respectively connected to the fully-connected computing controller and the storage controller, and is used to receive bias data, weight data and a third pulse matrix sent by the fully-connected computing controller, and perform a fully-connected operation based on the bias data, weight data and the third pulse matrix, and output a fully-connected result.

9. The hardware accelerator according to claim 1, characterized in that: The PE computing unit array is composed of a plurality of PEs for simulating spiking neurons; each PE is composed of four convolution units and one pooling unit; The matrix processing array is composed of four multi-bit AND calculation units and four multi-bit OR calculation units.

10. A method for accelerating a hardware accelerator supporting pulse Transformer calculation, characterized in that: The method comprises: The PE computing unit array is used to realize the pulse convolution calculation, pulse emission and membrane potential maximum pooling calculation tasks of the pulse Transformer neural network; The pulse matrix calculation task of the pulse Transformer neural network is realized by the matrix processing array based on the bitwise AND and bitwise OR operations; The computational task of spiking fully connected Transformer neural network is implemented through fully connected kernels.

Images