Hardware system for deploying sparse pulse transformer models

Abstract

The application belongs to the technical field of integrated circuits, and discloses a hardware system for deploying a sparse pulse Transformer model, which comprises an event input module, a space-time pruning module, a scheduling module, a sparse pulse calculation module and a classification head output module. The event input module converts a time sequence pulse data stream into a pulse token sequence. The space-time pruning module judges the importance of each time step and each token through a pulse accumulation value, generates a space-time pruning mask, and the scheduling module screens the token sequence according to the mask to form a sparse token sequence. The sparse pulse calculation module performs sparse inference on the sparse token sequence, and finally the classification head output module outputs a classification result. The system can be used for deploying a sparse pulse Transformer model through a special hardware control and scheduling mechanism, and provides accurate execution paths and efficient scheduling support for dynamically changing sparse data streams.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of integrated circuits, and more particularly relates to a hardware system for deploying a sparse spike Transformer model. BACKGROUND

[0002] With the continuous evolution of artificial intelligence algorithms, brain-like computing is gradually becoming an important direction of high-energy efficient intelligent chip design. Spike neural network (SNN) has been widely used in dynamic visual processing, low-power edge perception and other fields due to its event-driven and sparse activation characteristics, and has shown significant energy efficiency advantages in approaching the processing mode of biological neural systems.

[0003] To improve the modeling capability of SNN in complex tasks, in recent years, the self-attention mechanism with strong global modeling capability in the Transformer structure has been introduced into the SNN system, forming a spike Transformer model structure. This structure has improved in expression capability and time series modeling capability, but also brings problems such as increased data flow density, expanded calculation path, and rising structure deployment complexity. The current hardware system is usually designed for general spike networks, and does not have special control and calculation mechanisms for the redundancy of a large number of tokens and time steps in the sparse spike Transformer model at the structure level. In particular, when deploying complex models with pruning requirements and dynamic structure characteristics, the existing system cannot provide accurate sparse execution paths and efficient data scheduling support. SUMMARY

[0004] In view of the above defects or improvement needs of the prior art, the present application provides a hardware system for deploying a sparse spike Transformer model, which aims to provide a hardware system that can deploy a sparse spike Transformer model and provide accurate sparse execution paths and efficient data scheduling support for complex models with pruning requirements and dynamic structure characteristics.

[0005] To achieve the above-mentioned purpose, a hardware system for deploying a sparse spike Transformer model is provided, which comprises an event input module, a space-time pruning module with G pruning channels, a scheduling module with G scheduling channels, a sparse spike calculation module with G sparse inference layers, and a classification head output module, the gth pruning channel obtains the token sequence output by the (g-1)th sparse inference layer and generates a mask, the gth scheduling channel obtains the token sequence output by the (g-1)th sparse inference layer and the mask output by the gth pruning channel, and inputs the generated sparse token sequence into the gth sparse inference layer, g=1, 2, …, G, and the token sequence output by the 0th sparse inference layer is the token sequence output by the event input module.

[0006] The event input module is configured to convert the time-sequential pulse event data streams of different time steps into pulse token sequences of corresponding time steps, each token sequence having a plurality of tokens;

[0007] Each pruning channel is configured to select, from the token sequences of all time steps received thereby, a preset number of time steps with the largest pulse accumulation values as reserved time steps, and select, from the token sequences of each reserved time step, a preset number of tokens with the largest pulse accumulation values as reserved tokens in the corresponding reserved time step, to generate a time mask representing the time step selection result and a token mask representing the token selection result;

[0008] Each scheduling channel is configured to extract, according to the time mask and the token mask output by the corresponding pruning channel, the reserved tokens in the reserved time steps from the token sequences output by the corresponding sparse inference layer, to form a sparse token sequence;

[0009] Each sparse inference layer is configured to perform pulse-driven self-attention sparse inference calculation on the input sparse token sequence, to form a new token sequence;

[0010] The classification head output module is configured to obtain the token sequence of the Gth sparse inference layer and perform full connection mapping, to output a classification result.

[0011] Overall, compared with the prior art, the above technical solutions conceived by the present application mainly have the following beneficial effects.

[0012] 1. The present application constructs a complete hardware system from data input, dynamic space-time pruning to sparse inference calculation, in which the event input module can convert external time sequence pulse data stream into structured pulse token sequence, the space-time pruning module can judge the importance of each time step and each token in the same token sequence through pulse accumulation value, and perform space-time pruning based on this, generate the corresponding mask, the scheduling module can filter the input token sequence according to the mask of space-time pruning, form a sparse token sequence, and the sparse pulse calculation module can perform efficient sparse inference on the input sparse token sequence, and finally output the classification result by the classification head output module. The system uses a dedicated hardware control and scheduling mechanism to solidify the pruning strategy at the algorithm level into real-time logic inside the chip, providing accurate execution path and efficient scheduling support for dynamically changing sparse data stream without off-chip software intervention. This scheme effectively solves the core technical problems of insufficient control ability, low utilization of computing resources and inflexible path scheduling caused by the lack of special control and flexible data path when deploying complex pruning models on existing neuromorphic chips, providing key hardware support for high energy efficiency and high throughput of sparse pulse models on chip side.

[0013] 2. In an optional embodiment, the event input module includes a frame buffer unit, a token mapping unit and a token output buffer unit. Through this structure, the original input data can be efficiently mapped into a data structure that can be recognized and processed by hardware, ensuring the simplicity of token data access logic and the efficiency of hardware addressing.

[0014] 3. In an optional embodiment, the event input module as a whole follows a flow-like workflow scheduling, optimizing throughput and avoiding mutual blocking of data processing and data reading. When the input data of the current frame is performing token mapping operation, the frame buffer unit simultaneously performs data buffering of the subsequent time frame, greatly improving the event stream input rate and reducing processing waiting time. Moreover, the frame buffer unit has a cache space release mechanism, which can improve storage utilization, effectively reduce on-chip cache demand when the data stream size is large, and reduce cost and improve energy efficiency.

[0015] 4. In an optional embodiment, the data compression and splicing unit selects to merge output or separate output according to the sparsity of the token, realizes the automatic selection of "merge / independent" block mode, and flexibly adapts to different data sparse distribution, so as to improve the efficiency of one-time transmission and avoid the access delay caused by excessive cross-frame alignment during merging.

[0016] 5. In an optional embodiment, the down-sampling unit has a "multi-scale gating" mechanism, when certain spatial blocks are detected to have little pulse activity, the calculation of these regions can be directly skipped, and the high activity regions are convolved with regular step size, achieving reserved and down-sampling, thus achieving more efficient feature compression. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 is a structural schematic diagram of a hardware system for deploying a sparse pulse Transformer model in an embodiment of the present application.

[0018] Figure 2 is a structural schematic diagram of a scheduling module in an embodiment of the present application.

[0019] Figure 3 is a structural schematic diagram of a sparse inference structure in an embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0021] As Figure 1 is a structural schematic diagram of a hardware system for deploying a sparse pulse Transformer model in an embodiment of the present application.

[0022] The hardware system includes an event input module, a space-time pruning module, a scheduling module, a sparse pulse calculation module and a classification head output module. The space-time pruning module has G pruning channels, the scheduling module has G scheduling channels, and the sparse pulse calculation module has G sparse inference layers. The token sequence output by the event input module is transmitted to the first pruning channel to generate a mask and is transmitted to the first scheduling channel. The first scheduling channel acquires the mask generated by the first pruning channel and the token sequence output by the event input module and performs scheduling. The scheduling result is transmitted to the first sparse inference layer. The first sparse inference layer performs sparse calculation on the scheduling result of the first scheduling channel and transmits the token sequence calculated thereby to the second pruning channel. The second pruning channel generates a corresponding mask and transmits it to the second scheduling channel. The second scheduling channel performs scheduling processing and inputs the second sparse inference layer. By analogy, the token sequence is output by the Gth sparse inference layer. The token sequence output by the Gth sparse inference layer is transmitted to the classification head output module.

[0023] The event input module is configured to convert an externally input time-sequential pulse event data stream of T0 time steps into T0 pulse token sequences, each token sequence having N0 tokens, and each token having D binary characters, with a character of 1 representing a pulse and a character of 0 representing no pulse.

[0024] Specifically, the size of the time-sequential pulse event data stream of each time step is CxHxW, where C is the number of input channels, and HxW is the size of a frame image. The time-sequential pulse event data stream is recorded in the form of pulse events, each pulse event containing timestamp information and spatial position information, in the format of wherein represents the time step at which the event occurs, represents the two-dimensional coordinate position of the event in the corresponding frame image, i.e., the spatial position at which the event occurs, and c represents the corresponding channel. The event input module is configured to analyze and buffer the pulse event data stream, specifically, to convert the CxHxW time-sequential pulse event data stream of each time step into an N0xD binary token sequence, where N0 is the number of tokens, is the length of a single token, and a plurality of token sequences of time steps form a token tensor.

[0025] In an embodiment, the event input module includes a frame buffer unit, a token mapping unit, and a token output buffer unit.

[0026] The frame buffer unit is configured to record whether a pulse event occurs at each spatial position in the time-sequential pulse event data stream of each time step. Each spatial position corresponds to a buffer unit. If a pulse event occurs at a spatial position, the buffer unit corresponding to the spatial position stores 1. If no pulse event occurs at a spatial position, the buffer unit corresponding to the spatial position stores 0, forming a binary data structure of each time step.

[0027] Specifically, the frame buffer unit efficiently maps the time-sequential pulse event data stream of each time step into a binary data structure that can be recognized and processed by hardware. The total dimension of the binary data structure corresponding to each time step is CxHxW. For each channel, the coordinates of the frame image of HxW size are flattened into the format of a linear address interval. Each coordinate corresponds to a buffer unit, and each buffer unit stores 1 bit of information, indicating whether a pulse event occurs at the current spatial position. 1 represents that a pulse event occurs at the current time step and channel, and 0 represents that no pulse event occurs. By flattening the two-dimensional coordinates, the frame buffer unit can quickly address and read at the hardware level and ensure efficient processing of event pulses, providing fast data sources for subsequent mapping links.

[0028] Preferably, in the frame buffer unit, the data of each channel is independently stored without interference, thereby maintaining the original characteristics of the data to the greatest extent.

[0029] The token mapping unit is configured to perform two-dimensional convolution calculation on the binary data structure corresponding to each time step in the frame buffer unit, and then perform integral and threshold firing operation by using the LIF neuron module to obtain a token sequence at each time step.

[0030] Specifically, the token mapping unit extracts a CxHxW-dimensional binary data structure corresponding to each time step from the frame buffer unit each time, and maps the binary data structure to a binary token sequence with a size of N0xD, where N0 is a sequence length, a dimension of a single token, so that the subsequent token construction and pruning module can perform structured processing on the token. The CxHxW-dimensional binary data structure is mapped and processed by a two-dimensional convolution calculation array, and integral and threshold firing operation is performed by using the LIF neuron module to obtain a convolution result of D is a number of convolution output channels, and H'xW' is an image size of each convolution output channel. The values at the same two-dimensional coordinates in the D convolution output channels are combined to form a token at a corresponding position, that is, a single token includes D binary data, and there are H'xW' two-dimensional coordinates in total. Therefore, the number of tokens is N0=H'xW', and thus a token sequence with a dimension of N0xD at each time step is obtained.

[0031] It should be noted that the integral-firing process of the LIF neuron module is suitable for preprocessing of pulse data. The integral-firing process can simulate the accumulation of input pulses by biological neurons at a hardware level, and fire a pulse output once the threshold is exceeded. The integral-firing process not only preserves the timing characteristics of the pulse data, but also provides a good foundation for the structuring of the subsequent token sequence.

[0032] The token output buffer unit is configured to buffer the token sequence output by the token mapping unit to obtain T0 token sequences at T0 time steps.

[0033] Specifically, the token output buffer unit is configured to buffer the token output by the token mapping unit to form a T0xN0xD time token sequence, which is used to call the token sequence by a downstream module. In an optional implementation, the token output buffer unit and the downstream module can be connected through a dedicated high-speed interface to improve the data reading efficiency, and the buffer depth can be flexibly configured according to the hardware resource status and algorithm requirements.

[0034] In an embodiment, the event input module performs data scheduling in a pipeline mode, when the frame buffer unit inputs the binary data structure of the current time step to the token mapping unit, the frame buffer unit releases the cache space of the binary data structure of the current time step and performs the caching of the binary data structure of the next time step.

[0035] Specifically, when the input data of the current frame is subjected to the token mapping operation, the frame buffer unit simultaneously performs the data caching of the subsequent time frame, thereby improving the throughput of the event input and reducing the delay of the token mapping.

[0036] Specifically, the frame buffer unit can set a cache space release mechanism, when the pulse image data of one time step is cached, a "start" signal is generated to the downstream token mapping unit to start the token mapping operation, and when the data transmission is completed, a "completion" response is generated to immediately control the release of the cache space of the current frame, thereby saving the storage overhead.

[0037] The space-time pruning module has G pruning channels, each pruning channel is used to select a preset number of time steps with the largest pulse accumulation values from the token sequences of all time steps received thereby as reserved time steps and select a preset number of tokens with the largest pulse accumulation values from the token sequence of each reserved time step as reserved tokens in the corresponding reserved time step, and generate a time mask representing the time step selection result and a token mask representing the token selection.

[0038] After each pruning channel receives the token sequences of multiple time steps, the T' time steps with the largest pulse accumulation values are selected as the reserved time steps, realizing time pruning, and then for each reserved time step, the N' tokens with the largest pulse accumulation values are selected from the token sequence thereof as the reserved tokens, realizing space pruning.

[0039] The module filters the input data in the token dimension and the time step dimension based on the pulse accumulation value SS, only retains a number of time steps / tokens with larger accumulation values, and outputs sparse time step / token index information, so as to improve the efficiency of subsequent calculation.

[0040] To record the pruning, a time mask representing the time step selection result and a token mask representing the token selection result are generated. For example, for a pruning channel, the input has T time steps, each time step has N tokens, the length of the time mask is T, and the length of the token mask is N. The character positions in the time mask are one-to-one mapped with the time steps, and the characters at different positions represent whether the corresponding time step is retained, taking a value of 1 to represent retaining the time step, and taking a value of 0 to represent pruning the time step. For example, the time mask 0100101 indicates that the 1st, 3rd, and 6th time steps are retained, and the remaining time steps are pruned. Each retained time step corresponds to a token mask, and the character positions in the token mask are one-to-one mapped with the tokens, and the characters at different positions represent whether the corresponding token is retained, taking a value of 1 to represent retaining the token, and taking a value of 0 to represent pruning the token. For example, the time mask 0100101 indicates that the 1st, 3rd, and 6th tokens are retained, and the remaining tokens are pruned.

[0041] It can be understood that, as the multiple rounds of pruning proceed, the number of time steps T input by the pruning channel in the later stage is smaller and smaller, and the length N of the token sequence under each time step is shorter and shorter.

[0042] It should be noted that, if or , it indicates that no time step or token is pruned, and the comparison and sorting logic unit can only perform a pass-through process; if or is large, it can be considered as deep pruning, and the subsequent calculation load can be greatly reduced.

[0043] In an embodiment, the space-time pruning module has G pruning channels, each pruning channel including a pulse accumulation circuit unit, a comparison and sorting logic unit, and a mask generation unit.

[0044] The pulse accumulation circuit unit is configured to perform pulse summation on the token sequence SEQ t of each time step t to obtain the pulse accumulation value SS t of the corresponding time step t, and perform pulse summation on each token in the token sequence to obtain the pulse accumulation value SS n of the corresponding token, n being the number of the token in the corresponding token sequence.

[0045] Specifically, the pulse accumulation value SS t of the time step t can reflect the importance of the corresponding time step, and the pulse accumulation value SS nThe importance of the corresponding token can be reflected by calculating the pulse accumulation value, which can provide a pruning basis for the pruning operation.

[0046] The token sequence SEQ t at time step t t The calculation formula is as follows:

[0047]

[0048] In the formula, denotes the value of the dth element in the nth token in the token sequence SEQ t at time step t, and the value of 1 indicates that there is a pulse, and the value of 0 indicates that there is no pulse.

[0049] The calculation formula of the pulse accumulation value SS n of the nth token is as follows:

[0050]

[0051] In the formula, denotes the value of the dth element in the nth token, and the value of 1 indicates that there is a pulse, and the value of 0 indicates that there is no pulse.

[0052] In an embodiment, the accumulation process can use a multi-level addition tree to perform parallel summation, support sparse skip-zero accumulation (only operate on non-zero data), and cache the results in an output register array for subsequent comparison and sorting logic unit access and processing.

[0053] The comparison and sorting logic unit is configured to select T’ time steps with the largest pulse accumulation values in the T time steps as reserved time steps according to a preset time pruning ratio P t , and select N’ tokens with the largest pulse accumulation values in the token sequence as reserved tokens according to a preset space pruning ratio P s .

[0054] Specifically, the comparison and sorting logic unit is configured to perform time and space pruning on the token sequence of the T time steps according to the pulse accumulation values and the preset pruning ratios, reserve the T’ time steps in the front of the ranking and the first N’ tokens in the corresponding time steps, and output the corresponding index positions.

[0055] The calculation process of T’ and N’ is shown in formulas (3) and (4):

[0056]

[0057]

[0058] In specific operation, the comparison and sorting logic unit adopts a priority queue network. When time sorting is performed, the accumulated values of T time steps are input in parallel into a set of comparator networks to obtain sorting from large to small, and the reserved T' time steps are recorded in the form of time step index. When space sorting is performed, the accumulated values of N tokens in the token sequence are input in parallel into a set of comparator networks to obtain sorting from large to small, and the reserved N' time steps are recorded in the form of token index. The index information output by the unit will be passed to the mask generation unit for subsequent pruning selection.

[0059] The mask generation unit is used for generating a time mask representing the time step selection result and a token mask representing the token selection result according to the selection result of the comparison and sorting logic unit.

[0060] Specifically, the mask generation unit is composed of a bitmap constructor, a mask buffer and an output controller. The bitmap constructor includes a bitmap register array, and an index position is written into a bitmap by using lookup table mapping logic to convert the reserved index position into a fixed-length Boolean vector (i.e. a mask bitmap); the mask buffer is structured in a "time step / token sequence" structure for waiting to be read by a scheduling module; and the output controller outputs the mask to the scheduling module in groups as needed and controls whether the data channel is open.

[0061] In the present application, the index management cost in a sparse scenario can be reduced by the mask mechanism, the flexible expression of "continuous reservation" or "discontinuous reservation" after space-time pruning is supported, and the manageability of the data stream after pruning is enhanced. Moreover, the mask generation unit provides an explicit sparse data mask for subsequent neural network calculation, so that the system can shut down or omit the part not participating in the operation at the hardware level, thereby realizing energy-saving acceleration.

[0062] In the scheduling module, each scheduling channel is used to extract the reserved tokens in the reserved time steps from the token sequence output by the corresponding sparse inference layer according to the time mask and the token mask output by the corresponding pruning channel, to form a sparse token sequence.

[0063] Specifically, after the token sequence output by the sparse inference layer is sent into the corresponding pruning channel for pruning processing to generate a mask, the corresponding scheduling channel selects the reserved tokens from the token sequence output by the sparse inference layer according to the mask to form a sparse token sequence.

[0064] In an embodiment, the scheduling module includes G scheduling channels, as shown in Figure 2 As shown in FIG. 6, which is a structural schematic diagram of the scheduling module in an embodiment of the present application, each scheduling channel includes a time step filter, a token filter and rearranger, and a data compression and splicing unit.

[0065] The time step filter is configured to receive the time mask corresponding to the pruning channel and the token sequence corresponding to the sparse inference layer, and extract the token sequence corresponding to the reserved time step. Specifically, the time step filter in the gth scheduling channel receives the token sequence output by the (g-1)th sparse inference layer and the time mask output by the gth pruning channel, extracts the token sequence corresponding to the reserved time step according to the time mask, and skips the unselected time step. In an optional embodiment, the time step filter can further sort and output according to the priority or urgency of each time step to meet the priority scheduling requirement of certain online learning or real-time inference scenarios.

[0066] The token screening rearranger is configured to receive the token mask corresponding to the pruning channel and the token sequence corresponding to the reserved time step extracted by the time step filter, and extract the reserved token from the token sequence corresponding to each reserved time step to form a sparse token sequence.

[0067] Specifically, the token screening rearranger in the gth scheduling channel is configured to receive the token mask output by the gth pruning channel and the token sequence corresponding to the reserved time step output by the token screening rearranger in the gth scheduling channel. In each reserved time step, the token screening rearranger further performs token-by-token screening according to the token mask, and performs token rearrangement and index updating, so that the subsequent computing array obtains a sparse token stream with continuous numbering starting from "0".

[0068] The data compression splicing unit is configured to perform structured processing on the sparse token sequence output by the token screening rearranger and output the processed sparse token sequence.

[0069] Specifically, the data compression splicing unit is configured to compress and splice the dispersed sparse token data stream of multiple frames or batches after time step screening and token screening into a structured output. After screening and rearranging each time step, they can be combined into a continuous output segment, and the tokens reserved in each time step are written continuously.

[0070] In an embodiment, the data compression splicing unit supports two splicing modes of variable-length block and fixed-length block. When the total number of tokens in the continuous multiple time steps in the sparse token sequence is less than the preset output value, the tokens in the continuous multiple time steps whose total number is less than the preset output value are combined into an output block and then output, thereby improving the efficiency of one-time transmission. For time steps that do not meet the merging requirement, the token sequence of each time step is output as a separate output block, thereby avoiding the access delay caused by excessive cross-frame alignment during merging.

[0071] In one embodiment, the scheduling module further includes a multi-port scheduler, which is used to send the token sequences output by different sparse inference layers to the corresponding pruning channels for pruning processing.

[0072] Specifically, the multi-port scheduler manages the data streams of each sparse inference layer and correctly sends them to the corresponding pruning channels. The scheduler's inputs are connected to each sparse inference layer, and its outputs are connected to each pruning channel. The scheduler uses a multi-input arbitrator to determine which data stream is forwarded to the pruning module at any given time.

[0073] In one embodiment, the scheduling module further includes a synchronization controller, which is used to coordinate the scheduling flow between the modules.

[0074] Specifically, the synchronization controller is used to coordinate the clock, reset, and token data flow control signals between the scheduling module and other modules, coordinate the scheduling and flow of timing token sequences between modules, and ensure the timing alignment of data reading / output at each stage of the pipeline.

[0075] In the sparse pulse computation module, each sparse inference layer is used to perform pulse-driven self-attention sparse inference computation on the input sparse token sequence.

[0076] Specifically, the sparse impulse computation module is used to perform sparse inference computation in the impulse Transformer model. The impulse Transformer model has multiple sparse inference layers. Therefore, multiple sparse inference layers are set in the sparse impulse computation module, and each sparse inference layer performs sparse inference computation in the corresponding stage of the sparse impulse computation module.

[0077] In one embodiment, each sparse inference layer includes an input buffer, a sparse inference structure, and an output buffer connected in sequence.

[0078] The input buffer is used to obtain and cache the sparse token sequence output by the corresponding scheduling channel.

[0079] The Transformer-style SNN computation unit is used to perform pulse-driven self-attention operations, linear mapping operations, and spiking neuron activation updates on the token in the pulse domain, and then input the results into the output buffer.

[0080] like Figure 3 The diagram shown is a schematic of a sparse inference structure according to an embodiment of the present invention. The sparse inference structure includes a Transformer-type SNN computation unit, which includes a pulse-driven self-attention operation unit, a neuron activation unit, and a MAC operation unit.

[0081] First, the token sequence passes through the neuron activation unit and the MAC operation unit to generate the Q, K, and V matrices.

[0082] Specifically, the sparse token is obtained from the input buffer of the current level, and the sparse linear mapping operation is performed by the MAC operation unit to obtain the initial Q, K, and V matrix results. Since the spatio-temporal pruning module has screened out a large number of redundant tokens in advance, these tokens do not need to perform MAC operation, greatly reducing the waste of computing power. It should be noted that before inputting into the MAC operation unit, the token needs to pass through the neuron activation unit for activation operation to generate pulse representation, and after obtaining the initial Q, K, and V matrix results by the MAC operation unit, it still needs to pass through the neuron unit activation.

[0083] Then, the Q, K, and V matrices are operated by the pulse-driven self-attention operation unit, the neuron activation unit, and the MAC operation unit to complete the sparse inference operation.

[0084] Specifically, the Q and K are dot multiplied to obtain the attention score, wherein the sparse MAC operation is also performed, and the MAC operation unit only schedules the non-zero part of the corresponding token for operation. Since the operation in this stage is completed in the pulse domain, no additional SoftMax activation circuit is needed, saving logic overhead. After the attention score is calculated, the weighted accumulation operation is performed with the V vector, and finally the linear mapping is performed to obtain the result of the pulse-driven self-attention operation. The result is sent to the fully connected structure composed of the neuron activation unit and the MAC operation unit for further transformation operation, realizing feature fusion operation, and obtaining the sparse inference operation result.

[0085] It can be understood that the pulse neuron activation unit runs through all stages of the network and completes the specific operation of the neuron membrane potential leakage, integration, and firing process in the SNN. The neuron membrane potential is updated according to the input sparse MAC result, and once the threshold is exceeded, the pulse is output and the membrane potential is reset. This module also plays the role of sparse gating: if there is no valid token input to the neuron in the current period, the clock is turned off to reduce invalid power consumption.

[0086] The output buffer is used to obtain the token sequence output by the sparse inference structure and cache the token sequence generated by the corresponding sparse inference layer.

[0087] In an embodiment, the sparse inference structure further comprises a down-sampling unit for inputting the sparse token sequence in the input buffer into the Transform-style SNN calculation unit after down-sampling, so as to reduce the subsequent calculation amount. The down-sampling can be specifically implemented by a convolution calculation unit with a stride greater than 1. Further, when the down-sampling unit implements the down-sampling by convolution calculation, if it is detected that there is a region in the sparse token sequence with a pulse number less than a calculation threshold, the convolution of the region is directly skipped. Specifically, the down-sampling unit has a "multi-scale gating" mechanism. When it is detected that some spatial blocks / channels have no pulse activity, the calculation of these regions can be directly skipped based on the gating. The high-activity regions are subjected to regular stride convolution, so as to reserve and down-sample.

[0088] In an embodiment, each sparse inference structure only performs calculation on the region in the token sequence with a pulse number reaching a preset activity degree when performing sparse matrix multiplication operation, that is, only the non-zero weight corresponding to the active pulse token is subjected to multiplication and addition operation, so as to reduce the memory access pressure and reduce the operation power consumption.

[0089] In an embodiment, the sparse pulse calculation module further comprises a global control and register management module for uniformly scheduling the structures in each sparse inference layer, configuring the working mode of the MAC operation unit according to the pruning result, routing state and the like, and managing the cache of the network parameters such as the threshold value and leakage parameter of each layer.

[0090] The classification head output module is used for obtaining the token sequence of the Gth sparse inference layer and performing full connection operation and pulse neuron activation, and outputting the classification result. Specifically, the classification head output module is used for mapping the global feature vector to the representation in the classification space through a sparse linear mapping network, and supports outputting in the form of pulse activation. The classification head output module internally comprises a sparse matrix multiplier and a pulse neuron activation unit. The sparse matrix multiplier is used for performing full connection operation on the output of the sparse pulse calculation module, so as to realize linear mapping. The pulse neuron activation unit is used for performing integration and activation functions similar to the same type of module in the sparse pulse calculation module, so as to realize the output of the classification result, and the output is converted into pulse form.

[0091] The working process of this hardware system is as follows:

[0092] The event input module generates an initial time sequence token sequence, the synchronization controller controls the scheduling module to read the initial token sequence of the event input module, the multi-port scheduler routes data to the first pruning channel, after pruning, the sparse token sequence is obtained through the time step filter, the token screening rearranger and the data compression splicing unit, and is routed to the first input buffer; for the gth layer (g = 2, 3, …, G), the scheduling module obtains the token data that has completed sparse inference calculation but has not been pruned from the (g-1)th output buffer and sends it to the space-time pruning module to generate a mask; after the space-time pruning module outputs the mask, the scheduling module reads the token to be retained from the (g-1)th output buffer according to the mask and writes the rearranged and compressed sparse token data to the gth input buffer, wherein during the pruning process of the (g-1)th output buffer data, the scheduling module can continue to route the data of the gth output buffer to the pruning module of the gth layer, so that the scheduling module realizes the sequential "read-pruning-rearrangement-writing" of the multi-layer token sequence data through the handshake signals (such as "ready", "complete" and the like) with the output buffers of each layer and the pipeline scheduling of the space-time pruning control module, on the one hand, it ensures that the pruning process and subsequent data preparation can be executed in a pipelined manner, on the other hand, it also enables each layer to continuously receive tokens from the input buffer and perform calculations, and finally completes an efficient and sparse inference process in the G-layer cascaded system.

[0093] In a specific operation, the original pulse Transformer model can be processed by the following deployment method to obtain an optimized pulse Transformer model; the optimized pulse Transformer model can be deployed on the hardware system described above.

[0094] The deployment method includes:

[0095] S1, using the original pulse Transformer model as an initial neural network.

[0096] S2, the redundant convolution kernel or self-attention matrix part in the current neural network is preliminarily pruned or quantized and mapped to the hardware simulation unit of the hardware system, and the current first hardware simulation performance index is obtained. The hardware simulation performance index collected may be, for example, operation power consumption, throughput, congestion degree of on-chip data flow scheduling, etc.

[0097] S3, determine whether the first stop condition is met, if yes, go to S4; otherwise, return to S2 for a new round of pruning or quantization iteration; wherein the first stop condition includes: the current first hardware simulation performance index meets the set requirement, or the pruning or quantization iteration number reaches the first preset iteration number.

[0098] S4, determine whether the second stop condition is met, if yes, go to S6; otherwise, go to S5; wherein the second stop condition includes: the second hardware simulation performance index obtained under the current network topology meets the expected requirement.

[0099] S5, change the topology structure of the current neural network and redeploy it to the hardware simulation unit to reacquire the second hardware simulation performance index; then jump to S4 again to determine whether the second stop condition is met.

[0100] Specifically, the order of the Transformer SNN layer in the network, the pruning ratio, and the down-sampling strategy can be adjusted in structure; if necessary, the mapping strategy (sparse matrix multiplication dimension, pulse activation gating, etc.) of the classification head part can also be adapted to further improve the energy efficiency ratio and throughput of the network. After the deployment of the new structure is completed, the latest performance index of the hardware simulation unit is detected.

[0101] S6, the current neural network is used as the preset neural network after deployment and optimization are completed; it is the final output deployable network, which is the best coupling with each functional module (such as the space-time pruning module, the scheduling module, the sparse pulse calculation module, and the classification head output module) of the neuromorphic computing hardware system.

[0102] The technical features of the above-described embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described, but as long as the combinations of the technical features do not contradict, they should be considered within the scope of the present disclosure. It should be noted that the "in an embodiment of the present disclosure", "for example", "such as" and the like are intended to illustrate the present disclosure, but not to limit the present disclosure.

[0103] The above-described embodiments only express several embodiments of the present disclosure, which are described in detail and specifically, but should not be interpreted as limiting the scope of the patent application. It should be noted that for those skilled in the art, without departing from the concept of the present disclosure, some modifications and improvements can be made, which are within the scope of the present disclosure.

Claims

1. A hardware system for deploying a sparse impulsive Transformer model, characterized in that, include: The system includes an event input module, a spatiotemporal pruning module with G pruning channels, a scheduling module with G scheduling channels, a sparse impulse calculation module with G sparse inference layers, and a classification head output module. The g-th pruning channel acquires the token sequence output by the (g-1)-th sparse inference layer and generates a mask. The g-th scheduling channel acquires the token sequence output by the (g-1)-th sparse inference layer and the mask output by the g-th pruning channel, and inputs the generated sparse token sequence into the g-th sparse inference layer. g = 1, 2, ..., G. The token sequence output by the 0th sparse inference layer is the token sequence output by the event input module. The event input module is used to convert the time-series pulse event data streams at different time steps into pulse token sequences for the corresponding time steps, and each token sequence has multiple tokens; Each pruning channel is used to select a preset number of time steps with the largest pulse accumulation value from the token sequence of all time steps it receives as reserved time steps, and to select a preset number of tokens with the largest pulse accumulation value from the token sequence of each reserved time step as reserved tokens in the corresponding reserved time step, generating a time mask representing the time step selection result and a token mask representing the token selection result. Each scheduling channel is used to extract the retained tokens in the retained time step from the token sequence output by the corresponding sparse inference layer based on the time mask and token mask output by the corresponding pruning channel, forming a sparse token sequence; Each sparse inference layer is used to perform pulse-driven self-attention sparse inference computation on the input sparse token sequence to form a new token sequence; The classification head output module is used to obtain the token sequence of the Gth sparse inference layer and perform a fully connected mapping to output the classification result. In the sparse impulse computation module, each sparse inference layer includes an input buffer, a sparse inference structure, and an output buffer connected in sequence. The sparse inference structure includes a downsampling unit and a Transformer-style SNN computation unit. The input buffer is used to obtain and cache the sparse token sequence output by the corresponding scheduling channel; The downsampling unit is used to downsample the sparse token sequence in the input buffer and then input it into the Transformer-type SNN computing unit; The Transformer-type SNN computation unit is used to perform pulse-driven self-attention operation, linear mapping operation, and spiking neuron activation update on the token in the pulse domain and then input the result into the output buffer. The output cache is used to obtain the token sequence output by the sparse inference structure and cache it as the token sequence generated by the corresponding sparse inference layer.

2. The hardware system as described in claim 1, characterized in that, The event input module includes a frame buffer unit, a token mapping unit, and a token output buffer unit; The frame buffer unit is used to record whether a pulse event has occurred at each spatial position in the time-series pulse event data stream. Each spatial position corresponds to a buffer unit. If a pulse event occurs at a spatial position, its corresponding buffer unit stores 1; if no pulse event occurs at a spatial position, its corresponding buffer unit stores 0, forming a binary data structure for each time step. The Token mapping unit is used to perform two-dimensional convolution calculation on the binary data structure corresponding to each time step in the frame buffer unit, and then use the LIF neuron module to perform integration and threshold distribution calculation to obtain the token sequence under each time step. The Token output caching unit is used to cache the token sequence at each time step output by the Token mapping unit.

3. The hardware system as described in claim 2, characterized in that, The event input module performs data scheduling in a pipeline mode. When the frame buffer unit inputs the binary data structure of the current time step into the Token mapping unit, the frame buffer unit releases the buffer space of the binary data structure of the current time step and executes the caching of the binary data structure of the next time step.

4. The hardware system as described in claim 1, characterized in that, In the spatiotemporal pruning module, each pruning channel includes a pulse accumulation circuit unit, a comparison and sorting logic unit, and a mask generation unit; The pulse accumulation circuit unit is used to process the token sequence SEQ for each time step t. t The pulse summation is performed to obtain the pulse accumulation value SS corresponding to time step t. t And to obtain the pulse accumulation value SS for each token in the token sequence by summing the pulses. n n is the token number in the corresponding token sequence; The comparison sorting logic unit is used to perform pruning according to a preset time pruning ratio P. t Select the T' time steps with the largest pulse accumulation values ​​from the input time steps as the retained time steps, and then perform spatial pruning according to the preset spatial pruning ratio P. s Select the N' tokens with the largest pulse accumulation values ​​in the token sequence as the reserved tokens; The mask generation unit is used to generate a time mask representing the time step selection result and a token mask representing the token selection result based on the selection result of the comparison and sorting logic unit.

5. The hardware system as described in claim 1, characterized in that, In the scheduling module, each scheduling channel includes a time step filter, a token filter rearranger, and a data compression and splicing unit; The time step filter is used to receive the time mask of the corresponding pruning channel and the token sequence of the corresponding sparse inference layer, and extract the token sequence corresponding to the retained time step; The token filter and rearranger is used to receive the token mask of the corresponding pruning channel and the token sequence corresponding to the retained time step extracted by the time step filter, and extract the retained token from the token sequence corresponding to each retained time step to form a sparse token sequence. The data compression and splicing unit is used to perform structured processing on the sparse token sequence output by the token filter rearranger according to the set transmission mechanism before outputting it.

6. The hardware system as described in claim 5, characterized in that, The data compression and splicing unit is used to merge the tokens in multiple consecutive time steps with a total number of tokens lower than a preset output value into an output block and output it when the total number of tokens in multiple consecutive time steps in the sparse token sequence is lower than a preset output value. For time steps that do not meet the merging requirements, the token sequence of each time step is output as a separate output block.

7. The hardware system as described in claim 1, characterized in that, The scheduling module also includes a multi-port scheduler, which is used to send the token sequences output by different sparse inference layers to the corresponding pruning channels for pruning processing.

8. The hardware system as described in claim 1, characterized in that, The scheduling module also includes a synchronization controller, which is used to coordinate the scheduling flow between modules.

9. The hardware system as described in claim 8, characterized in that, Each sparse inference structure performs sparse matrix multiplication only on regions in the token sequence where the number of pulses reaches a preset activity level.

Images