Near-memory computing sparse data compression circuit and near-memory computing chip

Abstract

This invention relates to the fields of near-memory computing and integrated circuits, and discloses a sparse data compression circuit and a near-memory computing chip for near-memory computing. The invention includes: a first rearrangement unit, which rearranges the sparse input data of the current cycle into locally compact data; a buffer unit, which stores the buffer data of the previous cycle; a control unit, connected to the buffer unit, which generates an offset control signal based on the buffer data; a second rearrangement unit, connected to the first rearrangement unit, the buffer unit, and the control unit, which, based on the offset control signal, concatenates the effective data of the locally compact data and the buffer data, and outputs the concatenated data; and an output unit, connected to the buffer unit and the second rearrangement unit, which, when the concatenated data reaches a preset transmission bit width, extracts data corresponding to the preset transmission bit width from the concatenated data as target data for output, and stores the remaining data in the concatenated data as buffer data in the buffer unit. This addresses the overall data throughput performance problem in near-memory computing systems.

Description

Technical Field

[0001] This invention relates to the fields of near-memory computing and integrated circuits, and particularly to sparse data compression circuits and near-memory computing chips for near-memory computing. Background Technology

[0002] In near-memory computing architectures, memory-side processing units typically need to transfer data to the accelerator side via on-chip networks for subsequent processing. On-chip networks generally use fixed-width transmission units as the basic data transfer granularity. However, in typical application scenarios such as neural network inference, due to sparsification processes such as pruning of weights or activation data, the data read from the memory side often contains a large number of zero-value elements, and the effective data is discretely distributed in the original data.

[0003] Currently, efficiently compressing sparse data in near-memory computing systems to fully utilize the transmission bandwidth of on-chip networks remains a significant challenge in this field. Related sparse data processing methods suffer from insufficient payload fill rate of transmission units and low utilization of on-chip network bandwidth when dealing with continuous data stream transmission, thus affecting the overall data throughput performance of near-memory computing systems. Summary of the Invention

[0004] The purpose of this invention is to provide a sparse data compression circuit and a near-memory computing chip to solve the problems of insufficient payload fill rate of transmission units and low on-chip network bandwidth utilization in related technologies, which in turn affect the overall data throughput performance of near-memory computing systems.

[0005] To address the aforementioned technical problems, this invention provides a sparse data compression circuit for near-memory computation, comprising: a first rearrangement unit for rearranging sparse input data of the current period into locally compact data; a buffer unit for storing buffer data from the previous period; a control unit connected to the buffer unit for generating an offset control signal based on the buffer data; a second rearrangement unit connected to the first rearrangement unit, the buffer unit, and the control unit for concatenating the locally compact data and the effective data of the buffer data based on the offset control signal, and outputting concatenated data; and an output unit connected to the buffer unit and the second rearrangement unit for extracting data corresponding to the preset transmission bit width from the concatenated data as target data for output when the concatenated data reaches a preset transmission bit width, and storing the remaining data in the concatenated data as buffer data in the buffer unit.

[0006] The present invention also provides a near-memory computing chip, comprising: a sparse data compression circuit for near-memory computing as described above; a memory connected to the sparse data compression circuit; an on-chip network connected to the sparse data compression circuit; and an accelerator connected to the on-chip network. The sparse data compression circuit is configured to receive sparse input data from the memory and output target data to the on-chip network. The on-chip network is configured to provide the target data to the accelerator.

[0007] In this invention, the sparse data compression circuit transforms the traditional approach of sending a partially loaded transmission unit independently for each batch into a mode where multiple batches of valid data are accumulated and spliced ​​across cycles before being sent to a fully loaded transmission unit. Simultaneously, because the total amount of data sent to the on-chip network is significantly reduced, the arbitration load and queuing depth of the on-chip network router are also correspondingly reduced, effectively alleviating network congestion in high-sparse-rate scenarios. This allows the accelerator to receive dense, valid data at a more stable rate, reducing computational idle cycles caused by the accelerator waiting for data. Attached Figure Description

[0008] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0009] Figure 1 This is a structural block diagram of a sparse data compression circuit for near-memory computing provided by the present invention.

[0010] Figure 2 This is a schematic diagram of a butterfly network in a sparse data compression circuit for near-memory computing provided by the present invention;

[0011] Figure 3 This is an example diagram of the cascaded structure and data flow of the first rearrangement unit and the second rearrangement unit in the sparse data compression circuit for near-memory computing provided by the present invention.

[0012] Figure label:

[0013] Memory-10, On-chip Network-20, Accelerator-30, Sparse Data Compression Circuit-100, First Rearrangement Unit-110, Second Rearrangement Unit-120, Buffer Unit-130, Control Unit-140, Output Unit-150. Detailed Implementation

[0014] As described in the background section, in near-memory computing architectures, the data read from the memory side contains a large number of zero-value elements after pruning and other sparsification processes, and the effective data is discretely distributed. This leads to the problem that the sparse data processing methods in related technologies have insufficient effective payload filling rate of transmission units and low utilization of on-chip network bandwidth when facing continuous data stream transmission.

[0015] The inventors discovered that the root cause of this problem lies in two structural defects. At the single-batch processing level, existing sparse data rearrangement schemes typically employ a butterfly network to perform a one-time densification operation on an input vector of length N. This involves generating a control bit sequence based on the sparse bitmap of the input vector through log2(N) levels of hierarchical exchange logic, driving each layer of exchange units to gradually aggregate non-zero elements towards the head of the vector. However, this network structure releases all internal states after completing one rearrangement, lacking the ability to temporarily store data and maintain state across processing cycles. This means that when the number of non-zero elements in a single input batch is insufficient to fill the bit width of the on-chip network transmission unit, the butterfly network will still send that batch as an independent transmission unit, resulting in a large number of empty bits in the transmission unit that are not occupied by valid data. At the cross-batch splicing level, in near-memory computing systems, memory typically triggers data reads with a fixed burst length. The number of zero-value elements in the vector block contained in each burst depends on the model sparsity and data storage layout, making it unpredictable. Because the bit width of the on-chip network data channel is fixed, traditional butterfly networks, while capable of locally denserizing the sparse vectors of the current batch within the network, lack a mechanism to merge the compressed valid data in the current batch with the residual valid data from the previous batch that did not fill the transmission units. In other words, there is no data-level continuity between two adjacent processing cycles; the compression results of each batch are isolated, making it impossible to achieve cross-batch accumulation and compact filling of valid data at the transmission unit granularity. The combination of these two defects results in a large number of transmission units in the on-chip network being sent in a semi-empty or even nearly empty state under high sparsity operating scenarios. This not only directly wastes the available bandwidth of the on-chip interconnect but also increases the arbitration load and queuing depth of the on-chip network router due to the increased redundancy of the total number of transmission units. Consequently, the accelerator side experiences computational idle cycles while waiting for valid data, ultimately limiting the overall data throughput performance of near-memory computing heterogeneous systems.

[0016] The aforementioned problems are particularly pronounced in heterogeneous systems where near-memory computing and on-chip neural network processing units (NNRAMs) work in tandem. In such systems, the near-memory processing unit on the memory side performs high-bandwidth operations such as matrix-vector multiplication. Its results are stored in memory in a sparse format and need to be transmitted via the on-chip network to the accelerator-side NNRAM for subsequent nonlinear activation, normalization, or next-layer inference operations. However, the computational arrays of NNRAMs typically require input data in a continuous, dense format to maintain full pipeline operation. This means that the sparse data output by the near-memory processing unit not only faces insufficient utilization of the on-chip network's transmission bandwidth but also requires additional densification preprocessing steps upon reaching the accelerator due to data format mismatch, further increasing system latency. Therefore, if sparse data can be compressed into a dense format that meets the full load requirements of the transmission unit before it enters the on-chip network, it can not only improve the bandwidth utilization of the on-chip network, but also enable the neural network processing unit on the accelerator side to directly receive dense data that can be used for computation, eliminating the format conversion overhead on the accelerator side and solving the structural contradiction between sparse output on the memory side and dense input on the accelerator side in near-memory computing heterogeneous systems from the source.

[0017] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the various embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details are presented in the embodiments of the present invention to facilitate a better understanding of the invention. However, the technical solutions claimed in the present invention can be implemented even without these technical details and various variations and modifications based on the following embodiments. The division of the following embodiments is for ease of description and should not constitute any limitation on the specific implementation of the present invention. The various embodiments can be combined with and referenced by each other without contradiction.

[0018] One embodiment of the present invention includes: a first rearrangement unit for rearranging sparse input data of the current period into locally compact data; a buffer unit for storing buffer data of the previous period; a control unit connected to the buffer unit for generating an offset control signal based on the buffer data; a second rearrangement unit connected to the first rearrangement unit, the buffer unit, and the control unit for splicing the effective data of the locally compact data and the buffer data based on the offset control signal and outputting spliced ​​data; and an output unit connected to the buffer unit and the second rearrangement unit for extracting data corresponding to the preset transmission bit width from the spliced ​​data as target data for output when the spliced ​​data reaches a preset transmission bit width, and storing the remaining data in the spliced ​​data as buffer data in the buffer unit. In this invention, the sparse data compression circuit transforms the working mode of independently sending a partially loaded transmission unit in each batch in the traditional scheme into a working mode of accumulating and splicing multiple batches of effective data across periods and then sending a fully loaded transmission unit. Meanwhile, due to the significant reduction in the total number of transmission units sent to the on-chip network, the arbitration load and queuing depth of the on-chip network router are also reduced accordingly, effectively alleviating network congestion in high sparsity scenarios. This enables the accelerator side to receive dense and effective data at a more stable rate, reducing the computational idle cycles caused by the accelerator waiting for data.

[0019] One embodiment of the present invention relates to a sparse data compression circuit for near-memory computing, the sparse data compression circuit being disposed on the near-memory controller side between the memory and the on-chip network.

[0020] The memory includes, but is not limited to, dynamic random access memory or high-bandwidth memory. Optionally, the memory can be a memory chip with an integrated local processing unit, which performs matrix-vector multiplication and other operations inside the memory and then stores the sparse computation results back to the memory array.

[0021] The on-chip network is used to connect the memory side and the accelerator side. The accelerator includes, but is not limited to, neural network processing units, general matrix operation accelerators, or in-memory computing arrays.

[0022] In one optional application scenario, this sparse data compression circuit is deployed in a heterogeneous computing system where the neural network processing unit and the near-memory processing unit work together. Located on the data output path of the memory, it performs cross-batch dense compression processing on the sparse computation results generated by the near-memory processing unit before transmitting them to the neural network processing unit via the on-chip network. By performing sparse data compression on the memory side, the neural network processing unit on the accelerator side can directly receive continuous, dense, and effective data without performing additional sparse data unpacking or rearrangement operations on the accelerator side.

[0023] In another optional application scenario, the sparse data compression circuit can also be deployed in a standalone acceleration chip that supports in-memory computing, serving as a front-end processing module for the sparse data output interface, compressing the sparse computation results from the on-chip memory array before sending them to the on-chip interconnect network.

[0024] The following is a detailed description of the implementation details of the sparse data compression circuit for near-memory computation in the embodiments of the present invention. The following content is only for the convenience of understanding the implementation details and is not necessary for implementing this solution.

[0025] like Figure 1 As shown, the sparse data compression circuit 100 for near-memory computing provided in this embodiment includes a first rearrangement unit 110, a second rearrangement unit 120, a buffer unit 130, a control unit 140, and an output unit 150. This sparse data compression circuit 100 is connected between the memory 10 and the on-chip network 20, and is used to receive sparse input data from the memory 10 and output compressed target data to the on-chip network 20. The on-chip network 20 is connected to the accelerator 30, and is used to transmit the target data output by the sparse data compression circuit 100 to the accelerator 30.

[0026] In an alternative example, memory 10 is a memory chip with an integrated near-memory processing unit, accelerator 30 is a neural network processing unit, and sparse data compression circuit 100 is deployed on the near-memory controller side between memory 10 and on-chip network 20, so that the sparse data generated by the near-memory processing unit is compressed into a dense format before entering on-chip network 20, thereby enabling the neural network processing unit on the accelerator 30 side to receive continuous dense effective data with full-load transmission unit granularity.

[0027] like Figure 1 As shown, the output port of memory 10 is connected to the input port of the first rearrangement unit 110. The output port of the first rearrangement unit 110 is connected to part of the input port of the second rearrangement unit 120. The output port of buffer unit 130 is connected to another part of the input port of the second rearrangement unit 120 and to the input port of control unit 140. The output port of control unit 140 is connected to the control port of the second rearrangement unit 120. The output port of the second rearrangement unit 120 is connected to output unit 150. The output port of output unit 150 is connected to on-chip network 20, and the output port of on-chip network 20 is connected to accelerator 30. At the same time, output unit 150 is connected to buffer unit 130 through a feedback path to write the remaining data that has not met the transmission conditions back to buffer unit 130. The overall data path forms a forward transmission link of memory 10, first rearrangement unit 110, second rearrangement unit 120, output unit 150, on-chip network 20, and accelerator 30, as well as a feedback accumulation loop of "output unit 150, buffer unit 130, and second rearrangement unit 120".

[0028] The system includes a first rearrangement unit 110, which rearranges the sparse input data of the current cycle into locally compact data. A buffer unit 130 stores the buffered data from the previous cycle. A control unit 140 generates an offset control signal based on the buffered data. A second rearrangement unit 120 concatenates the locally compact data and the effective data from the buffered data based on the offset control signal, and outputs the concatenated data. An output unit 150, when the concatenated data reaches a preset transmission bit width, extracts data corresponding to the preset transmission bit width from the concatenated data as target data for output, and stores the remaining data in the concatenated data as buffered data in the buffer unit 130.

[0029] In an optional example, the M input ports of the second rearrangement unit 120 are used to access the output ports of at least one first rearrangement unit 110 and at least one buffer unit 130, and the cumulative number of all output ports accessed to the second rearrangement unit 120 is less than or equal to M. In this configuration, the number of output ports of each first rearrangement unit 110 can be the same or different, and the number of output ports of each buffer unit 130 can also be the same or different. This configuration allows the same second rearrangement unit 120 to simultaneously and concurrently receive and process data from multiple first rearrangement units 110 and multiple buffer units 130, thereby increasing the overall data processing throughput of the sparse data compression circuit 100 without increasing the number of second rearrangement units 120.

[0030] In a specific example, buffer unit 130 has a storage capacity of 'a' bytes to store buffered data remaining from the previous cycle or historical cycles. The amount of data output by buffer unit 130 to the second rearrangement unit 120 in each processing cycle is not limited to 'a' bytes; that is, the output data amount of buffer unit 130 can be less than or equal to its storage capacity 'a'. Specifically, when the amount of valid data currently stored in buffer unit 130 is 'b' bytes (b is less than or equal to 'a'), buffer unit 130 only outputs the 'b' bytes of valid data to the corresponding input port of the second rearrangement unit 120, and fills the remaining output ports not occupied by valid data with zero values ​​or invalid identifiers. This design decouples the storage capacity of buffer unit 130 from its actual output amount each time, avoiding bandwidth waste and rearrangement logic redundancy caused by forcibly outputting invalid data when the valid data is insufficient to fill the buffer capacity. For example, if the storage capacity a of the buffer unit 130 is 4 data elements, but only 2 valid data elements are currently stored, only the first 2 positions of the data output by the buffer unit 130 to the second rearrangement unit 120 carry valid data, and the remaining positions are zero values. The offset control signal generated by the control unit 140 accordingly also only reflects the offset of the 2 valid data elements.

[0031] It should be noted that "effective data" in this invention refers to non-zero data elements in the sparse input data, that is, non-zero data that still has practical computational significance after pruning and other sparsification processes. Correspondingly, zero-valued data elements in the sparse input data are invalid data. These zero-valued elements do not participate in subsequent substantive calculations on the accelerator side; they occupy bandwidth during transmission but do not contribute effective information. The core objective of the sparse data compression circuit of this invention is to remove these zero-valued elements, compactly arrange the effective data, and then transmit it via the on-chip network, thereby improving the effective payload fill rate of the transmission unit. In some implementation scenarios, the criteria for determining effective data can also be extended to data elements with an absolute value greater than a preset threshold. That is, near-zero data with an absolute value less than the threshold is also considered invalid data and removed to further improve the compression ratio. In this case, the sparse bitmap generation logic is correspondingly adjusted from strict zero-value judgment to threshold comparison judgment, while the rest of the compression process remains unchanged.

[0032] It is important to note that the buffered data from the previous cycle can include valid data accumulated over one or more historical processing cycles and written back to the buffer unit. Specifically, the buffer unit 130 is used not only to store the buffered data from the previous cycle but also to store historical buffered data. The historical buffered data includes residual valid data from two or more cycles prior to the current processing cycle. Specifically, in scenarios with extremely high sparsity, the amount of valid data generated in each batch across multiple consecutive processing cycles may be very small, and the valid data from a single cycle, even when concatenated with the buffered data from the previous cycle, may still be insufficient to reach the preset transmission bit width. In this case, the output unit 150 writes all valid data from the concatenated data back to the buffer unit 130 through the feedback path after each processing cycle. The data in the buffer unit 130 is the historical buffered data accumulated across multiple processing cycles. As the data accumulates over multiple consecutive cycles, the amount of valid data stored in the buffer unit 130 gradually increases until the total amount of valid data in the concatenated data in a certain cycle reaches or exceeds the preset transmission bit width, at which point the output unit 150 triggers a full-load transmission.

[0033] In this optional embodiment, the storage capacity of the buffer unit 130 needs to be able to accommodate valid data accumulated over multiple cycles. For example, when the preset transmission bit width corresponds to the capacity of N data elements, and the average number of valid data elements generated per cycle is much less than N, the storage capacity 'a' of the buffer unit 130 should be set to no less than N to ensure that all historically accumulated valid data can be completely cached before the final transmission is triggered. The control unit 140 reads the accumulated amount of currently stored valid data in the buffer unit 130 in each processing cycle and generates an offset control signal accordingly, enabling the second rearrangement unit 120 to accurately append locally compact data from a new batch to the historical buffer data. This mechanism ensures that even in extremely sparse scenarios, valid data will not be prematurely sent or discarded due to insufficient quantity in a single batch, but will continue to accumulate in the buffer unit 130, ultimately being efficiently output in the form of a fully loaded transmission unit.

[0034] In a specific example, the first rearrangement unit 110 has N input ports for receiving sparse input data of length N; the second rearrangement unit 120 has M input ports for connecting to the output ports of the buffer unit 130 and the output ports of the first rearrangement unit 110; where M is a natural number greater than N and N is a natural number greater than zero.

[0035] Specifically, the value of N corresponds to the data width of a single burst read from memory 10. For example, N can be 4, and in other implementations, N can also be 8, 16, 32, or other integer powers of 2. Of the M input ports of the second rearrangement unit 120, a portion are used to receive historical buffered data from buffer unit 130, while the remaining input ports are used to receive the current batch data output from the first rearrangement unit 110 after local densification processing. By setting M to a value greater than N, the input capacity of the second rearrangement unit 120 is greater than the data length of a single batch, thus enabling it to simultaneously accommodate historical residual data and new batch data within the same processing cycle, providing a structural basis for cross-batch data splicing. The partitioned connection method of this input port ensures that buffered data and new data naturally occupy two sub-segments when entering the second rearrangement unit 120, matching the divide-and-conquer principle-based switching logic of the butterfly network, allowing direct entry into the rearrangement process without additional data movement or alignment operations.

[0036] In a specific example, the M input ports of the second rearrangement unit 120 are used to access at least one output port of the first rearrangement unit 110 and at least one output port of the buffer unit 130, wherein the cumulative number of all output ports accessed to the second rearrangement unit 120 is less than or equal to M.

[0037] In other words, the sparse data compression circuit 100 supports the concurrent access of multiple first rearrangement units 110 and multiple buffer units 130 on the same second rearrangement unit 120. The number of output ports of each first rearrangement unit 110 is not limited to N, and the number of output ports of each buffer unit 130 is also not limited to N, as long as the cumulative number of all upper-layer output ports connected to the second rearrangement unit 120 does not exceed M. This design allows the sparse data compression circuit 100 to flexibly allocate the number of ports and units of each first rearrangement unit 110 and each buffer unit 130 according to the data port configuration and sparsity characteristics of the memory side in the actual application scenario.

[0038] In an optional example, M equals the product of N and k, where k is an integer greater than 1. Constraining M to be an integer multiple of N ensures that the number of network layers in the second rearrangement unit 120 maintains a strict logarithmic increase relationship with the number of layers in the first rearrangement unit 110, i.e., only log2k layers are added, making the hardware area increment controllable and predictable.

[0039] Taking k = 2 and M = 2N as an example, one possible configuration is to configure one first rearrangement unit 110 (with N output ports) and one buffer unit 130 (with N output ports), with the total number of output ports being 2N, which is exactly equal to M. Taking N = 4 as an example, the second rearrangement unit 120 has 8 input ports, of which 4 are connected to the output ports of the first rearrangement unit 110 and the other 4 are connected to the output ports of the buffer unit 130. Another possible configuration is to configure two first rearrangement units 110 (each with 3 output ports) and one buffer unit 130 (with 2 output ports), with the total number of output ports being 8, which is equal to M. The two first rearrangement units 110 can be connected to different data sources, each independently receiving and rearranging one sparse input data stream. Another optional configuration is to configure one first rearrangement unit 110 (with 4 output ports) and two buffer units 130 (each with 2 output ports), for a total of 8 output ports. In this case, the two buffer units 130 can cache historical data from different sources or different periods, allowing the second rearrangement unit 120 to merge multiple historical data streams with the current data in a single concatenation operation. Furthermore, the total number of upper-layer output ports can be less than M, and unused input ports can be set as default invalid data inputs without affecting the rearrangement logic of the second rearrangement unit 120.

[0040] In another alternative example, M may not be an integer multiple of N. For instance, when M equals 6 and N equals 4, one first rearrangement unit 110 (with 4 output ports) and one buffer unit 130 (with 2 output ports) can be configured, resulting in a total of 6 output ports, equal to M. This configuration is suitable for applications where the amount of buffered data is typically small. By appropriately reducing the number of output ports in the buffer unit 130, the network size and hardware area overhead of the second rearrangement unit 120 can be reduced.

[0041] By concurrently accessing multiple first rearrangement units 110 and buffer units 130 in the same second rearrangement unit 120, the hardware area redundancy caused by configuring a complete second rearrangement unit for each data path is avoided, and an effective balance is achieved between high concurrency processing capability and hardware resource overhead.

[0042] In a specific example, the first rearrangement unit 110 is a multi-layer network structure, and the number of network layers in the first rearrangement unit 110 is the rounded-up value of log2N; the second rearrangement unit 120 is a multi-layer network structure, and the number of network layers in the second rearrangement unit 120 is the rounded-up value of log2M.

[0043] Specifically, the number of network layers in the first rearrangement unit 110 determines the minimum number of swap levels required to completely rearrange an input vector of length N. Taking N equal to 4 as an example, log24 equals 2. The first rearrangement unit 110 contains two network layers. The swap span of the first layer is 2, and the swap span of the second layer is 1. Through two levels of swapping, non-zero elements of arbitrary distribution can be gathered to one end of the vector. The number of network layers in the second rearrangement unit 120 is the floor of log2M. When M equals 2N, log2(2N) equals log2N plus 1, meaning that the second rearrangement unit 120 only adds one network layer compared to the first rearrangement unit 110. The swap span of this additional layer is N, which corresponds exactly to the boundary position between the two sub-segments of buffered data and new data. It is specifically responsible for swapping and aligning valid elements from different data sources across segments globally. This layered design allows the second rearrangement unit 120 to reuse the switching logic of the first log2N layers of the first rearrangement unit 110 to perform local reorganization within each sub-segment, while completing global splicing between sub-segments through additional layers, thus achieving cross-batch rearrangement capability while minimizing hardware increments.

[0044] In a specific example, the first rearrangement unit 110 is a butterfly network, and the second rearrangement unit 120 is a butterfly network.

[0045] Specifically, a butterfly network is a multi-layer switching network based on the divide-and-conquer principle. Each layer contains several two-way switching units, and each switching unit completes data pass-through or cross-transmission under the drive of a control bit sequence. Figure 2 In the butterfly network structure shown, taking a butterfly network with N=8 as an example, this network contains 3 levels. The connection spans between adjacent switching units in each level are 4, 2, and 1, respectively. Progressive aggregation of non-zero elements is achieved by gradually reducing the switching span layer by layer. The control bit sequence of the butterfly network is generated through a left-rotation padding operation. This operation takes the number of switching units W corresponding to the target level and the count of preceding non-zero elements as input parameters. By performing a cyclic left shift and conditional padding operation on the sparse bitmap, the output is a switch configuration value corresponding one-to-one with the switching units of that level. Using a butterfly network to implement the first rearrangement unit 110 and the second rearrangement unit 120 allows the entire densification and splicing process to be completed entirely through hardware combinational logic in a pipelined manner. The latency is fixed and predictable, requiring no software scheduling intervention. Furthermore, the regularized wiring structure of the butterfly network facilitates physical backend layout and wiring optimization, making it suitable for high-density integration under the area constraints of the near-memory controller.

[0046] In an optional embodiment, the first rearrangement unit 110 and the second rearrangement unit 120 may also be implemented using other network structures with equivalent data rearrangement capabilities, including but not limited to Benes networks, Crossbar switching networks, or rearrangement structures based on shift register arrays, as long as the network structure can rearrange the non-zero elements of the input ports to consecutive positions of the output ports according to the control signal.

[0047] In a specific example, the control unit 140 is specifically used to: count the number of valid data in the buffer data; and generate an offset control signal based on the number of valid data in the buffer data.

[0048] Based on this, the function of control unit 140 can be broken down into two sub-steps. In the first sub-step, control unit 140 internally maintains an incrementing counter, which is updated at the end of each processing cycle based on the number of valid data newly written to buffer unit 130 in the current cycle. At the beginning of each processing cycle, control unit 140 reads the current value of the counter, which is the accumulated number of valid data currently stored in buffer unit 130. In the second sub-step, control unit 140 uses this value as an offset parameter and provides it to second rearrangement unit 120 to generate its additional network layer control bit sequence. The physical meaning of this offset parameter is: the starting write address of the first valid element in the newly input locally compact data in the output space of second rearrangement unit 120. By directly mapping the number of valid data in the buffer to an offset, control unit 140 enables second rearrangement unit 120 to complete precise alignment of cross-batch data based on only a scalar parameter without traversing the buffer contents, avoiding the latency overhead and logical complexity caused by element-by-element address calculation.

[0049] In a specific example, the second rearrangement unit 120 is used to: insert locally compacted data after buffered data based on the offset corresponding to the offset control signal to obtain a continuous sequence including locally compacted data and buffered data; or, insert buffered data after locally compacted data based on the offset corresponding to the offset control signal to obtain a continuous sequence including locally compacted data and buffered data.

[0050] Specifically, the two splicing directions mentioned above correspond to the two input port connection configurations and control bit sequence generation strategies of the second rearrangement unit 120, respectively. In the first splicing method, buffered data occupies the low address segment of the input vector of the second rearrangement unit 120, while locally compacted data occupies the high address segment. The offset provided by the control unit 140 indicates the number of bits that the locally compacted data needs to be shifted towards the low address direction, so that the first valid element of the new data is exactly following the last valid element of the buffered data. In the second splicing method, the connection relationship of the input ports is reversed, with locally compacted data occupying the low address segment and buffered data occupying the high address segment. The offset indicates the number of bits that the buffered data needs to be shifted towards the low address direction to complete the splicing. The two methods are functionally equivalent, both of which can achieve continuous splicing of valid data across batches, and the valid elements in the final spliced ​​data are arranged in the same order. Providing two selectable splicing directions allows the sparse data compression circuit 100 to flexibly select the arrangement order of the input ports according to wiring constraints and timing requirements during physical implementation, without being limited by a fixed connection topology.

[0051] In a specific example, the sparse data compression circuit 100 is connected to the accelerator 30 through the transmission channel of the on-chip network 20, and the preset transmission bit width is less than or equal to the bit width of the transmission unit of the transmission channel.

[0052] Specifically, the transmission channel is the data link in the on-chip network 20, and its transmission unit is usually called a flit, which has a fixed bit width. For example, the bit width of a flit can be 256 bits. The preset transmission bit width can be equal to the bit width of a flit, that is, the target data sent by the output unit 150 each time exactly fills a complete flit, thereby achieving full-load transmission of the transmission unit. In another optional configuration, the preset transmission bit width can also be set to a value smaller than the bit width of a flit to adapt to the specific constraints on the effective payload area of ​​the transmission unit in the on-chip network 20 protocol. For example, when the header of a flit needs to carry routing information or control fields, the payload area available to carry effective data is smaller than the total bit width of the flit. In this case, the preset transmission bit width should be set to be aligned with the payload area. By constraining the preset transmission bit width to not exceed the transmission unit bit width, it is ensured that the amount of data sent by the output unit 150 each time transmission is triggered will not exceed the single flit carrying capacity of the on-chip network 20, avoiding packet splitting transmission and additional network arbitration overhead caused by data overflow.

[0053] In a specific example, the output unit 150 is used to store the spliced ​​data as buffer data in the buffer unit 130 when the spliced ​​data does not meet the preset transmission bit width.

[0054] Specifically, when the total amount of valid data in the spliced ​​data output by the second rearrangement unit 120 is less than the data element capacity corresponding to the preset transmission bit width, the output unit 150 determines that the accumulated valid data in the current cycle is insufficient to fill a complete transmission unit. In this case, the output unit 150 does not trigger the transmission operation of the on-chip network 20, but instead writes all valid data in the spliced ​​data back to the buffer unit 130 through the feedback path, waiting for the arrival of a new batch of data in the next processing cycle. Figure 1 As shown, this feedback path corresponds to the dashed line connection between output unit 150 and buffer unit 130. Control unit 140 updates the counter value to the current total amount of valid data for use in the next round of splicing operation. This write-back mechanism ensures that even in scenarios with extremely high sparsity, if the total amount of valid data accumulated over multiple consecutive processing cycles is insufficient to fill the transmission unit, the valid data will not be discarded or prematurely sent. Instead, it will continue to accumulate in buffer unit 130 until a full-load transmission is triggered after reaching the preset transmission bit width. This strategy eliminates the half-load transmission behavior caused by the fragmented distribution of sparse data, ensuring that each flit of on-chip network 20 carries the maximum density of valid data. Furthermore, when the current processing cycle is the last round of the data stream, even if the total amount of valid data in the spliced ​​data does not reach the preset transmission bit width, output unit 150 will still output all the valid data in buffer unit 130 as the target data to ensure that residual valid data at the end of the data stream is not left in the buffer.

[0055] like Figure 3 As shown, the following section, using a specific data processing example, details the collaborative operation of each unit in the sparse data compression circuit 100 during continuous processing cycles. Figure 3In this context, Input Butterfly represents the butterfly network in which the first rearrangement unit 110 performs local densification processing, and Final Butterfly Stage represents the additional network layer in which the second rearrangement unit 120 performs cross-batch data concatenation. Input BF and Final are abbreviations for the butterfly network of the first rearrangement unit 110 and the additional network layer of the second rearrangement unit 120, respectively. Buffer represents the buffered data stored in buffer unit 130. PP-In represents the count value of non-zero elements in the current batch output by the first rearrangement unit 110. PP-Reg represents the counter inside the control unit 140 used to store the cumulative number of valid data. rst represents the operation of the control unit 140 resetting the counter. push out represents the operation of the output unit 150 triggering the output of the target data. Is "full" indicates that output unit 150 determines whether the total effective data volume of the concatenated data has reached the preset transmission bit width; "LROTC" is an abbreviation for left rotation padding operation. "LROTC(W, PP_i)" means that a left rotation padding operation is performed with the number of exchange units W and the count of preceding non-zero elements PP_i as parameters. "Shift×PP_i" means that a shift operation is performed on the sparse bitmap based on the count of preceding non-zero elements PP_i. Assuming N equals 4, that is, the first rearrangement unit 110 has 4 input ports and the second rearrangement unit 120 has 8 input ports (M equals 2N equals 8), and the preset transmission bit width corresponds to the capacity of 4 effective data elements.

[0056] Step 1: Local Denseening Processing of the Current Batch. At the beginning of each processing cycle, memory 10 reads out an original data vector of length N in a fixed burst manner. This vector contains several non-zero valid data elements and several zero-value elements, with the valid data elements discretely distributed in the vector. After receiving this sparse input data, the first rearrangement unit 110 performs a layer-by-layer rearrangement operation on the non-zero elements based on the sparse bitmap of the input data through its internal multi-layer switching network.

[0057] Specifically, the control logic of the first rearrangement unit 110 generates a sparse bitmap based on whether each element in the input data is zero, and performs a left-rotation padding operation on the sparse bitmap to generate the control bit sequence of each level of switching units. For the nth level of the first rearrangement unit 110, the number of its switching units W is equal to 2 raised to the power of (n-1), and the corresponding prefix sum parameter is the cumulative count of all non-zero elements at all positions before this level. Driven by the control bits, each level of switching units sequentially completes the pass-through or cross-transmission of data, causing non-zero elements to continuously gather towards the lower address end of the vector during the layer-by-layer propagation process. In this paper, the first position of the vector corresponds to the lower address end, and the last position corresponds to the higher address end. After processing through the log2N layer, the first rearrangement unit 110 outputs locally compact data of length N, where all non-zero elements are compactly arranged at the lower address end of the vector, and zero-value elements are pushed towards the higher address end.

[0058] by Figure 3 Taking a specific numerical example, assume the input data read from memory 10 in the current cycle is [D, 0, C, 0], where D and C are non-zero valid data elements. The first rearrangement unit 110 generates control bit sequences for each layer based on the sparse bitmap [1, 0, 1, 0]. After two layers of butterfly swapping, the output locally compact data is [D, C, 0, 0], meaning the two non-zero elements are compactly arranged in the first two positions of the vector. Simultaneously, the first rearrangement unit 110 outputs the non-zero element count for the current batch, which is 2 in this example.

[0059] Step 2: Generation of offset control signal. In each processing cycle, the control unit 140 obtains the status information of the current buffered data from the buffer unit 130, and generates the offset control signal required by the second rearrangement unit 120 accordingly.

[0060] For the second rearrangement unit 120, since its total number of input ports is 2N, its additional log2(2N)-th layer, that is, the additional network level compared to the first rearrangement unit 110, has a number of switching units W equal to N. The corresponding prefix and parameter are directly taken as the offset provided by the control unit 140, that is, the number of valid data in the buffered data. Through left-hand rotation and padding, the control bit sequence of this level can accurately indicate that each non-zero element in the newly input locally compact data should be shifted to the target position in the output vector, so that the new data is exactly followed by the valid data in the buffered data, achieving one-to-one mapping alignment across batches.

[0061] From a technical perspective, the essence of this offset control mechanism lies in converting the number of valid positions already occupied in the buffered data into the starting offset address of the new data in the output space of the second rearrangement unit 120. This allows the additional network layers of the second rearrangement unit 120 to shift the new data to the correct splicing position as a whole. This mechanism ensures that, in multiple consecutive processing cycles, different batches of valid data in the output buffer always follow a strictly increasing continuous filling pattern, preventing element loss, overlap, or misalignment.

[0062] Continue with Figure 3 The example shown illustrates this. Assume that prior to the current processing cycle, buffer unit 130 already stores buffered data [0, 0, B, A] from the previous cycle, where B and A are valid data elements located at the high address end of the buffer. Control unit 140 reads a counter value of 2, indicating that there are currently 2 valid data elements in the buffer. Control unit 140 provides this offset value of 2 to the second rearrangement unit 120 to generate the control bit sequence for the additional level of the second rearrangement unit 120.

[0063] Step 3: Cross-batch data concatenation. The first N input ports of the second rearrangement unit 120 receive buffered data from the buffer unit 130, and the remaining N input ports receive locally compacted data output from the first rearrangement unit 110. From the perspective of the second rearrangement unit 120, the buffered data and the locally compacted data are respectively regarded as the left and right sub-segments of the input vector. The second rearrangement unit 120 uses its log2(2N) layer switching network to perform global rearrangement on these two sub-segments under the drive of the offset control signal provided by the control unit 140. The N switching units in this layer shift the non-zero elements from the first rearrangement unit 110 to consecutive positions in the output vector immediately after the valid elements of the buffered data, based on the control bit sequence calculated by the offset. After this layer of processing, the second rearrangement unit 120 outputs concatenated data of length 2N, in which all valid data elements from different batches are compactly arranged at one end of the vector in chronological order, forming a continuous and uninterrupted dense sequence.

[0064] Continue with Figure 3 The example shown illustrates this. The first four input ports of the second rearrangement unit 120 receive buffered data [0, 0, B, A], and the last four input ports receive locally compact data [D, C, 0, 0]. Under the control of an offset of 2, the additional level of the second rearrangement unit 120 shifts D and C to the 3rd and 4th positions (counting from right to left) of the output vector, respectively, so that the concatenated output data is [0, 0, 0, 0, D, C, B, A], that is, the four valid data elements D, C, B, and A are compactly arranged at the low address end of the vector, forming a continuous dense sequence.

[0065] Step 4: Threshold Decision and Output Control. After the splicing operation is completed in each processing cycle, the output unit 150 compares the total amount of valid data in the spliced ​​data with the preset transmission bit width, and executes the corresponding data splitting strategy based on the comparison result.

[0066] When the total amount of valid data in the spliced ​​data equals the data element capacity corresponding to the preset transmission bit width, the output unit 150 uses all the valid data in the spliced ​​data as the target data and sends it to the accelerator side via the transmission channel of the on-chip network 20 in the form of a full-load transmission unit. After the transmission is completed, the control unit 140 resets the counter to zero, the buffer unit 130 is cleared, and the sparse data compression circuit 100 enters the initial state of the next round of processing.

[0067] When the total amount of valid data in the concatenated data exceeds the data element capacity corresponding to the preset transmission bit width, the output unit 150 extracts N valid data elements corresponding to the preset transmission bit width at the lower address end of the concatenated data as target data for output, and writes the remaining valid data elements into the buffer unit 130 through the feedback path. In a specific implementation, the buffer unit 130 includes a first output buffer and a second output buffer. When the total amount of valid data in the concatenated data exceeds N, the lowest N valid data elements are written into the first output buffer and immediately pushed to the on-chip network 20, while the valid data elements exceeding N are written into the second output buffer, which remains compactly arranged. When processing the next round of input, the first and second output buffers are logically swapped, so that the residual data in the second output buffer is used as the buffered data input for the new round of concatenation operation. This logical swap can be implemented through a two-way selector and a bit configuration register. At the same time, the control unit 140 performs a bit truncation update on the counter value, that is, updates the counter value to the remainder after subtracting N from the total amount of valid data, thereby directly adapting to the case of unaligned overflow.

[0068] Continue with Figure 3 The example shown illustrates this. After the concatenation in step three, the total amount of valid data in the concatenated data is 4 (D, C, B, A), which is exactly equal to the capacity N corresponding to the preset transmission bit width, which is 4. Therefore, the output unit 150 uses these 4 valid data elements [D, C, B, A] as target data and pushes them to the on-chip network 20 in a full-load transmission unit manner. After the transmission is completed, the control unit 140 resets the counter to zero, the buffer unit 130 is cleared, and the sparse data compression circuit 100 prepares to receive the next batch of sparse input data.

[0069] In an optional embodiment, the control bit generation logic of the first rearrangement unit 110 and the second rearrangement unit 120 can share the same set of left-hand padded operation hardware, generating control bit sequences for the two-level networks respectively through time-division multiplexing. Specifically, in the processing stage of the first rearrangement unit 110, the left-hand padded operation hardware uses the sparse bitmap of the current batch and the feedforward sum of each level as input to generate the control bit sequences of each level of the first rearrangement unit 110. In the processing stage of the second rearrangement unit 120, the operation hardware uses the effective data quantity of the buffered data as an offset parameter to generate the control bits of the additional levels of the second rearrangement unit 120. By sharing the control logic hardware, the area overhead of the sparse data compression circuit 100 can be further reduced, making it suitable for near-memory computing scenarios with strict chip area constraints.

[0070] In an optional embodiment, the sparse data compression circuit 100 may further incorporate a pipeline register to divide the processing of the first rearrangement unit 110 and the second rearrangement unit 120 into different pipeline stages. Specifically, a pipeline register is inserted between the output port of the first rearrangement unit 110 and the input port of the second rearrangement unit 120, allowing the processing of the i+1th batch of data by the first rearrangement unit 110 and the concatenation operation of the i-th batch of data by the second rearrangement unit 120 to overlap in time. This pipeline design can shorten the effective processing cycle of the sparse data compression circuit 100 to the propagation delay of a single-stage network, further improving data throughput in high-frequency operating scenarios.

[0071] In embodiments of the present invention, the sparse data compression circuit 100 transforms the traditional working mode of independently sending one incomplete transmission unit per batch into a working mode of accumulating and splicing multiple batches of valid data across cycles before sending a full-load transmission unit. Simultaneously, because the total number of transmission units sent to the on-chip network 20 is significantly reduced, the arbitration load and queuing depth of the router in the on-chip network 20 are also correspondingly reduced, effectively alleviating network congestion in high sparsity scenarios. This allows the accelerator 30 to receive dense valid data at a more stable rate, reducing the computational idle cycles of the accelerator 30 while waiting for data.

[0072] It is worth mentioning that all modules involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this invention, this embodiment does not introduce units that are not closely related to solving the technical problem proposed by this invention; however, this does not mean that other units are absent from this embodiment.

[0073] In the description of the embodiments of this invention, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this invention, "multiple" means two or more, unless otherwise explicitly defined.

[0074] In the description of the embodiments of the present invention, unless otherwise explicitly specified and limited, the technical terms such as "installation," "connection," "joining," and "fixing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in the embodiments of the present invention according to the specific circumstances.

[0075] Furthermore, the examples mentioned in the above embodiments can be freely combined, and any combination can be understood as an embodiment. The terms "embodiment" or "example" appearing in various locations in the specification do not necessarily refer to the same embodiment, nor are they independent or alternative embodiments mutually exclusive with other embodiments. Those skilled in the art will understand that the embodiments described herein can be combined with other embodiments.

[0076] Another embodiment of the present invention relates to a near-memory computing chip, comprising: a sparse data compression circuit 100 for near-memory computing as described above; a memory 10 connected to the sparse data compression circuit 100; an on-chip network 20 connected to the sparse data compression circuit 100; and an accelerator 30 connected to the on-chip network. The sparse data compression circuit 100 is configured to receive sparse input data from the memory 10 and output target data to the on-chip network 20; the on-chip network 20 is configured to provide the target data to the accelerator 30.

[0077] It is not difficult to see that this embodiment is a device embodiment corresponding to the above method embodiments, and this embodiment can be implemented in conjunction with the above method embodiments. The relevant technical details mentioned in the above method embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above method embodiments.

[0078] Those skilled in the art will understand that the above embodiments are specific embodiments for implementing the present invention, and in practical applications, various changes in form and detail may be made without departing from the spirit and scope of the present invention.

Claims

1. A sparse data compression circuit for near-memory computing, the sparse data compression circuit comprising: include: The first rearrangement unit is used to rearrange the sparse input data of the current period into locally compact data; A buffer unit is used to store buffered data from the previous cycle; A control unit, connected to the buffer unit, is used to generate an offset control signal based on the buffer data; The second rearrangement unit, connected to the first rearrangement unit, the buffer unit, and the control unit, is used to splice the effective data of the locally compact data and the buffer data based on the offset control signal, and output the spliced ​​data. The output unit, connected to the buffer unit and the second rearrangement unit, is used to extract data corresponding to the preset transmission width from the spliced ​​data as target data and output it when the spliced ​​data reaches the preset transmission width, and store the remaining data in the spliced ​​data as buffer data in the buffer unit.

2. The sparse data compression circuit for near-memory computing of claim 1, wherein, The first rearrangement unit has N input ports for receiving sparse input data of length N; The second rearrangement unit has M input ports for connecting to the output ports of the buffer unit and the output ports of the first rearrangement unit; Where M is a natural number greater than N, and N is a natural number greater than zero.

3. The sparse data compression circuit for near-memory computing according to claim 1, characterized in that, The output unit is used to store the spliced ​​data as buffer data into the buffer unit when the spliced ​​data does not meet the preset transmission bit width.

4. The sparse data compression circuit for near-memory computing according to claim 1, characterized in that, The control unit is specifically used to: count the number of valid data in the buffer data; and generate the offset control signal based on the number of valid data in the buffer data.

5. The sparse data compression circuit for near-memory computing according to claim 4, characterized in that, The second rearrangement unit is used for one of the following: Based on the offset corresponding to the offset control signal, the locally compacted data is inserted after the buffered data to obtain a continuous sequence including the locally compacted data and the buffered data; or, Based on the offset corresponding to the offset control signal, the buffered data is inserted after the locally compacted data to obtain a continuous sequence including the locally compacted data and the buffered data.

6. The sparse data compression circuit for near-memory computing according to claim 2, characterized in that, The M input ports of the second rearrangement unit are used to connect to at least one output port of the first rearrangement unit and at least one output port of the buffer unit; wherein the cumulative number of all output ports connected to the second rearrangement unit is less than or equal to M.

7. The sparse data compression circuit for near-memory computing according to claim 2, characterized in that, The number of network layers in the first rearrangement unit is the rounded-up value of log2N; The number of the second rearranged unit network layers is the rounded-up value of log2M.

8. The sparse data compression circuit for near-memory computing according to any one of claims 1 to 7, characterized in that, The first rearrangement unit is a butterfly network, and the second rearrangement unit is a butterfly network.

9. The sparse data compression circuit for near-memory computing according to any one of claims 1 to 7, characterized in that, The sparse data compression circuit is connected to the accelerator via an on-chip network transmission channel. The preset transmission bit width is less than or equal to the bit width of the transmission unit of the transmission channel.

10. A near-memory computing chip, characterized in that, include: Sparse data compression circuit for near-memory computation as described in any one of claims 1 to 9; The memory connected to the sparse data compression circuit; On-chip network connected to the sparse data compression circuit; Accelerators connected to the on-chip network; The sparse data compression circuit is used to receive sparse input data from the memory and output target data to the on-chip network. The on-chip network is used to provide the target data to the accelerator.

Images