Transformer in-memory computing circuit supporting dynamic key-value caching and filtering and applications thereof

Abstract

The present application relates to the field of integrated circuits, and particularly to a Transformer in-memory computing circuit supporting dynamic key-value caching and screening and an application thereof. It comprises an attention weight calculation unit, a second CIM array, a score memory, an addition tree, a KV Cache, a screener and a loader. The attention weight calculation unit and the second CIM array constitute the basic circuit for implementing attention value calculation in this embodiment; the score memory, the addition tree, the KV Cache, the screener and the loader are used to directly evaluate the key values with greater contribution on the hardware level by using the intermediate results generated in the attention calculation process, and based on the prediction and gating strategy of partial calculation results, to suppress or skip the redundant calculation and access of low-contribution key value data in the early stage of calculation, thereby reducing the overall power consumption and reducing the calculation delay. The present application solves the problem that the existing in-memory computing circuit cannot be applied to attention calculation tasks with gradually increasing sequence length and key-value pairs.

Description

Technical Field

[0001] The invention relates to the field of integrated circuits, and in particular to a Transformer in-memory computing circuit that supports dynamic key-value caching and filtering, a CIM chip, and its application in data processing tasks involving self-attention computing. Background Technology

[0002] With the continuous advancement and development of deep learning technology, Large Language Models (LLMs) based on the Transformer architecture have become the mainstream model architecture in natural language processing, computer vision, and complex logical reasoning tasks. Transformer models use a self-attention mechanism to weighted model information at different positions in a sequence, giving them a significant advantage in understanding long-distance dependencies. However, during the model inference stage, especially when using an auto-regressive decoding approach, this architecture also brings significant storage and computational burdens.

[0003] During Transformer decoding, each new token generated requires association calculation with the key vector (K) and value vector (V) corresponding to all historical tokens. To avoid repeatedly calculating historical Keys and Values ​​at each time step, existing technologies commonly introduce a Key-Value Cache (KV Cache) mechanism to cache key-value pairs generated at previous times in memory for later reuse. This mechanism can effectively reduce computational redundancy in short-sequence inference, but in application scenarios such as long-sequence generation, long-text understanding, and multi-turn dialogue, the size of the KV Cache will continue to grow with the number of generation steps, and its storage resource consumption will increase linearly or even superlinearly.

[0004] To alleviate the performance and energy efficiency issues caused by the memory wall, compute-in-memory (CIM) technology has received widespread attention in recent years. This technology tightly couples computing units with the storage array, performing multiplication and addition operations directly within or near the storage array, thus significantly reducing the overhead of data transfer between storage and computing units. However, when processing neural network data processing tasks, existing CIM architectures are mostly designed for applications with static or quasi-static weights, lacking dedicated hardware management mechanisms for the dynamic growth characteristics of KV caches. This makes it difficult to efficiently filter and evict historical key-value pairs while writing new ones. When the cache size exceeds the on-chip SRAM capacity, the system still needs to rely on off-chip storage, making it difficult to maintain the energy efficiency advantage of CIM architectures in long-sequence inference. The large number of key-value pairs stored also increases the computational scale of attention calculations, further reducing the data processing efficiency of CIM circuits for neural network tasks.

[0005] Therefore, how to provide a new computing architecture to accelerate attention computing while ensuring computational accuracy has become a technical challenge that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] To address the shortcomings of existing in-memory computing circuits in attention computing tasks, this invention provides a Transformer in-memory computing circuit that supports dynamic key-value caching and filtering, a CIM chip, and its application in data processing tasks involving self-attention computing.

[0007] The technical solution provided by this invention is as follows: A Transformer in-memory computation circuit that supports dynamic key-value caching and filtering includes: an attention weight calculation unit, a second CIM array, a score memory, an addition tree, a KV cache, a filter, and a loader.

[0008] The attention weight calculation unit includes a first CIM array, a softmax module, and a quantization module, used to calculate the attention weights of the input query vector Q and each pre-stored key vector K. The second CIM array is used to multiply each pre-stored value vector V with its corresponding attention weight to obtain the attention value.

[0009] The score memory comprises several storage units; each storage unit stores the historical accumulated values ​​of the attention weights for each key vector K. In the score memory, half of the storage units serve as the new addition area, and the other half as the key area. The key area stores the highest historical accumulated values ​​of several attention weights; the attention weights of each new key vector K are cyclically updated in the new addition area in chronological order.

[0010] The addition tree is used to cumulatively update each storage unit using the attention weight calculated in each round as the increment. The KV cache is used to store all historical key-value pairs saved in each round and their position indices in the first and second CIM arrays.

[0011] The filter is used after each round of attention calculation to compare the attention weight of the earliest stored key vector K in the newly added region with the attention weight of the region with the smallest value, and then makes the following decision based on the comparison result: (1) If the former is greater than the latter, the former is used to cover the latter in the critical area, and a deletion mark is added to the position index of the latter in the critical area of ​​KV Cache.

[0012] (2) If the former is not greater than the latter, then add a deletion mark to the position index of the former in the new area of ​​KV Cache.

[0013] The loader receives the new Q, K, and V values ​​in each round and pre-stores the new K and V values ​​in the indexes of the positions containing the deletion markers in the first and second CIM arrays (i.e., replacing the previously filtered key-value pairs with lower attention weights with the new key-value pairs); then it performs a new round of attention calculation.

[0014] As a further improvement of the present invention, the Transformer in-memory computation circuit that supports dynamic key-value caching and filtering also includes a result memory, which is used to store the current Q and the attention values ​​of all key-value pairs obtained after each round of attention computation.

[0015] As a further improvement of this invention, the attention weight calculation unit includes: a first CIM array, a Softmax module, and a quantization module. The first CIM array is used to calculate the dot product between the input query vector Q and each pre-stored transposed key vector K. The Softmax module is used to perform Softmax processing on the output of the first CIM array. The quantization module is used to quantize the output of the Softmax module to obtain the attention weight of the corresponding key vector K.

[0016] As a further improvement of the present invention, both the first CIM array and the second CIM array include a storage array and a computing module.

[0017] Let the number of valid key-value pairs in the attention computation task be N, and the vector length be M. Then the storage array is composed of an array of SRAM cells arranged in N rows and M columns. Each SRAM cell in each row of the storage array is used to pre-store the value of each key vector K or value vector V bit by bit.

[0018] The calculation module includes a multi-bit multiplication operation module consisting of multiple two-to-one selectors and a shift accumulator. The multi-bit multiplication operation module takes the values ​​of each bit of the query vector Q as input and the values ​​of each bit of the pre-stored key vector K as weights, and then calculates the product of each bit of the input and each bit of the weights. Then, in conjunction with the shift accumulator, the result of the multiplication operation of the multi-bit input and the multi-bit weights is obtained.

[0019] As a further improvement of the present invention, the calculation module in the first CIM array adopts a multiplication module capable of performing multiplication operations between M / 2-bit inputs and M / 2-bit weights. It consists of M / 2+1 2-to-1 multiplexers, including one M / 2-bit 2-to-1 multiplexer MUX0 and M / 2 single-bit 2-to-1 multiplexers mux; the circuit connection is as follows: The two inputs of MUX0 are connected to the high M / 2 bits and low M / 2 bits of the weight data, respectively. The output of MUX0 outputs a strobed M / 2 bit result, and the values ​​of each bit of this output result are used as inputs to one of the M / 2 muxes. The other input of each mux is connected to the values ​​of each bit of the input data. The output of each mux is used to output the product of the corresponding bit of the weight and the corresponding bit of the input.

[0020] As a further improvement of this invention, the Transformer in-memory computation circuit supporting dynamic key-value caching and filtering also includes a prediction unit. The prediction unit first obtains the Softmax result corresponding to the dot product of the high M / 2 bits of the calculated query vector Q and the high M / 2 bits of each key vector K, and then determines whether the Softmax result is greater than e. -5 : (1) If so, continue to calculate the attention weight corresponding to the full-precision dot product of the query vector Q and the key vector K, and use it as the required attention weight.

[0021] (2) Otherwise, the Softmax result at that position is determined to be 0 after quantization, the corresponding attention weight is directly reset to 0, and subsequent low-bit calculations are skipped to reduce redundant operations.

[0022] As a further improvement of the present invention, in the Transformer in-memory computing circuit that supports dynamic key-value caching and filtering, the first N cycles of the application phase are the initialization period; in the first N / 2 cycles, since a new set of Q, K, V is generated in each cycle, the new area is gradually filled from blank; in the N / 2+1 cycle, the entries in the new area are moved to the key area in sequence, the data at the corresponding address is updated, and the new area is refilled; in the Nth cycle, the new area is filled again.

[0023] Starting from the N+1th period, a key-value pair dynamic update period begins. Based on the attention weight of the earliest key-value pair in the newly added region and the minimum attention weight in the key region, a key-value pair is selected and discarded in the next period.

[0024] As a further improvement of the present invention, the first N cycles of the application phase are the initialization period; wherein, in the first N / 2 cycles, each newly added attention weight is stored in the key area in sequence; in the latter N / 2 cycles, each newly added attention weight is stored in the new area in sequence. Starting from the N+1th period, the key-value pair dynamic update period begins. The newly added attention weights are updated cyclically in the new addition area in chronological order. Key-value pairs to be discarded are selected based on the relationship between the attention weight of the earliest key-value pair in the new addition area and the minimum attention weight in the key area, and are deleted in the next cycle.

[0025] As a further improvement of the present invention, the filter includes a minimum value search module and a comparator; the minimum value search module performs a minimum global search on each storage unit of the key region only after the minimum attention weight in the key region has been covered.

[0026] The comparator is used to compare the attention weight corresponding to the earliest key vector K after the new region is updated with the minimum attention weight of the key region in each cycle.

[0027] The present invention also includes a CIM chip, which is packaged from the aforementioned Transformer in-memory computing circuit that supports dynamic key-value caching and filtering.

[0028] The present invention also includes the application of the aforementioned CIM chip in data processing tasks involving self-attention computation.

[0029] The present invention has the following beneficial effects: This invention introduces a dynamic filtering and management mechanism for key-value cache (KV Cache) oriented towards autoregressive inference processes at the hardware level. By continuously evaluating and controlling the contribution of historical key-value data through a score memory and a filter, it effectively alleviates the storage pressure caused by the growth of key-value cache as the sequence length increases and improves the utilization efficiency of limited on-memory cache resources.

[0030] Meanwhile, this invention combines prediction and gating strategies based on partial calculation results in the attention weight calculation process, which can suppress or skip redundant calculations and accesses to low-contribution key-value data in the early stages of calculation, thereby reducing overall power consumption and computational latency.

[0031] During each round of loading of valid key-value pairs, this invention utilizes the location index information stored in the KV Cache to update only the single data that needs to be covered, without needing to globally update the weights pre-stored in the CIM array. This helps reduce the pressure of data movement.

[0032] In summary, without compromising model inference accuracy, this invention significantly improves the overall performance, energy efficiency, and system scalability of the Transformer decoder in long sequence inference scenarios. Attached Figure Description

[0033] Figure 1 This is an architecture diagram of the Transformer in-memory computing circuit that supports dynamic key-value caching and filtering, as provided in Embodiment 1 of the present invention.

[0034] Figure 2 This is a schematic diagram of the attention weight calculation unit, which includes the shift module.

[0035] Figure 3 The circuit schematics are for the first and second CIM arrays, which contain storage arrays and computing modules.

[0036] Figure 4 This is a circuit diagram of the filter used in Embodiment 1 of the present invention.

[0037] Figure 5 This is an architecture diagram of the in-memory computing circuit that introduces a prediction unit to achieve half-precision arithmetic, as provided in Embodiment 2 of the present invention.

[0038] Figure 6 For simulation testing, the circuit of this invention implements the signal timing diagram for filtering, decision-making, and submission of KV Cache.

[0039] Figure 7 For simulation testing, the timing diagram of the signal for updating the mapping relationship and aligning the output with the column of the 16×204CIM array is provided.

[0040] Figure 8 In simulation testing, the circuit of this invention implements the joint timing of KV Cache filtering and KV mapping update.

[0041] Figure 9 In simulation testing, the circuit of this invention utilizes a prediction unit to implement a signal timing diagram that skips low-order operations. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the specification of this invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "or / and" as used herein includes any and all combinations of one or more of the associated listed items.

[0044] Example 1 Existing hardware circuits with attention calculation capabilities typically perform full-coverage calculations on all historical key-value pairs, failing to effectively utilize the sparsity of attention scores at the hardware level. Low-contribution key-value pairs are still redundantly calculated and retained for extended periods, leading to a continuously expanding computational scale. This not only reduces computational efficiency but also poses challenges to circuit design and computational management. To address this issue, this embodiment provides a Transformer in-memory computation circuit that supports dynamic key-value caching and filtering. This in-memory computation circuit can evaluate the attention contribution of each historical key-value pair at the hardware level. Through a replacement and update mechanism, under conditions of limited cache resources, it prioritizes retaining key-value pairs that contribute significantly to the current and subsequent inference processes, discarding some low-contribution key-value pairs. This ensures that limited in-memory computation and storage resources can prioritize serving high-contribution key-value data while maintaining the accuracy of attention calculations. This reduces storage pressure and redundant computational overhead during autoregressive inference, improving overall system energy efficiency and scalability.

[0045] Specifically, this embodiment constructs an in-memory computing array (first CIM array and second CIM array) that supports dynamic writing and parallel computing, so that key vectors and value vectors can be directly written into the in-memory computing unit during the autoregressive generation process, and participate in the calculation as in-memory weights in the subsequent attention weight and attention value calculation, thereby reducing the frequent movement of key-value data between the storage unit and the computing unit and improving the access efficiency of KV Cache during the inference process.

[0046] In the key-value cache filtering process, this embodiment introduces a prediction and gating mechanism based on partial calculation results. During the dot product calculation stage, intermediate results with a partial bit width are calculated first, and the contribution of historical key values ​​to the current output is evaluated based on the prediction results. Calculation paths corresponding to key-value data with low contribution are suppressed or skipped, thereby reducing invalid calculations and accesses to low-contribution KVs.

[0047] At the KV Cache management level, this embodiment constructs a cache filtering and update mechanism based on the cumulative attention weight contribution. This mechanism continuously evaluates the contribution of historical key-value data across multiple time steps and dynamically allocates and adjusts the in-memory cache space based on the accumulated attention weight. This ensures that limited in-memory computing and storage resources prioritize key-value data that contributes significantly to the current and subsequent inference processes. When a key-value pair arrives, it is first fed into the first CIM array for calculation, and then passes through a Softmax unit to obtain the attention weight. The score memory stores 204 data points, all of which are the result of accumulating the attention weights from previous time periods. When the 102 data points in the new addition area begin to overflow, the earliest data point in the new addition area is removed, and its placement in the critical area is evaluated. The evaluation method involves taking the earliest data point in the new addition area for each time period, comparing it with the data point with the smallest value in the critical area, and using the larger value to overwrite the smallest value in the critical area; the other is discarded.

[0048] In detail, to achieve the above functions, such as Figure 1 As shown, the circuit modules in the Transformer in-memory computing circuit that supports dynamic key-value caching and filtering provided in this embodiment include: an attention weight calculation unit, a second CIM array, a score memory, an addition tree, a KV cache, a filter, and a loader.

[0049] The attention weight calculation unit adopts an in-memory computing architecture, which includes a first CIM array. The attention weight calculation unit calculates the attention weights between the input query vector Q and the pre-stored key vectors K. The formula for calculating the attention weight U is: ; In the above formula, softmax(·) represents the Softmax operation; d k This indicates the dimension of the word vector after embedding.

[0050] In practical applications, the attention weight calculation unit in this embodiment can consist of three parts: a first CIM array, a Softmax module, and a quantization module. The first CIM array is used to calculate the dot product of the input query vector Q and the pre-stored transposed key vectors K. That is, based on the vectors Q and K, QK is obtained. T The calculation results are as follows. Both the query vector Q and the K vector can be represented as multi-bit numbers. In this embodiment, the first CIM array only needs to employ a circuit module that can perform in-memory calculations capable of multiplying multi-bit numbers.

[0051] The Softmax module is used to perform Softmax processing on the output of the first CIM array. In this embodiment, the Softmax module can employ various existing circuit modules with corresponding data processing functions. It should be noted that the dimensionality of the embedded word vectors in the circuit... d k In the task of calculating attention weights U, which is fixed and ensures accuracy, QK T Divide by the dimension of the embedded word vector d k The square root (i.e.) The operation can be simplified to a shift operation of a specified number of bits. The Softmax module in this embodiment can optionally integrate a shift operation to achieve this. The circuit module for computation. Of course, when using a Softmax module without shift operations, it can also be done as follows: Figure 2 As shown, a shift module is added to the attention weight calculation unit and placed between the first CIM array and the Softmax module.

[0052] The quantization module is used to quantize the output of the Softmax module to obtain the attention weight U of the corresponding key vector K. In this embodiment, the output of the Softmax module is typically an analog electrical signal. The function of the quantization module in this embodiment is to perform analog-to-digital conversion on the output signal of the Softmax module to obtain the corresponding digital quantity for subsequent calculations or analysis.

[0053] After obtaining the attention weights based on the query vector Q and the key vector K, multiplying them by the value vector V yields the attention value Attention(Q,K,V): .

[0054] In this embodiment, the second CIM array is used to multiply each pre-stored value vector V with its corresponding attention weight to obtain the attention value. Similar to the first CIM array in the attention weight calculation unit, in order to efficiently process multi-bit multiplication tasks in batches, this embodiment also uses a computing module based on an in-memory computing architecture, namely the second CIM array, when performing the multiplication task between attention weights and value vectors V.

[0055] In practical applications of this embodiment, the first CIM array and the second CIM array can use the same circuit modules. The difference between the two lies in the data loading method. Specifically, the first CIM array needs to complete the transpose of Q and K (i.e., K). T The first CIM unit performs a multiplication operation between K and V, while the second CIM unit needs to perform a multiplication operation between U and V. K and V are pre-stored in the CIM array in the form of weights in both CIM arrays before the subsequent multiplication operation is performed. Therefore, the loading methods for K and V differ between the two CIM arrays.

[0056] Specifically, at the hardware implementation level, such as Figure 3 As shown, both the first and second CIM arrays include a storage array and a computing module. Assuming the number of valid key-value pairs in the attention calculation task is N, and the vector length is M, then in each round of attention calculation, the circuit needs to complete the attention value calculation task between the newly input query vector Q and N different sets of K, V data pairs. To address this computational requirement, the storage array in the CIM array of this embodiment can be arranged as an N-row, M-column SRAM cell array; each SRAM cell in each row of the storage array is used to pre-store the value of each key vector K or value vector V bit by bit.

[0057] Each SRAM cell stores a single bit of data from either the key vector K or the value vector V. Each SRAM cell in the same row of the storage array stores exactly one key vector K or value vector V. The entire storage array can store N key vectors K (denoted as K1~K1) of length M. N ) or N value vectors V (denoted as V1~V) of length M. N ).

[0058] The computation modules in the first and second CIM arrays include a multi-bit multiplication module composed of multiple 2-to-1 selectors and a shift accumulator. The multi-bit multiplication module takes the values ​​of each bit of the query vector Q as input and the values ​​of each bit of the pre-stored key vector K as weights, then calculates the product of each bit of the input and each bit of the weights. The shift accumulator then yields the result of the multiplication of the multi-bit input and the multi-bit weights. Of course, in the computation module based on the CIM circuit architecture, the aforementioned multi-bit multiplication module can be implemented using NMOS transistors as transmission transistors, or designed using transmission gates, in addition to the aforementioned 2-to-1 selectors. This embodiment does not limit the circuit structure and principle of the multi-bit multiplication operation used in the computation module, as long as it can complete the required computation task.

[0059] The score memory in this embodiment includes several storage units; when the number of valid key-value pairs set in the circuit is N, the number of storage units in the score memory in this embodiment is N. Each storage unit is used to store the historical accumulated value of the attention weights for each key vector K.

[0060] Specifically, to evaluate the contribution of different key-value pairs in attention calculation, this embodiment divides each storage unit in the score memory into two partitions. One half of the storage units serves as the new addition area, comprising N / 2 storage units. The other half serves as the key area, also comprising N / 2 storage units.

[0061] In this embodiment, the key region is used to store several historically accumulated attention weights with relatively high values. The attention weights of the newly added key vector K are cyclically updated in the new region in chronological order. Under this mechanism, when the new region is full, each time a new key-value pair is added, it means that the historically accumulated attention weight calculated from the first key-value pair stored in the new region will "overflow" from the new region. At this time, this embodiment needs to consider the "historical contribution" of this overflowing key-value pair in the attention calculation, and then determine whether it needs to be retained. If its contribution is large, it should be retained, and this embodiment can transfer it to the key region. If its historical contribution is small, this embodiment considers discarding it directly and no longer allowing it to participate in subsequent attention calculations. The process of evaluating the historical contribution of each key-value pair and deciding whether to retain or discard it based on the evaluation results will be implemented jointly by an addition tree, a filter, a KV cache, and a loader, which will be explained in detail below.

[0062] In this embodiment, the historical contribution of each key-value pair in attention calculation can be determined by the magnitude of the historical accumulated value of the attention weight U obtained in each round of computation. Therefore, in this embodiment, the addition tree can be composed of multiple adders, which are used to cumulatively update each storage unit with the attention weight calculated in each round as the increment. That is, the addition tree extracts the values ​​from each storage unit in the original score memory, adds them to the new value calculated in the current round, and uses this as the new value of the historical accumulated value of the attention weight of the key-value pair after the end of the current round of computation, and stores it back into each storage unit of the score memory.

[0063] Furthermore, it is important to emphasize that since a key-value pair with a small attention contribution is selected and discarded in each round, when the addition tree updates the historical cumulative values ​​of the attention weights of each key-value pair, it will only update the data of the N-1 key-value pairs that were not discarded. The stored data of the key-value pair that was selected to be discarded will be overwritten by the attention weights of the newly added key-value pairs in this round.

[0064] The KV Cache is used to store all historical key-value pairs saved in each round and their position indices in the first and second CIM arrays. Specifically, the first function of the KV Cache in this embodiment is to store all historical key-value pairs saved in each round (up to N pairs), thereby enabling the key vector K and value vector V of the key-value pairs to be injected into the first and second CIM arrays respectively through the loader, so as to participate in the next round of attention calculation. In practical applications, considering that only one key-value pair evaluated as having the lowest contribution in the previous round will be discarded before each round of attention calculation, the first and second CIM arrays are pre-stored in each key-value pair in the CIM array in adjacent rounds of calculation, and only one key-value pair needs to be updated. Based on this situation, the KV Cache in this embodiment can also store the position indices of all historical key-value pairs saved in each round in the first and second CIM arrays. Under these conditions, when the subsequent loader injects key-value pairs before each round of attention calculation, it only needs to replace the key-value pair that was selected and discarded with the newly added key-value pair in this round based on the known position index, without having to perform a global update on all the key-value pairs pre-stored in the first and second CIM arrays.

[0065] The filter's role is to evaluate the historical contribution of each key-value pair in the attention computation and find the one with the smallest contribution that should be discarded. Specifically, such as... Figure 4 As shown, the filter implements this filtering function as follows: After each round of attention calculation, the filter compares the attention weight of the earliest stored key vector K in the newly added region with the attention weight of the key region with the smallest value, and then makes the following decision based on the comparison result: (1) If the former is greater than the latter, the former is used to cover the latter in the critical area, and a deletion mark is added to the position index of the latter in the critical area of ​​KV Cache.

[0066] (2) If the former is not greater than the latter, then add a deletion mark to the position index of the former in the new area of ​​KV Cache.

[0067] The underlying mechanism of this selection process is as follows: Before each round of calculation, for the key-value pairs corresponding to the data stored in the N / 2 storage units of the newly added area, the number of rounds in which they participate in attention calculation increases sequentially, from 1 to N / 2 times, and the value of the stored data of each object usually also increases sequentially. The key-value pairs corresponding to the data stored in each storage unit of the key area are mainly selected from the attention weights of the key-value pairs that have participated in the calculation the longest in the newly added area (corresponding to the attention weight of the earliest stored key vector K). These are the top N / 2 key-value pairs with relatively large historical contributions, and their number of attention calculations is usually greater than N / 2 times. Therefore, the newly added area stores N / 2 "potential stocks," while the key area stores N / 2 "high-scoring players." Thus, in each round of selection, it is only necessary to compare the highest-scoring potential stock with the lowest-scoring high-scoring player to determine which potential stock has the highest score and can be included in the high-scoring player sequence in the next round.

[0068] Specifically, if we compare the attention weight of the earliest stored key vector K in the newly added region with the attention weight of the smallest key region, and the former is greater than the latter, it means that the contribution of the newly added region as an "object" is greater than the last one in the top N / 2, and therefore it should be included in the top N / 2. Conversely, if the former is less than the latter, it means that the contribution of the former is lower than the last one in the top N / 2, and it cannot enter the high contribution sequence corresponding to the top N / 2.

[0069] In this embodiment, the aforementioned functions of the filter can be implemented using a circuit module including a minimum value search module and a comparator. The minimum value search module performs a global minimum value search on each memory cell of the key region only after the minimum attention weight in the key region has been covered. The comparator is used to compare the attention weight corresponding to the earliest key vector K after the new region is updated with the minimum attention weight of the key region in each cycle.

[0070] Finally, the loader receives the newly added Q, K, and V in each round, and pre-stores the newly added K and V into the position index of the deletion mark in the first CIM array and the second CIM array respectively (that is, replace the key-value pairs with low attention weights selected earlier to be deleted with the newly added key-value pairs); then it performs a new round of attention calculation.

[0071] In summary: the attention weight calculation unit and the second CIM array constitute the basic circuit for calculating attention values ​​in this embodiment; while the score memory, addition tree, KV cache, filter, and loader constitute the execution objects of the new strategy implemented in this embodiment to filter and dynamically update each key-value pair according to its "historical contribution". Specifically, the addition tree calculates the historical cumulative value of the attention weight of each key-value pair in each round of calculation, for subsequent analysis of the historical contribution of each key-value pair. The score memory stores the calculated historical cumulative values ​​of these key-value pairs. The filter selects the key-value pair with the smallest historical contribution and deletes it based on the data representing the historical contribution of the key-value pair stored in each storage unit of the score memory. The KV cache stores the key-value pairs that should be retained after filtering. The loader loads the key-value pairs retained after filtering from the KV cache, along with the newly added key-value pairs in this round, into the first and second CIM arrays to participate in the next round of attention calculation. In order to avoid the loader needing to perform a global update of the pre-stored key-value pairs in the CIM array in each round of attention calculation, the KV Cache in this embodiment can also record the position index of each key-value pair in the two CIM arrays, so that the loader only needs to update the discarded one when injecting data.

[0072] In particular, combined Figure 1 The Transformer in-memory computation circuit that supports dynamic key-value caching and filtering provided in this embodiment may also include a result memory, which is used to store the current query vector Q and the attention values ​​of all key-value pairs (K and V) obtained after each round of attention computation.

[0073] Existing research has shown that during Transformer decoding, attention distribution typically exhibits highly sparse characteristics, meaning that each query vector contributes significantly only to a small number of historical tokens (Heavy Hitters), while the remaining large number of historical key-value pairs have minimal impact on the final output. However, the Transformer in-memory computation circuitry supporting dynamic key-value caching and filtering provided in this embodiment can perform importance filtering of historical key-value pairs at the hardware level. This eliminates the need for complex data transfer operations as key-value pairs continue to grow, exceeding on-chip available storage space and overflowing into off-chip DRAM. It also avoids the continuous expansion of data computation scale due to the introduction of low-contribution key-value pairs into the computation. Therefore, the solution in this embodiment can significantly reduce computational and storage overhead and improve the computational efficiency of attention while maintaining computational accuracy.

[0074] In particular, when the Transformer in-memory computing circuit that supports dynamic key-value caching and filtering provided in this embodiment is actually applied, it is also necessary to perform initialization settings to overcome the filtering problem faced by historical key-value pairs when the number of historical key-value pairs is less than the maximum cache size supported by the score memory and KV Cache.

[0075] In practical applications, this embodiment can consider the first N cycles of the application phase as the initialization period. Specifically, during the first N / 2 cycles of the initialization period, each newly added attention weight is sequentially stored in the critical region; during the latter N / 2 cycles, each newly added attention weight is sequentially stored in the new region.

[0076] Starting from the N+1th period, the key-value pair dynamic update period begins. The newly added attention weights are updated cyclically in the new addition area in chronological order. Key-value pairs to be discarded are selected based on the relationship between the attention weight of the earliest key-value pair in the new addition area and the minimum attention weight in the key area, and are deleted in the next cycle.

[0077] Of course, in other embodiments, only the newly added area can be filled in the first half of the initialization period, and then the overflowing data from the newly added area can be filled into the key area one by one in the second half. Alternatively, the newly added area can be filled in the first half of the initialization period, and then the data from the newly added area can be moved to the key area in the second half, and then the newly added area can be refilled.

[0078] Example 2 In the technical solution provided in Embodiment 1, during the calculation of attention weights, the CIM circuit still incurs considerable computational overhead when the first and second CIM arrays perform multiplication operations on the query vector Q and key vector K of length M bits, or the attention weight U and value vector V. This computational overhead is reflected in the hardware scale of the relevant circuits and the timing length required for the calculation process, including area overhead, time cost, and power consumption.

[0079] For example, when the vector length M of QKV is 8, the CIM circuit that supports multiplication operations between 8-bit numbers is often large in scale. Therefore, this embodiment provides a more optimized Transformer in-memory computing circuit that supports dynamic key-value caching and filtering based on the scheme in Embodiment 1, which can "accelerate" the calculation task of attention weight U.

[0080] To achieve this acceleration mechanism, the computation module in the first CIM array of this embodiment employs a multiplication module capable of performing multiplication between M / 2-bit inputs and M / 2-bit weights. It consists of M / 2+1 2-to-1 multiplexers, including one M / 2-bit 2-to-1 multiplexer MUX0 and M / 2 single-bit 2-to-1 multiplexers mux; the circuit connection is as follows: The two inputs of MUX0 are connected to the high M / 2 bits and low M / 2 bits of the weight data, respectively. The output of MUX0 outputs a strobed M / 2 bit result, and the values ​​of each bit of this output result are used as inputs to one of the M / 2 muxes. The other input of each mux is connected to the values ​​of each bit of the input data. The output of each mux is used to output the product of the corresponding bit of the weight and the corresponding bit of the input.

[0081] That is, this circuit uses a "half-precision" arithmetic circuit capable of performing multiplication between M / 2 bits, rather than a "full-precision" arithmetic circuit capable of performing multiplication between M / 2 bits. Furthermore, as... Figure 5 As shown, the Transformer in-memory computation circuit supporting dynamic key-value caching and filtering also includes a prediction unit. The prediction unit first obtains the Softmax result corresponding to the dot product of the high M / 2 bits of the calculated query vector Q and the high M / 2 bits of each key vector K, and then determines whether the Softmax result is greater than e. -5 : (1) If so, continue to calculate the attention weight corresponding to the full-precision dot product of the query vector Q and the key vector K, and use it as the required attention weight.

[0082] (2) Otherwise, the Softmax result at that position is determined to be 0 after quantization, the corresponding attention weight is directly reset to 0, and subsequent low-bit calculations are skipped to reduce redundant operations.

[0083] In the circuit with a prediction unit provided in this embodiment, for the query vector and key vector represented by int8, the system divides them into two parts: significant bits and significant bits. At the beginning of the calculation, the system prioritizes performing a dot product calculation on the significant bits. The intermediate result obtained from the significant bit calculation can numerically reflect the relative importance of the corresponding key-value pair well, and therefore is promptly sent to the subsequent prediction and filtering logic to determine whether the key-value pair needs to continue participating in subsequent calculations (by judging whether the Softmax result is greater than e). -5 To determine).

[0084] When the subsequent logic determines that the contribution of a certain intermediate result is low (not greater than e), -5 When the result of the most significant bit indicates that the key-value pair may contribute significantly to the final output (greater than e), the system may choose to skip the supplementary calculation of the least significant bit portion of the path, thereby reducing unnecessary multiplication-addition operations and bit line switching activities; only when the result of the most significant bit indicates that the key-value pair may contribute significantly to the final output (greater than e) will the system skip the supplementary calculation of the least significant bit portion of the path. -5 Only when the value of the least significant bit is reached will the system trigger the supplementary dot product calculation for the least significant bit portion and merge the calculation results of the most significant and least significant bits to restore the full precision.

[0085] Example 3 To better apply the solutions in Embodiments 1 and 2, this embodiment further provides a CIM chip, which is packaged from a Transformer in-memory computing circuit supporting dynamic key-value caching and filtering as described in Embodiments 1 or 2. This CIM chip provided in this embodiment is mainly used in data processing tasks involving self-attention computation, and it can significantly improve the computational efficiency of such data processing tasks.

[0086] Taking a QKV vector length M of 8 and a number of valid key-value pairs N of 204 as an example, the chip-based attention calculation method in this embodiment includes the following process: At each time t, a set of query vectors Q is input. t Key vector K t AND value vector V t Among them, K t Q is stored as a weight in the first CIM array. t Q is completed as input within the first CIM array. t ×K t The calculation results are fed into the Softmax module. The output of the Softmax module, after being processed by the quantization module, is used as the attention weight and input into the second CIM array, where it is compared with the corresponding value vector V pre-stored within the second CIM circuit. t The resulting attention value is written to the result memory after multiplication.

[0087] In this embodiment, the storage arrays in the first and second CIM arrays are configured with a size of 16×204, corresponding one-to-one with the K entries in the KV Cache. The quantized attention weights are accumulated and updated in the addition tree. At each time step, the weights are updated based on the accumulated results of the previous time step and stored in a 1×204 score memory. This storage module is logically divided into two parts: the first 102 positions are designated as the new addition area, used to store the 102 most recently arrived Q values. t ×K t As a result, entries are not compared in size but are updated cyclically in chronological order. The last 102 positions are designated as key regions, used to store entries with higher cumulative attention weights and participate in minimum value comparisons.

[0088] At each time step, the earliest entry among the first 102 newly added regions is selected as a candidate for elimination, and its cumulative score is compared with the lowest score among the last 102 key regions. If the newly added region selected as the elimination candidate has a higher attention weight, its corresponding K and V are retained to replace the key region with the lowest attention weight, and their position indices are updated. Otherwise, the existing key region entries remain unchanged, and there is no need to update the KV Cache.

[0089] To further reduce computational overhead, coarse-grained Q×K operations are initially performed using only the high 4 bits in the first CIM array. T The calculation is performed, and the results of the Softmax prediction are combined to determine whether it is necessary to continue to perform fine calculations on the lower 4 bits, thereby achieving dynamic pruning of the subsequent calculation path.

[0090] To verify the performance and advantages of the Transformer in-memory computing circuit supporting dynamic key-value caching and filtering provided by this invention, technicians conducted simulation tests on the functional blocks of the relevant circuit. The signal timing for the circuit implementation of filtering and submission of the KV Cache is as follows: Figure 6 As shown; the signal timing for the circuit to implement mapping relationship updates and column alignment output with the 16×204 CIM array is as follows. Figure 7 As shown; the timing diagram for the circuit implementing joint KV cache filtering and KV mapping update is as follows. Figure 8 As shown; the circuit uses a prediction unit to implement the signal timing for skipping low-order operations, as shown in the figure. Figure 9 As shown.

[0091] Based on this, the technicians systematically evaluated the accuracy and robustness of the KV Cache management and filtering of the circuit of this invention in an autoregressive inference scenario, and compared and analyzed it with the existing Transformer decoding hardware implementation scheme, finding that: Existing technologies typically use external storage or centralized caching structures to store all historical key-value (KV) caches. During attention calculations, all historical keys need to be accessed and used in the computation, introducing frequent data movement and memory accesses. Furthermore, as the sequence length increases, the number of KV caches expands linearly, leading to a significant increase in storage consumption and computational latency. In addition, these solutions generally lack hardware-level KV cache filtering mechanisms, requiring full-precision dot product and softmax operations to be performed on all historical tokens. Even if many keys contribute little to the current output, they still inevitably participate in the computation, resulting in significant redundancy overhead.

[0092] In contrast, this invention directly introduces a dynamic filtering and updating mechanism for the KV Cache into the in-memory computing architecture, enabling key-value vectors to complete contribution evaluation within the in-memory computing array and making retention or eviction decisions accordingly. Specifically, this invention utilizes intermediate results generated during attention computation as the filtering criterion. Through prediction and gating logic of partial computation results, it directly suppresses computation paths corresponding to low-contribution keys at the hardware level, preventing them from continuing to occupy in-memory cache resources and participate in subsequent inference processes. Compared to schemes that rely on centralized cache management of the KV Cache, this method does not require complete access to all historical key-value pairs, nor does it rely on additional sorting or reordering operations. Dynamic filtering and updating of the KV Cache can be completed through lightweight control logic, effectively reducing storage redundancy and computational complexity.

[0093] A comprehensive comparison shows that, without introducing additional precision loss, this invention significantly reduces invalid storage and redundant calculations caused by low-contribution key values ​​by filtering and directly managing the KV Cache. Compared with existing technologies, it can achieve more efficient autoregressive inference with lower hardware resources and energy consumption, especially in long sequence scenarios, demonstrating better cache utilization efficiency and system scalability.

[0094] The above-described embodiments are merely one implementation of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the inventive concept, and these all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. A Transformer in-memory computation circuit supporting dynamic key-value caching and filtering, comprising: The attention weight calculation unit includes a first CIM array and is used to calculate the attention weights between the input query vector Q and the pre-stored key vectors K. The second CIM array is used to multiply each pre-stored value vector V with the corresponding attention weight to obtain the attention value; The score memory comprises several storage units; half of which is used as the new addition area and the other half as the key area; the key area is used to store several historically accumulated values ​​of attention weights with higher values; the attention weights of the new key-value pairs added in each round are cyclically updated in the new addition area in chronological order. An addition tree is used to cumulatively update each storage unit with the attention weight calculated in each round as the increment; KV Cache is used to store and save historical key-value pairs and their position indexes in the first CIM array and the second CIM array; A filter is used to compare the attention weight of the earliest stored K in the new region with the attention weight of the smallest value in the key region after each round of attention calculation: if the former is greater than the latter, the former is used to cover the latter in the key region, and a deletion mark is added to the position index of the latter in the KV Cache; otherwise, a deletion mark is added to the position index of the former in the KV Cache. The loader receives the newly added Q, K, and V in each round, pre-stores the newly added K and V into the position index corresponding to the deleted marker in the first CIM array and the second CIM array respectively, and then performs a new round of attention calculation.

2. The Transformer in-memory computing circuit supporting dynamic key-value caching and filtering according to claim 1, characterized in that: It also includes a result memory, which stores the current Q and the attention values ​​of all key-value pairs obtained after each round of attention calculation.

3. The Transformer in-memory computing circuit supporting dynamic key-value caching and filtering according to claim 1, characterized in that; The attention weight calculation unit includes: The first CIM array is used to calculate the dot product of the input query vector Q and the pre-stored transposed key vectors K. The Softmax module is used to perform Softmax processing on the output of the first CIM array; The quantization module is used to quantize the output of the Softmax module to obtain the corresponding attention weights for K.

4. The Transformer in-memory computing circuit supporting dynamic key-value caching and filtering according to claim 3, characterized in that: Both the first CIM array and the second CIM array include a storage array and a computing module; Let the number of valid key-value pairs in the attention computation task be N, and the vector length be M. Then the storage array is composed of an N-row, M-column SRAM cell array. Each SRAM cell in each row of the storage array is used to pre-store the value of each key vector K or value vector V bit by bit. The calculation module includes a multi-bit multiplication operation module composed of multiple two-to-one selectors and a shift accumulator; the multi-bit multiplication operation module takes the values ​​of each bit of the query vector Q as input and the values ​​of each bit of the pre-stored key vector K as weights, and then calculates the product of each bit of the input and each bit of the weights. Then, a shift accumulator is used to obtain the result of the multiplication operation between the multi-bit input and the multi-bit weight.

5. The Transformer in-memory computing circuit supporting dynamic key-value caching and filtering according to claim 4, characterized in that: The computation module in the first CIM array employs a multiplication module capable of performing multiplication between M / 2-bit inputs and M / 2-bit weights; it consists of M / 2+1 2-to-1 multiplexers, including one M / 2-bit 2-to-1 multiplexer MUX0 and M / 2 single-bit 2-to-1 multiplexers mux; the circuit connection is as follows: The two inputs of MUX0 are connected to the high M / 2 bits and low M / 2 bits of the weight data, respectively. The output of MUX0 outputs an M / 2 bit of the selected result, and the value of each bit of the output result is used as the input of one of the M / 2 muxes. The other input of each mux is connected to the value of each bit of the input data. The output of each mux is used to output the product of the corresponding bit of the weight and the corresponding bit of the input.

6. The Transformer in-memory computing circuit supporting dynamic key-value caching and filtering according to claim 5, characterized in that: It also includes a prediction unit; The prediction unit is used to first obtain the Softmax result corresponding to the dot product of the high M / 2 bits of the calculated query vector Q and the high M / 2 bits of each key vector K, and then determine whether the Softmax result is greater than e. -5 : (1) If yes, continue to calculate the attention weight corresponding to the full-precision dot product of the query vector Q and the key vector K, and use it as the required attention weight; (2) Otherwise, the Softmax result at that position is determined to be 0 after quantization, the corresponding attention weight is directly reset to 0, and subsequent low-bit calculations are skipped to reduce redundant operations.

7. The Transformer in-memory computing circuit supporting dynamic key-value caching and filtering according to claim 5, characterized in that: The first N cycles of the application phase are the initialization period; in the first N / 2 cycles, each newly added attention weight is stored in the key area in sequence; in the last N / 2 cycles, each newly added attention weight is stored in the new area in sequence. Starting from the N+1th period, the key-value pair dynamic update period begins. The newly added attention weights are updated cyclically in the new addition area in chronological order. Key-value pairs to be discarded are selected based on the relationship between the attention weight of the earliest key-value pair in the new addition area and the minimum attention weight in the key area, and are deleted in the next cycle.

8. The Transformer in-memory computing circuit supporting dynamic key-value caching and filtering according to claim 5, characterized in that: The filter includes a minimum value search module and a comparator; the minimum value search module performs a global minimum value search on each storage unit of the key region only after the minimum attention weight in the key region has been covered. The comparator is used to compare the attention weight corresponding to the earliest key vector K after the new region is updated with the minimum attention weight of the key region in each cycle.

9. A CIM chip, characterized in that: It is encapsulated by a Transformer in-memory computing circuit that supports dynamic key-value caching and filtering as described in any one of claims 1-8.

10. An application of the CIM chip as described in claim 9 in a data processing task involving self-attention computation.

Images