Calculation circuit and sparse attention calculation method

Abstract

The embodiment of the invention provides a calculation circuit and a sparse attention calculation method. A hardware circuit is improved in combination with the change of a calculation mode. In the aspect of a calculation mode, a sparse process of searching in a K vector through a Q vector in attention calculation is adjusted to firstly calculate a product of a high-bit group of the Q vector and the K vector, and the K vector with higher possibility is selected for sparse attention calculation by comparing the product of the high-bit group. In hardware design, the product of high bit groups of a Q vector and a K vector is calculated through ordered arrangement and combination of a calculation unit and a shift register of a shift storage unit, sparse selection of the K vector is completed, and therefore introduction of a module corresponding to a prediction model is avoided, and sparse attention calculation is performed in a sparse attention calculation stage. And multiplexing the calculation unit and the calculation result in the sparse selection process. According to the circuit design, the area-effect ratio and the energy efficiency ratio are greatly improved.

Description

TECHNICAL FIELD

[0001] One or more embodiments of the present specification relate to the field of computer technology, and in particular, to a computing circuit and a method for sparse attention computation. BACKGROUND

[0002] Attention mechanism in deep learning is a method that simulates human visual and cognitive system, which allows neural networks to focus on relevant parts when processing input data. By introducing attention mechanism, neural networks can automatically learn and selectively focus on important information in the input, improving the performance and generalization ability of the model. Attention mechanism usually calculates the output of attention aggregation as a weighted sum of values. The attention weight distribution of values is achieved by attention aggregation of Query and Key. That is, given a Query, calculate the relevance of Query and Key, and according to the relevance of Query and Key, the attention weight distribution of Value is generated to generate the final output result. In complex models based on attention network (such as large language models containing Transformer architecture), there are often a large number of attention computation tasks, which need to be optimized, compressed or pruned.

[0003] Sparse attention is an optimized attention mechanism that can map a query vector and a set of key-value pairs to an output vector, but unlike traditional single-head attention and multi-head attention, it does not calculate the similarity between the query vector and all key vectors, but only calculates the similarity between the query vector and part of the key vector, thereby reducing the amount of calculation and memory consumption. As for which key vectors do not need to be calculated, the conventional technology usually predicts and determines through a mask prediction task. Based on this, the conventional hardware solution of sparse attention usually adopts a hybrid architecture and designs an independent hardware function module for the mask prediction task. However, this may introduce redundant logic and storage units, resulting in a decrease in the area efficiency and energy efficiency of the hardware circuit. SUMMARY

[0004] One or more embodiments of the present specification describe a computing circuit and a method for sparse attention computation to solve one or more problems mentioned in the background.

[0005] According to a first aspect, a computing circuit is provided, comprising a computing unit, a shift storage unit and a KV storage unit, wherein: the computing unit comprises at least one processor configured to compute a matrix product, and compute a first product and a dot product between a query vector and a key vector in a sparse attention mechanism, a single first product being a dot product of a high-bit group vector of a single query vector and a high-bit group vector of a single key vector, the high-bit group vector being a vector formed by taking a high-bit group of each element of the corresponding query vector or key vector, and a single high-bit group being determined by a highest predetermined bit of the corresponding element; the shift storage unit comprises a plurality of shift registers configured to store the first products and corresponding key vector identifiers in a descending order by each shift register, and provide the computing unit with the first products within a predetermined number from large to small for computing the dot product of the query vector and the key vector, and provide the KV storage unit with the corresponding key vector identifiers as indexes for obtaining a vector group formed by a key vector and a value vector; and the KV storage unit is configured to store each vector group and provide the computing unit with vectors in each vector group.

[0006] In an embodiment, the circuit further comprises a configuration register configured to configure a working mode of the circuit, the working mode comprising a first mode and a second mode; wherein: in the first mode, the circuit performs sparse attention computation using the computing unit, the shift storage unit and the KV storage unit; and in the second mode, the shift storage unit is bypassed, and the circuit performs matrix multiplication computation by the computing unit from the KV storage unit to obtain each vector group.

[0007] In an embodiment, the number of effective shift registers is the predetermined number, and the first products arranged in front of the predetermined number in a descending order are written into the effective shift registers, and the first products arranged outside the predetermined number are discarded.

[0008] According to a second aspect, a method for sparse attention computation is further provided, the method comprising: obtaining a first query vector and corresponding key vectors in an attention mechanism; performing high-bit group multiplication computation on the first query vector and each key vector respectively to obtain each first product, a single first product being a dot product of a high-bit group vector of the first query vector and a high-bit group vector of a single key vector, the high-bit group vector being a vector formed by taking a high-bit group of each element of the corresponding query vector or key vector, and a single high-bit group being determined by a highest predetermined bit of the corresponding element; obtaining corresponding candidate vector groups according to key vector identifiers corresponding to a predetermined number of first products arranged in front, a single candidate vector group comprising a single key vector and a single value vector; and performing sparse attention computation based on each candidate vector group and each first product.

[0009] In an embodiment, the storing of the obtained respective first products in descending order comprises: for a current first product, comparing it with each of the stored respective first products one by one; and determining the storage manner of the current first product based on the comparison result.

[0010] In a further embodiment, only a predetermined number of first products and corresponding key vector identifiers are stored for the first query vector; and the determining the storage manner of the current first product based on the comparison result comprises: in case that a stored first product smaller than the current first product is detected, inserting the current first product into the stored first products, moving other first products smaller than and closest to the current first product and subsequent first products backward, and discarding first products whose size order exceeds the predetermined number; and in case that the number of stored first products is the predetermined number and no stored first product smaller than the current first product is detected, discarding the current first product.

[0011] In an embodiment, the sparse attention calculation based on each candidate vector group and each first product comprises: calculating second products and third products of the first query vector and candidate key vectors in each candidate vector group, wherein a single second product comprises a sum of a product of a high bit group vector of the first query vector and a low bit group vector of a single candidate key vector and a product of a low bit group vector of the first query vector and a high bit group vector of the single candidate key vector, and a single third product is a product of the low bit group vector of the first query vector and the low bit group vector of the single candidate key vector, and a single low bit group vector is a vector formed by taking low bits of respective elements of the corresponding query vector or candidate key vector; determining respective attention weights of each candidate key vector, wherein a single attention weight corresponding to a single candidate key vector is positively correlated with a value obtained by adding the respective single second product and the single first product left shifted by s bits, and then adding the single third product left shifted by s bits, wherein s is the number of bits of the other bits; and performing weighted summation on value vectors in each candidate vector group by using the respective attention weights to obtain an attention vector.

[0012] In an embodiment, in case that the computing unit comprises a plurality of processors, the plurality of processors can be used for serial or parallel processing in the vector multiplication process.

[0013] In an embodiment, the method is implemented by the circuit of the first aspect.

[0014] According to a third aspect, a computer readable storage medium is provided, having stored thereon a computer program which, when executed in a computer, causes the computer to perform the method of the second aspect.

[0015] According to a fourth aspect, a computing device is provided, comprising at least one of: the computing circuit provided by the first aspect; a processor and a memory, the memory storing executable code, when the processor executes the executable code, the method provided by the second aspect is implemented.

[0016] By the computing circuit and method provided by the embodiments of the present specification, based on the sparse attention computing process, the hardware circuit is improved by combining the change of the computing mode. From the computing mode, in the sparse process of retrieving the Q vector in the K vector in the attention computing, the K vector is sparsified by calculating the product of the high bit group of the Q vector and the K vector and comparing the size, thereby replacing the design of sparsifying the K vector by containing the prediction model. In this way, in the hardware design, the product of the high bit group of the Q vector and the K vector can be calculated by the ordered arrangement of the calculation unit and the shift register of the shift storage unit, and stored in the shift register in size order, so as to effectively filter the K vector, obtain the corresponding KV vector group, and calculate the sparse attention by multiplexing the product of the high bit group stored in the shift register. The special circuit design for sparse attention computing under this technical concept avoids introducing additional filtering modules, and the area efficiency and energy efficiency are greatly improved. In addition, the sparse attention computing mode under this technical concept can also be realized by using existing computing resources, but in the case of using the special hardware computing circuit design provided by the present specification, the computing is more efficient. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0018] Figure 1 A specific implementation architecture diagram of attention computing is shown;

[0019] Figure 2 A calculation unit diagram of a group of Q, K, and V vectors in attention computing is shown;

[0020] Figure 3 A hardware circuit design architecture diagram for sparse attention computing in conventional technology is shown;

[0021] Figure 4 A correlation comparison diagram of Q vector and K vector in actual application is shown;

[0022] Figure 5 A calculation principle diagram under the technical concept of the present specification is shown;

[0023] Figure 6 A hardware circuit design architecture diagram under the technical concept of the present specification is shown.

[0024] Figure 7 A high-bit group product storage and a shift storage unit action diagram according to one specific example of the present specification are shown.

[0025] Figure 8 A flow diagram for sparse attention calculation according to an embodiment of the present specification is shown.

[0026] Figure 9 A structural block diagram of an apparatus for controlling sparse attention calculation according to one embodiment of the present specification is shown. DETAILED DESCRIPTION

[0027] The schemes provided in the present specification will be described below with reference to the accompanying drawings.

[0028] Figure 1 A specific implementation architecture diagram of attention calculation is shown. It can be understood that the Attention mechanism is essentially an addressing process. Given a query Q (denoted as a vector Q) related to a task, the attention distribution is calculated with a key K (denoted as a vector K), and the value V (denoted as a vector V) is acted on, so as to calculate the value under the attention mechanism (Attention Value) for processing. This process is actually an embodiment of relieving the complexity of the neural network through the attention distribution. Instead of inputting all inputs (such as N, N is a natural number greater than 1) to the neural network for calculation, more attention is paid to the information related to the task that is input into the neural network.

[0029] Under the attention calculation mechanism, the core is to introduce the attention calculation of vectors Q, K, and V. Vectors K and V are usually obtained by transforming the hidden vectors (denoted as X) output by the encoder through the corresponding weight matrices W K 、W V , and the vector Q is obtained by transforming the current state (hidden state of the output sequence) of the decoder through the corresponding weight matrix W Q . It can be understood that K and V are the global language feature library of the encoder output vector, and Q is the keyword processed by the decoder from the output sequence to query the feature library. As an example, for an input or hidden vector X, linear transformation is performed to obtain Q=XW Q 、K=XW K 、V=XW V , wherein W Q 、W K 、WV are three trainable parameter matrices.

[0030] Referring to Figure 1 As shown, K vectors (such as Key1, Key2, Key3, etc.) and V vectors (such as Value1, Value2, Value3, etc.) are one-to-one correspondence, and Q vector (Query) is used as a query vector, which is fused with each Key (usually by dot product to determine similarity), and the fusion result is applied to Value, such as to obtain the action results z1, z2, z3, etc., and then the action results are fused to obtain the attention value Attention Value.

[0031] As a specific example, the calculation of the attention matrix is represented as: where d k is the dimension of K, usually as a normalization scale.

[0032] In order to facilitate understanding, a specific example is used to describe the meaning of the above calculation. In the translation scene, assuming that Q is the embedding vector of "I", and the KV vector group includes the generated vectors K and V respectively, and the KV vector group is, for example, the encoding vector group corresponding to the words "me", "meet", "some", etc. In order to query and determine the translation result of "I", the embedding vector of "I" is calculated with the K vector in the encoding vector group corresponding to the words "me", "meet", "some", etc. (such as dot product), and then the similarity result is used as the weight of the V vector in the corresponding encoding vector group, and the weighted result is used to determine the corresponding attention value, and is mapped to the probability distribution of "I" translated into "me", "meet", "some", etc.

[0033] In order to clarify the attention calculation process, referring to Figure 2 As shown, in a set of K, V vector corresponding to an attention calculation unit, first calculate the vector product (i.e. dot product) QK T of Q and K by matrix multiplication, and then use d k to normalize and Softmax activation, and the obtained result is calculated with the vector product of V by matrix multiplication, as the calculation result Z of the attention calculation unit. The calculation result Z can be used as the final calculation result or hidden output (as the input of the next attention calculation unit).

[0034] In complex attention architectures, such as multi-head attention, cross-attention, etc., the computational complexity of calculating Q querying K is large. For example, in a large language model or translation model based on a transformer architecture, there are tens of thousands of candidate words, and the relevance of the current word (corresponding to Q) and the K vectors (such as tens of thousands) corresponding to each candidate word needs to be calculated. In order to save computational complexity, a mask processing can be used to filter out K with low relevance to Q. For example, Figure 2 The mask step is shown in the dashed box. Among them, Figure 2 Only as an example, in practice, the mask step can be performed before normalization or after normalization. Performing it before normalization can reduce the computational complexity in the normalization process.

[0035] In the case of a large number of Q and a large number of K, attention calculation of each Q with each K is required. In order to filter out K with low relevance to Q, various mask methods can be used. In one implementation, for a single Q, the dot product similarity of Q and each K can be calculated first, and K with a dot product similarity lower than a predetermined threshold can be filtered out (such as Figure 2 Mask processing in the above). In another implementation, for a single Q, K with low relevance to Q can be filtered out before calculating the dot product similarity of Q and each K. This step can be implemented by a prediction model, thereby reducing the computational complexity of matrix calculation of Q and K, which is sparse attention calculation. In other implementations, other ways of mask processing can also be used, which will not be described here. This specification discusses sparse attention calculation.

[0036] In order to accelerate attention calculation, a special hardware structure can be designed. Figure 3 A specific implementation architecture of a hardware structure for sparse attention calculation in conventional technology is shown. Referring to Figure 3 As shown in the figure, under the concept of sparse attention calculation, for a single Q, k Ks are selected from each K, denoted as top k, for example, by model prediction of the top k Ks with the largest relevance to Q, then the dot product of Q and the k Ks is calculated, and then the attention calculation is performed based on the dot product, that is, the fusion result of the corresponding V is calculated to obtain the attention value. In one example, in order to avoid errors caused by inaccurate model selection, a relatively loose selection can be performed by the model, that is, n Ks are actually used, k Ks are selected to calculate the dot product with Q, and n is a natural number less than k. Before calculating the fusion result of the dot product of Q and the k Ks with V, the dot product of the top n Q and K can also be selected based on the dot product, and the fusion result of the corresponding n V is calculated as the attention calculation result.

[0037] Corresponding to the hardware implementation, the selection unit 301 can select the k most relevant Ks from each K, wherein the selection unit 301 can include a storage module of a dedicated selection logic (such as a prediction model), such as a Specific Buffer, and a corresponding dedicated computing core, such as a Specific PE. The storage module, for example, contains a dedicated selection topk selection logic, such as a correlation prediction model, wherein the selection logic can use each computing unit (PE, Processing Element) in the dedicated computing core to complete the correlation calculation. In this way, the k Ks with high correlation to a single Q can be predicted. Then, the corresponding k KV vector groups can be obtained (Index) from the KV storage unit (K / V Buffer) 302, and the attention result of the action of Q and the k KV vector groups can be calculated via the attention unit 303.

[0038] Figure 3 The hardware solution of the sparse attention shown adopts a hybrid architecture, and a separate hardware function module is designed for the mask prediction task. The hardware function module of the mask prediction task introduces a dedicated logic unit and a storage unit. The data flow is selection unit 301→KV storage unit 302→attention unit 303. In the hardware circuit, the area efficiency ratio and the energy efficiency ratio are important indicators for measuring the practical performance. Among them, the area efficiency ratio refers to the ratio between the chip performance (Performance) and the area (Area), and the energy efficiency ratio refers to the ratio between the chip performance (Performance) and the power (Power). Optimization of the area efficiency ratio and the energy efficiency ratio is conducive to reducing the volume, improving the energy efficiency, and thus improving the overall efficiency.

[0039] Therefore, based on the split and reorganization calculation principle of Q and K, the present specification proposes a new calculation method and circuit design, which effectively reduces the circuit area and improves the overall efficiency.

[0040] First, the calculation principle under the technical concept of the present specification is described.

[0041] Those skilled in the art can understand that when comparing the size of two numerical values, the comparison can be from high bits to low bits, and the numerical value of the larger high bit is compared first. For example, 1101 1011 and 1011 1111, the first numerical value has a larger numerical value from the second bit from the highest bit, and therefore the numerical value is larger. In practice, in order to reduce the number of comparisons, multiple bits can be compared together, such as 4 bits together, 1101 is larger than 1011, and it is concluded that the first numerical value is larger.

[0042] In addition, in binary calculation, the multiplication of two numbers can be performed by splitting and shifting. Among them, splitting can split the numerical value into a high bit group and a low bit group. As the name implies, the high bit group can be at least one bit arranged in the highest predetermined bit (such as the highest 4 bits), and the low bit group can be at least one bit arranged in the low bit. For example, the numerical value 1101 0110, assuming that the number of bits of the high bit group and the number of bits of the low bit group are both 4, it can be split into a high bit group 1101 and a low bit group 0110.

[0043] Taking the product of two 8-bit numerical values as an example, two numerical values are denoted as P0 and P1, and according to 4 bits per group, a single numerical value can be split into a 4-bit high bit group and a 4-bit low bit group, such as P0 = (M0 « 4) + L0 and P1 = (M1 « 4) + L1. Among them, “« 4” represents left shifting 4 bits, M0 and M1 are the high bit groups of P0 and P1 respectively, and L0 and L1 are the low bit groups of P0 and P1 respectively. In this way, the dot product of P0 and P1 can be denoted as: P0 × P1 = {[(M0M1 « 4) + M0L1 + L0M1] « 4} + L0L1, wherein M0M1 corresponds to the high bit group of the dot product, the numerical value needs to be left shifted by 8 bits, (M0L1 + L0M1) corresponds to the middle bit group of the dot product, the numerical value needs to be left shifted by 4 bits, and since the number of bits of M0L1 + L0M1 may exceed 4 bits, the left carry affects the value of the bit (bit) where M0M1 is located. Therefore, after adding M0M1 left shifted by 4 bits and M0L1 + L0M1, and then left shifting the whole by 4 bits, the number of bits of the left shift operation can also be kept consistent.

[0044] The above takes the product of two 8-bit numbers as an example for description, in practice, it can be the product of more bit numbers, the number of bit groups split can also increase, and the number of bits of the high bit group and the low bit group can be the same or different, which will not be described here.

[0045] Considering the attention mechanism in the transformer architecture, the correlation between Q and K usually has a large difference, in other words, the degree of correlation (such as dot product similarity) between K related to Q and Q is greatly different from the degree of correlation between other K and Q. For example, Figure 4As shown, regarding the sentence "Transformers is my favorite science fiction action film.", after word-by-word disassembling, 8 words are obtained, and the correlation between each word is shown in Table 401 under the condition of attention calculation, where the darker the color, the greater the correlation, and the lighter the color, the smaller the correlation. Taking the query of "my" as an example, the correlation probability of each word is shown in Table 402, and it can be seen that the correlation of "my" with itself and the word "favorite" is the largest, far higher than other words (i.e. other Key). Through experiments, in the field of sparse attention calculation, the corresponding high similarity vector can be effectively screened out by calculating the high bit group product sorting screening first.

[0046] In this way, combined with the law of binary multiplication, the correlation can be sorted by the product of the high bit group of the Q vector and the K vector (such as M0M1 in the previous example), so that according to the sorting result, the predetermined number of K vectors with the largest correlation are taken for a Q vector, and the sparse attention is calculated, without calculating the similarity of the Q vector and each K vector. According to the law of binary multiplication, after high bit calculation, the product can also be used for the final calculation of the dot product correlation, that is, the calculation of other bits combined with the non-highest bit, and the shift according to the rules described above, and the fusion with the high bit product to obtain the final dot product correlation calculation result (such as product QK T ).

[0047] That is, the order of multiplication calculation between the Q vector and each K vector is disturbed, and the high bit group product (which can also be recorded as the dot product of the high bit group vector) is calculated first, sorted, and the K vector corresponding to the maximum k product is taken to calculate the similarity, and the corresponding V vector is queried to perform attention calculation.

[0048] The calculation method is disturbed as shown in Figure 5 . Taking the multiplication of two numbers as an example first. The product of the two numbers is divided into a high bit group and a low bit group. According to the principle described above, the high bit group and high bit group product, high bit group and low bit group product, and low bit group and low bit group product of the two numbers are calculated, and the multiplication of the high bit group and high bit group product, high bit group and low bit group product, and low bit group and low bit group product is added to restore the multiplication of the two numbers.

[0049] The numerical product is replaced by vector product, and the Figure 5In the embodiment, it is assumed that the query vector Q is split into a high-bit group vector M and a low-bit group vector L, the high-bit group vector is a vector in which each element takes a predetermined high bit, and the low-bit group vector is a vector in which each element takes other bits except the high bit. Similarly, the key vectors K0, K1, and the like are split into high-bit group vectors and low-bit group vectors, which are denoted as "K0: M0, L0", "K1: M1, L1", and the like.

[0050] Those skilled in the art can understand that, Figure 5 In the point product calculation example of Q and K shown in the embodiment, actually, the point product calculation of the corresponding bit group vectors is performed (see the point product formula a·b = ∑a i ·b i for details), and the shift and summation operations of each point product also need to be combined. Taking the product of Q and K0 as an example, denoted as MM0, which is the point product of the high-bit group vectors of Q and K0. For the convenience of description, MM0 can be denoted as the first product. Similarly, the products of the high-bit group vector and the low-bit group vector of Q and K0, ML0 and LM0, can be the point product of the high-bit group vector of each element of one of Q and K0 and the low-bit group vector of the other. In this specification, ML0 + LM0 can be denoted as the second product, and the point product of the low-bit group vectors of Q and K0, LL0, can be denoted as the third product. Finally, the point product of Q and K0 is {[ML0 + LM0 + (MM0 << 4)] << 4} + LL0. In this way, the point products of Q and the key vectors K0, K1, and the like can be calculated respectively.

[0051] With reference to the embodiment shown in Figure 5 Under the technical concept of this specification, in order to sparsify the K vectors, the point product calculation between Q and each K is based on the above split and mixed processing. First, the point product of Q and the high-bit group vector of each K is calculated (such as the MMs process in Figure 5 In other words, for Q and a single K (denoted as Ks, s is the subscript of any K), the first product MMs (s = 0, 1,...) is calculated. As shown in Figure 5 In the calculation process, the storage unit is sorted and stored with the point product of Q and the high-bit group vector of each K. For example, one storage result is MM5 (corresponding to the product of Q and K5), MM3 (corresponding to the product of Q and K3), and the like.

[0052] Using the sorting result, the first k Ks with the largest corresponding point product correlation can be obtained. In an optional implementation, the number of effective storage units can also be set to k, and at most k high-bit products corresponding to the Ks are stored, and the others are discarded, so that the k Ks are directly screened out. For the screened k key vectors K, the point product calculation of the high-bit group vector and the low-bit group vector, the low-bit group vector and the low-bit group vector, that is, the calculation of the second product and the third product, can be further performed, and the accurate value of the correlation of Q and the selected k Ks can be obtained in combination with the corresponding first product (such asFigure 5 The QKk process is shown.

[0053] Based on the above calculation principle, the corresponding hardware circuit design and sparse attention calculation method based on the hardware circuit design are provided under the technical concept of the present specification. The following will be described in detail in combination with the drawings.

[0054] Figure 6 The hardware circuit design architecture shown can be used for sparse attention calculation under the above principle. As shown in Figure 6 The hardware circuit under the technical concept of the present specification can include a calculation unit 601 (Unified PEs), a shift storage unit 602 (MsbSel), and a KV storage unit 603 (K&V Buffer). The data flow thereof can include: the calculation unit 601 flows to the shift storage unit 602, the KV storage unit 603 flows to the calculation unit 601, the shift storage unit 602 flows to the calculation unit 601, and the shift storage unit 602 flows to the KV storage unit 603.

[0055] The calculation unit 601 includes at least one processor (such as Figure 6 MAC in ), and in the case of multiple processors, the calculation can be completed in parallel or in series. The calculation unit 601 can be used for matrix multiplication calculation, and vector multiplication (i.e., dot product calculation) is a kind of matrix multiplication, such as Figure 5 QK0, QK1, MMs, QKk, etc. in . The shift storage unit 602 can store the calculation results of the calculation unit 601 and the K vector identifiers in size order, such as Figure 5 k calculation results MMs, MM3, etc. in , and vector identifiers 5, 3, etc. Shift Regs in the shift storage unit 602 represent shift registers, which are used to store the dot product (first product) calculation results of high-bit group vectors, and can perform transposition operations on the stored data in size order, etc.

[0056] In combination with the technical concept of the present specification, referring to Figure 6 For a single Q vector, first obtain the high-bit group vectors (M, M) of the Q vector and each K vector, calculate each first product MM corresponding to the high-bit group vectors of the Q vector and each K vector through the calculation unit 601 (without performing shift operation, such as Figure 6The first products MM are stored in the shift register of the shift storage unit 602. For each first product MM calculated, the first product is compared with the first products already obtained (e.g., the first products stored in the shift storage unit 602) in the process of being stored in the shift register, and is inserted into the register in descending order. In an optional embodiment, the number of effective shift registers can be set to k, and the calculation results exceeding the number of shift registers can be discarded in the order of descending, so as to select the k results retained in the shift register and record the key vector identifiers of the corresponding k K vectors. The k K vectors can be denoted as candidate key vectors. Alternatively, all shift registers can be effective shift registers, or the number of shift registers is determined according to the number of candidate K vectors required by sparse attention calculation.

[0057] Reference Figure 7 As shown, assuming that the number of effective shift registers is set to 4, in the nth cycle of the first stage (Stage 1, Cycle n), the order is MM2, MM5, MM0, MM7. After the calculation unit 601 calculates the first product MMn of the Q vector and the nth K vector, the calculation result (Val) obtained by inserting the calculation result in descending order, the order in the register becomes MM2, MM5, MM0, MM7, and MM7 is discarded (see the (n+1)th cycle of the first stage: Stage 1, Cycle n+1). Wherein, the subscripts 2, 5, n, etc. can represent the key vector identifier of the K vector. n

[0058] The KV storage unit 603 can be used to store K vector and V vector data. The K vector and the V vector are stored in the form of a KV vector group. After the first product of the Q vector and each K vector is calculated, the key vector identifiers of the first k calculation results arranged in descending order in the shift register can be obtained, so as to obtain the corresponding KV vector data from the KV storage unit 603 by using the key vector identifier as an index, which is denoted as a candidate vector group.

[0059] Based on the candidate vector group provided by the KV storage unit 603 and the first product provided by the shift storage unit 602, the dot product of the Q vector and the K vector is further calculated by using the calculation unit 601, and the dot product is used as a similarity result to act on the V vector to obtain the final attention value. In the calculation of the dot product of the Q vector and the K vector, the first product in the shift register can be returned to the calculation unit 601 to participate in the attention calculation, and the first product is reused data (Reused Data (Top-k)), and the calculation unit 601 is a reused calculation unit. This corresponds to the second stage (Stage 2) in the sparse attention calculation in the prior art, as shown in FIG. 1B. Figure 7 Figure 7 ​​The Query Key Multiplication attention calculation process is shown. 2, 5, n, 0 can be used as an index to obtain the corresponding vector from the KV cache.

[0060] Compared with Figure 6 and Figure 3 , the hardware circuit design architecture for sparse attention calculation under the technical concept of the present specification discards the shift storage unit containing the special selection model and selection logic, adjusts the bit splitting and calculation order of the Q and K vectors, and cooperates the calculation by the calculation unit 601 and the ordered storage of the shift storage unit 602 to complete the K vector screening in sparse attention calculation and perform sparse selection of the KV vector group. In this way, the sparse attention calculation can be synchronized with selection and calculation, avoiding the additional area and energy consumption overhead caused by introducing too many hardware modules.

[0061] In addition, referring to Figure 6 and Figure 3 , the calculation unit 601 under the technical concept of the present specification can be reused in the K vector selection and sparse attention calculation process, greatly reducing the hardware circuit area. Compared with the conventional hardware design containing selection model and selection logic, the area can be reduced from 1.26 to 1.74 square millimeters (mm 2 ) to 0.21 square millimeters, the power consumption can be reduced from 102.65 to 969.36 milliwatts (mW) to 62.34 milliwatts, and the area efficiency and energy efficiency are greatly improved.

[0062] In addition, referring to Figure 6 , from the data flow direction, data can flow bidirectionally between the calculation unit 601 and the shift storage unit 602, and there is data flow from the KV storage unit 603 to the calculation unit 601. Therefore, in the case of bypassing the shift storage unit 602, normal matrix calculation can be performed through the data flow from the KV storage unit 603 to the calculation unit 601. The bypassing of the shift storage unit 602 can be achieved by a bypass switch or by a configuration register. The configuration register is used to configure Figure 6 The working mode of the entire circuit is shown. It can be configured as a first mode and a second mode. In the first mode, the circuit uses the calculation unit 601, the shift storage unit 602, and the KV storage unit 603 to perform sparse attention calculation. In the second mode, the shift storage unit 602 is bypassed, and the circuit performs matrix multiplication calculation by the calculation unit 601 from the KV storage unit 603 to obtain each vector group.

[0063] Therefore, the application range and function expansion of the circuit module can be increased, and the area ratio of the shift storage unit 602 to the calculation unit 601 is about 0.014:0.196, and the power consumption ratio is about 3.71:58.63, so that the area efficiency and energy efficiency after the bypass are almost not affected.

[0064] In the process of performing sparse attention calculation by the circuit shown in Figure 6 The execution subject can be any computer, device, or server with certain calculation capability, which can include the circuit shown in Figure 6 The process of performing sparse attention calculation by the circuit shown in Figure 6 For example, the process of performing sparse attention calculation by the circuit shown in includes: obtaining a first query vector and corresponding respective key vectors in an attention mechanism; performing dot product calculation of high-bit group vectors on the first query vector and the respective key vectors by a calculation unit, and storing each first product obtained in a shift storage unit in descending order; obtaining corresponding candidate vector groups from a KV storage unit according to key vector identifiers corresponding to a predetermined number of first products arranged in the front of the shift storage unit, and providing the candidate vector groups to the calculation unit; and performing sparse attention calculation by the calculation unit based on the respective candidate vector groups obtained from the KV storage unit and the respective first products provided by the shift storage unit.

[0065] In addition, it is worth noting that, in combination with the hardware circuit implementation shown in Figure 6 The hardware circuit can be directly implemented by the hardware circuit shown in Figure 6 For example, in the case where a business system to be completed calculation obtains a first query vector and corresponding respective key vectors in an attention mechanism for sparse attention calculation, the business system can interact with the configuration register in the hardware circuit shown in Figure 6 The relevant data is provided to the circuit shown in Specifically, after the circuit obtains Q, K, and V data, the K and V data are stored in the KV storage unit 603 as KV vector groups, the calculation unit 601 takes high-bit groups in the Q and K vectors to perform first product calculation, and the calculation result is stored in the shift storage unit 602 in descending order. According to the storage result of the shift storage unit 602, the vector identifier of the candidate key vector is obtained, and the corresponding candidate vector group is retrieved from the KV storage unit 603. Then, the shift storage unit 602 feeds back the first product corresponding to the candidate key vector to the calculation unit 601, the KV storage unit 603 feeds back the candidate vector group to the calculation unit 601, and the calculation unit 601 further calculates the dot product of the first query vector and each candidate key vector and acts on the V vector to obtain the attention value calculation result.

[0066] That is, in the design process, the data flow and result feedback mode are designed for the hardware circuit, and when used, only the corresponding Q and K, V vector data are input, and the corresponding sparse attention calculation can be automatically completed.

[0067] In the case of implementation through the control module, the control module can be a hardware module or a computer instance, which is used to call the hardware circuit as shown in Figure 6 to complete the corresponding sparse attention calculation. This is not limited.

[0068] Based on the calculation principle in this paper, the hardware circuit as a special circuit can accelerate the completion of calculation. In fact, without setting the hardware circuit, sparse attention calculation can also be completed by using the existing resources of the computing device through this idea.

[0069] Figure 8 The flowchart for sparse attention calculation based on the above technical concept is shown. The execution subject of the flowchart is any computer, device, or server with certain computing power. In the case of using Figure 6 the circuit shown in the figure, the sparse attention calculation can be accelerated.

[0070] As shown in Figure 8 , the flow for sparse attention calculation can include the following steps: step 801, obtaining a first query vector and corresponding each key vector in the attention mechanism; step 802, performing multiplication calculation on the high bit group of the first query vector and each key vector respectively, and storing each first product obtained in descending order, a single first product is the dot product of the high bit group vector of the first query vector and the high bit group vector of a single key vector, the high bit group vector is a vector composed of the high bit group of each element of the corresponding query vector or key vector, and a single high bit group is determined by the highest predetermined bit of the corresponding element; step 803, obtaining the corresponding candidate vector group according to the candidate key vector identifier corresponding to the top pre-determined number of first products; step 804, based on each candidate vector group and each first product provided by the shift storage unit, performing sparse attention calculation.

[0071] First, in step 801, a first query vector and corresponding each key vector in the attention mechanism are obtained.

[0072] It can be understood that under the attention mechanism, multiple attention calculations (such as one attention calculation unit as shown in Figure 2 ) in parallel or succession can be involved, and one or more query vectors and multiple key vectors, value vectors can be involved in one attention calculation process. Figure 8The illustrated flow is described by taking an example of a process in which one query vector queries multiple key vectors, and attention summation is performed for multiple value vectors, as shown in Figure 1 The query vector here can be any one query vector, such as the first query vector. The key vectors queried by the first query vector are denoted as their corresponding key vectors.

[0073] Next, via step 802, multiplication calculation of the high-bit group is performed on the first query vector and each key vector, respectively, and each first product obtained is stored in descending order.

[0074] According to the technical concept of the present specification, for the first query vector and a single key vector, multiplication calculation of the high-bit group can be understood as taking the high-bit group of each element in the first query vector and the single key vector, respectively, to form a high-bit group vector, and then performing multiplication operation on the high-bit group vector to obtain the product of the high-bit group of the first query vector and the single key vector, denoted as the first product. The highest bit group of a single element is a group of bit values from the highest bit to a predetermined position (such as 4 bits, 2 bits, etc.).

[0075] It can be understood that the multiplication of the high-bit group vector is essentially still a dot product operation, but the data size is much smaller than the source data. For example, a single element in the source data contains 16 bits, and the high-bit group contains only the first 2 bits or 4 bits. The data amount of the dot product calculation is greatly reduced. In some optional embodiments, for the dot product calculation with a small amount of data, conventional methods such as splicing, pipelining calculation, and parallel calculation can be used to speed up the calculation and improve the calculation efficiency. In this way, each first product corresponding to the first query vector and each key vector can be obtained in batches or one by one. Each first product obtained can be stored in order of size, which will not be described here.

[0076] In the case of implementing sparse attention calculation by the circuit as shown in Figure 6 The storage process is described by taking the shift storage unit of the circuit as shown in Figure 6 The shift storage unit can include multiple shift registers. The data stored in the shift register can be moved in order. Here, a single shift register can store a single first product. Each first product can be written into the shift register one by one. In the case of writing a single first product (denoted as the current first product), for the first first product, it can be written into the first shift register as the largest first product. The first product written subsequently can compare it with other first products already written into the shift storage unit, so as to store the current first product in the shift storage unit according to the comparison result. The comparison order can be from back to front according to the order of the shift register, or from front to back, which is not limited here.

[0077] Specifically, the comparison result can be divided into the following two cases: there is another first product smaller than the current first product; there is no other first product smaller than the current first product. Assuming that only a predetermined number of first products and corresponding key vector identifiers are stored for the first query vector. In the case of detecting a stored first product smaller than the current first product, the current first product is inserted into the stored first product, moving the other first products smaller than and closest to the current first product and subsequent first products backward, and discarding the first products exceeding the predetermined number in size order; in the case that the number of stored first products is the predetermined number and no stored first product smaller than the current first product is detected, the current first product is discarded.

[0078] In combination Figure 6 The circuit shown, the number of shift registers is usually limited. Among them, in the case that there is another first product smaller than the current first product, the current first product can be inserted into the shift register storing another first product smaller than and closest to the current first product, and the first products in the other shift registers and subsequent shift registers are moved to the rear shift register. If there is a first product exceeding the number of shift registers in size order, it is discarded. The shift register storing another first product smaller than and closest to the current first product is determined as follows: if the comparison order is from front to back according to the shift register, it is the shift register corresponding to the first other first product smaller than the current first product; if the comparison order is from back to front according to the shift register, it is the shift register corresponding to the first other first product smaller than the current first product. In the case that no other first product smaller than the current first product is detected, if each shift register has a vacancy, the current first product is stored in the first vacancy from front to back, otherwise, if each shift register stores a first product, the current first product is discarded.

[0079] It is worth noting that the number of shift registers in the shift storage unit is usually fixed, and in actual business, all shift registers can be used to store first products, or a predetermined number of shift registers can be set as effective shift registers to store first products.

[0080] Then, by step 803, the corresponding candidate vector group is obtained according to the key vector identifier corresponding to the top-ranked predetermined number of first products.

[0081] Under the sparse attention mechanism, only one or more K vectors with the largest correlation degree with the Q vector are focused on and acted on the corresponding V vector for attention calculation. Usually, the stored first product is each first product ranked from large to small, corresponding to the K vector with a larger correlation degree with the Q vector.

[0082] Based on the predetermined quantity (e.g., k) at the top of the list, in using Figure 6 In the circuit shown, the first product (which can be the same as the number of effective shift registers) can sequentially obtain a predetermined number of key vectors that are ranked first, as candidate key vectors, with the corresponding key vector identifiers serving as indices, such as... Figure 7 The numbers 2, 5, n, and 0 are shown in the diagram. When only a predetermined number of first products are stored, the candidate key vector can be the key vector corresponding to all stored first products. When the number of stored first products is greater than the predetermined number, the predetermined number of key vectors that appear first in the sequence are selected as candidate key vectors.

[0083] By identifying the key vector, the corresponding candidate vector group can be obtained from the KV storage unit. A single candidate vector group typically includes a single K vector (denoted as the candidate key vector) and a single V vector (denoted as the candidate value vector).

[0084] Further, in step 804, sparse attention is calculated based on the acquired candidate vector groups and the first products.

[0085] It's understandable that for sparse attention computation, the product of the Q vector and one or more K vectors with the highest correlation can be calculated, and this product is applied to the V vector as a correlation. When calculating the product of the Q vector and one or more K vectors with the highest correlation, the splitting method described earlier can be used, i.e., calculating the product between the split bit groups and then combining it with shifting to obtain the final dot product. For example, the product of Q and K1 is calculated as: {[(M0M1<<s)+M0L1+L0M1]<<s}+L0L1. This calculation process also requires the use of the corresponding first product. Correspondingly, in Figure 6 In the circuit shown, a shift storage unit can provide the first product M0M1 corresponding to the candidate vector group containing K1. Furthermore, M0L1, L0M1, and L0L1 can be further calculated and merged into a product of Q and K1 through shifting, addition, and other operations. Here, s represents the number of bits in the low-order bit group, such as 4. For ease of description, M0L1 + L0M1 can be denoted as the second product of Q and K1, and L0L1 as the third product of the second and third products of Q and K1. Thus, the correlation calculation in the above manner can be performed for each candidate key vector.

[0086] Furthermore, the computational unit can determine the weights of the corresponding candidate value vectors based on the product of Q and each candidate key vector, and then weight each candidate value vector to obtain the sparse attention computation result. The product of Q and each candidate key vector can be used as the corresponding weight, or it can be normalized using a normalization coefficient, such as dividing by the dimension d of the K vector. k The square root of the value is normalized and used as the corresponding weight, which is not limited here.

[0087] In view of the above process, the scheme for sparse attention calculation provided under the technical concept of the present specification improves the hardware circuit in combination with the change in the calculation manner. In terms of the calculation manner, in the sparse process of retrieving in the K vector by the Q vector in the attention calculation, the process of sparsifying the K vector by calculating the product of the high-bit groups of the Q vector and the K vector and comparing the sizes is used to replace the design of sparsifying the K vector by the prediction model. In this way, in the hardware design, the product of the high-bit groups of the Q vector and the K vector can be calculated by the calculation unit, and the high-bit groups can be stored in the shift register of the shift storage unit in the order, so as to effectively screen the K vector, obtain the corresponding KV vector group, and perform the sparse attention calculation by multiplexing the product of the high-bit groups stored in the shift register. The calculation manner under this technical concept provides a brand-new calculation idea, avoids introducing an additional screening module in the circuit design, and greatly improves the area efficiency and energy efficiency.

[0088] According to an embodiment, an apparatus for sparse attention calculation is also provided. The apparatus can be arranged in a computer, a terminal, a server with a certain calculation capability, as a control module to control the connected hardware circuit as shown in Figure 6 to perform sparse attention calculation.

[0089] Figure 9 An apparatus 900 for sparse attention calculation according to an embodiment is shown. As shown in Figure 9 , the apparatus 900 can include: a first obtaining unit 901, configured to obtain a first query vector and corresponding respective key vectors in an attention mechanism; a first calculation unit 902, configured to perform multiplication calculation of high-bit groups on the first query vector and the respective key vectors respectively, and store each first product obtained in descending order, a single first product being a dot product of a high-bit group vector of the first query vector and a high-bit group vector of a single key vector, the high-bit group vector being a vector composed of high-bit groups of respective elements of the corresponding query vector or key vector, and a single high-bit group being determined by a highest predetermined bit of the corresponding element; a second obtaining unit 903, configured to obtain a corresponding candidate vector group according to the key vector identifier corresponding to a predetermined number of first products arranged in front; and a second calculation unit 904, configured to perform sparse attention calculation based on the obtained respective candidate vector groups and the respective first products.

[0090] It is worth noting that, Figure 9 the apparatus 900 shown corresponds to the method described in Figure 8 , Figure 8 the corresponding description in the method embodiment shown is also applicable to the apparatus 900, and will not be described here again.

[0091] According to another aspect, an embodiment also provides a computer readable storage medium having stored thereon a computer program which, when executed in a computer, causes the computer to perform the method described in conjunction with Figure 8

[0092] According to yet another aspect, an embodiment also provides a computing device comprising: Figure 6 a circuit shown to implement sparse attention computation, or matrix multiplication computation; or, comprising a memory having stored therein executable code which, when executed by a processor, implements the method described in conjunction with Figure 8

[0093] Those skilled in the art should be aware that, in one or more examples described above, the functions described in the embodiments of the present specification can be implemented in hardware, software, firmware or any combination thereof. When implemented in software, the functions can be stored in a computer readable medium or transmitted as one or more instructions or codes on a computer readable medium.

[0094] The above description is a further detailed description of the technical concept of the present specification, purposes, technical solutions and beneficial effects, and it should be understood that the above description is only a specific embodiment of the technical concept of the present specification, and is not used to limit the protection scope of the technical concept of the present specification. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solutions of the embodiments of the present specification shall be included in the protection scope of the technical concept of the present specification.​​

Claims

1. A computing circuit comprising a computing unit, a shift storage unit and a KV storage unit, wherein: the computing unit comprises at least one processor configured to compute matrix multiplication, and compute first products and dot products between query vectors and key vectors in a sparse attention mechanism, a single first product being a dot product of a high-bit group vector of a single query vector and a high-bit group vector of a single key vector, the high-bit group vector being a vector composed of high-bit groups of respective elements of the query vector or the key vector, a single high-bit group being determined by a highest predetermined bit of a respective element; the shift storage unit comprises a plurality of shift registers configured to store the first products and corresponding key vector identifiers in a descending order of size through the shift registers, and provide the computing unit with first products within a predetermined number from large to small for computing dot products between the query vectors and the key vectors, and provide the KV storage unit with the corresponding key vector identifiers as indexes for obtaining vector groups composed of key vectors and value vectors; and the KV storage unit is configured to store the vector groups and provide the computing unit with vectors in the vector groups. The circuit further comprises a configuration register configured to configure a working mode of the circuit, the working mode comprising a first mode and a second mode; wherein: in the first mode, the circuit performs sparse attention computation using the computing unit, the shift storage unit and the KV storage unit; and in the second mode, the shift storage unit is bypassed, and the circuit performs matrix multiplication computation by the computing unit from the KV storage unit to obtain the vector groups. The number of effective shift registers is the predetermined number, and first products arranged in front of the predetermined number in a descending order are written into the effective shift registers, and first products arranged outside the predetermined number are discarded. 4.A method of sparse attention computation, the method comprising: obtaining a first query vector and respective key vectors in an attention mechanism; performing high-bit group multiplication computation on the first query vector and the respective key vectors respectively to obtain respective first products, a single first product being a dot product of a high-bit group vector of the first query vector and a high-bit group vector of a single key vector, the high-bit group vector being a vector composed of high-bit groups of respective elements of the query vector or the key vector, a single high-bit group being determined by a highest predetermined bit of a respective element; obtaining respective candidate vector groups according to key vector identifiers corresponding to a predetermined number of first products arranged in front, a single candidate vector group comprising a single key vector and a single value vector; and performing sparse attention computation based on the respective candidate vector groups and the respective first products.

2. The circuit of claim 1, wherein, The obtaining of the respective first products in a descending order comprises: comparing the current first product with the stored respective first products one by one; and determining a storage mode for the current first product based on a comparison result. For the first query vector, only a predetermined number of first products and corresponding key vector identifiers are stored, and the determining of the storage mode for the current first product based on the comparison result comprises: ​ 3. The circuit of claim 1, wherein, ​ ​ ​ ​ ​ ​ 5. The method of claim 4, wherein, ​ ​ ​ 6. The method of claim 5, wherein, ​ in case that a stored first product less than the current first product is detected, inserting the current first product into the stored first products, moving other first products less than and closest to the current first product and subsequent first products backward, and discarding first products whose size order exceeds a predetermined number; in case that the number of stored first products is the predetermined number and no stored first product less than the current first product is detected, discarding the current first product.

7. The method of claim 4, wherein, the sparse attention calculation based on each candidate vector group and each first product comprises: calculating second products and third products of the first query vector and candidate key vectors in each candidate vector group, wherein a single second product comprises a sum of a product of a high bit group vector of the first query vector and a low bit group vector of a single candidate key vector and a product of a low bit group vector of the first query vector and a high bit group vector of the single candidate key vector, and a single third product is a product of a low bit group vector of the first query vector and a low bit group vector of the single candidate key vector, and a single low bit group vector is a vector composed of low bits of each element of a corresponding query vector or candidate key vector, and the low bits are determined according to bits other than the highest predetermined bit; determining each attention weight corresponding to each candidate key vector, and a single attention weight corresponding to a single candidate key vector is positively correlated with a value obtained by adding a single second product and a single first product left shifted by s bits, and then adding a single third product left shifted by s bits again, wherein s is the number of bits of the low bit group; performing weighted summation on value vectors in each candidate vector group by using each attention weight to obtain an attention vector.

8. The method of claim 4, wherein, in case that the calculation unit comprises multiple processors, the multiple processors can be used for serial or parallel processing in the vector multiplication process.

9. The method of claim 4, wherein, the method is implemented by the circuit of claim 1. 10.A computer readable storage medium having stored thereon a computer program which, when executed in a computer, causes the computer to perform the method of any one of claims 4-9. 11.A computing device comprising at least one of: the computing circuit of any one of claims 1-3; a processor and a memory, and the memory stores executable code, and the processor executes the executable code to implement the method of any one of claims 4-9.

Images