Head-based symbol pruning method and electronic device

Abstract

The present disclosure provides a symbol pruning method and an electronic device. The method includes: calculating a plurality of query matrices and a plurality of key matrices corresponding to a plurality of heads; for each head, calculating an attention matrix corresponding to a plurality of symbols and including a plurality of attention values according to the corresponding query matrix and the key matrix; for each head, calculating a corresponding importance of each symbol according to the corresponding attention value, and determining at least one pruned symbol according to the importance, wherein the pruned symbols of the first head are different from the pruned symbols of the second head; and omitting the respective corresponding pruned symbols of the heads in subsequent calculations. Thus, the amount of calculation can be reduced.

Description

Technical Field

[0001] This disclosure relates to a head-based symbol pruning method and an electronic device using this method. Background Technology

[0002] With the continuous evolution of artificial intelligence technology, the field of Natural Language Processing (NLP) has widely adopted deep neural network models based on the Transformer architecture to solve various language tasks, such as machine translation, semantic analysis, dialogue generation, and question answering systems. Transformer models, through multi-layered self-attention mechanisms and feedforward neural networks, can effectively capture long-distance dependencies between tokens in the input sequence and improve semantic representation capabilities. However, with the increasing demands of applications, the number of parameters and computational complexity of these models have also increased significantly, leading to enormous computational resource and time costs in practical deployments, especially in edge devices or low-latency applications.

[0003] To address these issues, various model compression techniques have been proposed, one of which is symbol pruning. Pruning reduces the computational burden of subsequent layers by removing symbols that have minimal impact on the results. Such methods can effectively reduce computational load while maintaining prediction accuracy. However, most common symbol pruning methods currently employ fixed pruning ratios and cannot dynamically adjust the pruning strategy according to different contexts, resulting in limited effectiveness when handling diverse inputs. Summary of the Invention

[0004] In order to perform dynamic pruning, this disclosure proposes a symbolic pruning method and electronic device.

[0005] According to one aspect of this disclosure, a symbol pruning method is provided, comprising: calculating a plurality of query matrices and a plurality of key matrices corresponding to a plurality of heads; for each head, calculating an attention matrix based on the corresponding query matrix and key matrix, the attention matrix corresponding to a plurality of symbols and including a plurality of attention values; for each head, calculating the importance corresponding to each symbol based on the attention values, and determining at least one pruned symbol based on the importance, wherein the pruned symbol of a first head is different from the pruned symbol of a second head; and omitting the pruned symbols corresponding to each head in subsequent calculations.

[0006] According to another aspect of this disclosure, an electronic device is provided, including a memory and a processor. The memory stores a plurality of symbols. The processor is electrically connected to the memory for reading symbols and performing the symbol pruning method described above.

[0007] To make the above-described features and advantages of this disclosure more apparent and understandable, specific embodiments are described below in conjunction with the accompanying drawings. Attached Figure Description

[0008] Figure 1 This is a schematic diagram of an electronic device according to one embodiment.

[0009] Figure 2 This is an architectural diagram of a neural network according to one embodiment.

[0010] Figure 3 This is a schematic diagram illustrating an attention matrix for multiple heads according to one embodiment.

[0011] Figure 4 This is a schematic diagram illustrating, according to one embodiment, how to determine an importance threshold based on the number of variants.

[0012] Figure 5 This is a schematic diagram illustrating the multiplication of the attention matrix and the value matrix using a mask, according to one embodiment.

[0013] Figure 6 This is a schematic diagram illustrating the use of a mask to add matrix T to matrix O according to one embodiment.

[0014] Figure 7 This is a schematic diagram illustrating a sparse-dense-sparse mapping according to one embodiment.

[0015] Figure 8 This is a diagram illustrating the internal hardware configuration of processor 110 according to one embodiment.

[0016] Figure 9 This is a flowchart illustrating a symbol pruning method according to an embodiment. Detailed Implementation

[0017] Some embodiments of this disclosure will now be described in detail with reference to the accompanying drawings. Component symbols used in the following description, when appearing in different drawings, are considered to be the same or similar components. These embodiments are only a part of this disclosure and do not reveal all possible implementations of this disclosure. More precisely, these embodiments are merely examples of systems and methods within the scope of this patent application.

[0018] The terms "first," "second," etc., used in this article do not specifically refer to order or sequence; they are merely used to distinguish elements or operations described using the same technical terms.

[0019] Figure 1 This is a schematic diagram illustrating an electronic device according to one embodiment. Please refer to... Figure 1The electronic device 100 may be a smartphone, tablet computer, personal computer, notebook computer, server, distributed computer, cloud server, industrial computer, graphics card, or various electronic devices with computing capabilities, etc., and this disclosure is not limited thereto. The electronic device 100 includes a processor 110 and a memory 120, with the processor 110 electrically connected to the memory 120. The processor 110 may be a central processing unit, a deep-learning processing unit (DPU), a neural network processing unit (NPU), a tensor processing unit (TPU), an application-specific integrated circuit (ASIC), a programmable logic device (PLD), etc. The memory 120 may be random access memory, flash memory, floppy disk, hard disk, optical disk, USB flash drive, magnetic tape, or a database accessible via the Internet, which stores various data required to execute a neural network, including weights, input symbols, etc. The processor 110 reads this data from the memory 120. In particular, the processor 110 executes a symbol pruning method, thereby omitting the computation of some symbols in the neural network.

[0020] Figure 2 This is a diagram illustrating the architecture of a neural network according to an embodiment. Please refer to... Figure 2 First, a symbol sequence 210 is received, which includes N symbols, where N is a positive integer. In this embodiment, each symbol represents a word, but in other embodiments, each symbol may also represent one or more letters, one or more pixels, a sound signal, or other types of data, which is not limited to this.

[0021] Next, in step 220, the embedding for each symbol is calculated, for example, using a linear or convolutional layer. Each embedding is a vector of length D, where D is a positive integer. Therefore, the symbol sequence 210 can be represented as a matrix T of size N×D. The neural network consists of multiple layers; the computation of the first layer 230 is described here.

[0022] The first layer 230 has H headers, where H is a positive integer. Taking header 240 as an example, matrix T first passes through three linear layers 241-243. The weight matrices used in these three linear layers 241-243 are as follows: , ,as well as , where h represents the h-th head. The size of these weight matrices is... However, this disclosure is not limited thereto; in other embodiments, the weight matrix may have more or fewer columns. Linear layers 241-243 are used to calculate the value matrix. Query matrix and the key matrix .

[0023] Based on the query matrix and the key matrix An attention matrix can be calculated. For example, the query matrix... Bond matrix After multiplying by the transposes, and then applying the normalized exponential function (e.g., softmax), the attention matrix can be obtained. This calculation can be expressed as the following formula 1.

[0024] (Formula 1)

[0025] Attention Matrix The matrix is ​​of size N×N, where each row and each column corresponds to a symbol, and each element in the matrix is ​​also called an attention value (or attention probability). A corresponding value matrix can be computed for each head. Query matrix and the key matrix The corresponding attention matrix can also be calculated. Next, for each head, based on the attention matrix... The attention value is used to calculate the importance of each symbol, and based on the importance, it is determined which symbols will be pruned (also known as pruned symbols).

[0026] Figure 3 This is a schematic diagram illustrating attention matrices for multiple heads according to one embodiment. Attention matrix 310 for a first head and attention matrix 320 for a second head are shown here. In this example, there are four symbols: "Awful", "if", "your're", and "blind". The symbols processed by different heads are the same. The normalization exponent function 244 mentioned above is applied to elements in the same row of the attention matrix; therefore, the attention values ​​in the same row of attention matrices 310 and 320 will sum to 1. In some embodiments, for each symbol, the importance can be obtained by summing the corresponding attention values ​​(in the same column). This calculation can be expressed as the following formula 2.

[0027] (Formula 2)

[0028] in This represents the attention value in the i-th column and n-th row of the attention matrix corresponding to the h-th head. Importance This corresponds to the h-th head and the n-th symbol. For example, summing the attention values ​​in the first column of attention matrix 310 yields an importance value of "1.4". Similarly, summing up the attention values ​​in the first column of attention matrix 320 yields an importance value of "1.67". It is worth noting that because different heads focus on different runes, the same rune will be calculated with different importance in different heads. For example, in the first head, the fourth rune is the second most important (importance of 1.28), but in the second head, the third rune is the second most important (importance of 0.96).

[0029] Furthermore, summing up the importance of all heads corresponding to the same symbol element yields the layer importance. The calculation is as follows: Formula 3.

[0030] (Formula 3)

[0031] in This represents the layer importance of the nth symbol. There are several ways to determine which symbols are to be pruned. In this embodiment, in each header, it can be determined whether the importance of each symbol is less than an importance threshold; if so, the symbol is considered a pruned symbol. In other embodiments, the K most important symbols can be retained in each header, and the other symbols can be designated as pruned symbols, where K is a positive integer. Therefore, the pruned symbols in the first header may differ from those in the second header. In some cases, the number of pruned symbols in the first header may differ from the number of pruned symbols in the second header. Related techniques utilize layer importance. The method determines which symbol to be pruned (e.g., the second symbol), so the pruned symbols are the same in different heads. However, in this invention, different symbols can be pruned in different heads. In some experiments, related techniques reduced accuracy by 1.1% after pruning, but the method in this embodiment only reduces accuracy by 0.5% after pruning the same number of symbols, thus achieving better performance.

[0032] Several approaches are proposed to determine the aforementioned importance threshold. In one embodiment, for each head, the dispersion of importance of all symbols can be calculated, and the dispersion can be multiplied by a constant to calculate the importance threshold corresponding to that head. Since the dispersion of different heads may be different, the importance threshold of the first head may be different from the importance threshold of the second head. The dispersion can be determined by variance or entropy.

[0033] Figure 4 This is a schematic diagram illustrating how an importance threshold is determined based on the number of variants, according to one embodiment. Please refer to... Figure 4 Chart 410 corresponds to the first head, and Chart 420 corresponds to the fourth head. The horizontal axis of Charts 410 and 420 indicates which is more important, and the vertical axis is the numerical value of the importance. Figure 4 This is just an example; its importance is not the same as... Figure 3 The calculated importance values ​​are shown. The variance for the four importance values ​​in Chart 410 is 1.61, while the variance for the four importance values ​​in Chart 420 is 0.27. A larger variance indicates that importance is concentrated in a few symbols, making it easier to determine which symbols are more important. Therefore, a larger importance threshold can be set to prune multiple symbols. Conversely, a smaller variance indicates that the importance of these symbols is close to each other, making it difficult to distinguish which symbols are more important. Therefore, a smaller importance threshold can be set to retain more symbols. In this example, the constant is "0.97", so the importance threshold 412 for the first head is "1.56", and the importance threshold 422 for the fourth head is "0.26". The pruned symbols in the first head are the first, second, and third symbols, while the pruned symbol in the fourth head is the second symbol.

[0034] In some embodiments, the importance threshold can be determined based on entropy. For example, for each header, the importance of all symbols can be normalized using an exponential function so that the sum equals 1 (representing probability), and then the entropy of these importance values ​​can be calculated. The entropy calculated based on the importance in Chart 410 is smaller, while the entropy calculated based on the importance in Chart 420 is larger. Therefore, an importance threshold and entropy can be set to be negatively correlated. For example, the inverse of the entropy can be multiplied by a constant to obtain the importance threshold, or a constant can be subtracted from the entropy to obtain the importance threshold.

[0035] Based on the above approach, different importance thresholds can be set for different heads. In some embodiments, the same importance threshold can also be set for different heads. For example, please refer to... Figure 3 After calculating the layer importance of all symbols Then, the dispersion of importance of these layers can be calculated (e.g., the variance), and then the dispersion can be multiplied by a constant to calculate the importance threshold. In such an example, although different heads use the same importance threshold, different pruned symbols will still be determined because the importance in different heads is different.

[0036] The aforementioned constants can be determined in any suitable way, such as through genetic algorithms, particle swarm optimization, or by human intervention; this invention is not limited to these methods. For example, when using a genetic algorithm, the following formula 4 can be used as the fitness score.

[0037] (Formula 4)

[0038] Where x represents a candidate constant, and there are multiple candidate constants in each iteration of the gene algorithm. L represents the reduction in accuracy of the model by using this candidate constant. This represents the average value of all candidate constants. Those skilled in the art will understand the gene algorithm, and it will not be described in detail here.

[0039] Please refer to Figure 2 After determining the pruned symbols, the pruned symbols corresponding to these heads are omitted in subsequent calculations. This subsequent calculation may include the attention matrix. Sum matrix The computation involves multiplication, calculation of the feedforward neural network, and / or calculation of the next layer. In some embodiments, a mask is generated based on the pruned symbols, which can then be applied to subsequent calculations. For example, the attention matrix is ​​calculated in step 245. Sum matrix This mask can be used when multiplying.

[0040] Figure 5 This is a schematic diagram illustrating the multiplication of the attention matrix and the value matrix using a mask, according to one embodiment. Please refer to... Figure 4 and Figure 5 ,continue Figure 4For example, attention matrix 510 belongs to the first head, where the first, second, and third symbols are pruned symbols. This mask can be represented using two-dimensional data, or in other forms (e.g., as a sequence indicating the first, second, and third symbols). When the mask is applied to attention matrix 510, since columns 1-3 and rows 1-3 correspond to pruned symbols, only the elements in the fourth column and fourth row remain. Furthermore, rows 1-3 in value matrix 511 also correspond to pruned symbols and are therefore omitted. Multiplying attention matrix 510 by value matrix 511 using the corresponding mask yields attention output 512. In attention output 512, columns 1-3 also correspond to pruned symbols and are therefore masked elements (e.g., null values ​​or the value "0"). Similarly, attention matrix 520 belongs to the fourth head, continuing... Figure 4 In the example, the second symbol is a pruned symbol. Therefore, the mask of the fourth head is used to indicate the second symbol. The elements in the second column and second row of the attention matrix 520 are masked, and the second row of the value matrix 512 is masked. Multiplying the attention matrix 520 by the value matrix 521 using this mask yields the attention output 522, where the second row includes multiple masked elements.

[0041] Please refer to Figure 2 Next, in step 250, the attention outputs of all heads will be concatenated. Here, we use... Let represent the attention output corresponding to the h-th head, where h = 1 to H. When concatenated, they can be represented as a matrix. Its size is N×D. Next, in step 260, addition and normalization are performed. The addition operation, also called the residual layer, is used to add matrix O to matrix T to obtain matrix F, as shown below. Figure 6 As shown, Figure 6 This is a schematic diagram illustrating the addition of matrix O to matrix T using a mask, according to one embodiment. In other words, during the addition, the masked elements in matrix O are not added to matrix T, or the masked elements in matrix O are the value "0" and therefore do not affect the corresponding values ​​in matrix T. In this example, the second and third symbols in the first head (h=1) are pruned symbols, and the second symbol in the second head (h=2) is a pruned symbol. Furthermore, normalization is, for example, subtracting the mean from the values ​​in the matrix and then dividing by the standard deviation. In some embodiments, step 260 can be removed or replaced with other calculations; for generalization, matrix F is referred to herein as including multiple attention outputs.

[0042] Next, in Figure 2Step 270 involves feedforwarding. In other words, matrix F is input to a feedforward neural network (FFN), which includes a linear layer 281, an activation function 282, and a linear layer 283. Linear layer 281 multiplies the input by matrix W1, and linear layer 283 multiplies the input by matrix W2. In some embodiments, activation function 282 is a Gaussian Error Linear Unit (GeLU). Since the operations of linear layers 281 and 283 are identical, only one is described here. Specifically, because different heads contain different pruned symbols, matrix F is sparse. Therefore, matrix F is split into multiple dense matrices, multiplied, and then merged into a sparse matrix. This process is called sparse-dense-sparse mapping.

[0043] Figure 7 This is a schematic diagram illustrating a sparse-dense-sparse mapping according to one embodiment. Please refer to... Figure 7 The four columns of matrix F correspond to four symbols: "Awful", "if", "you're", and "blind", with the numbers 1 through 4 representing the symbol indices. Matrix F includes the attention outputs of multiple heads. In the attention output of the first head (h=1), the second and third symbols are occluded, and in the second head (h=2), the second symbol is occluded. For each head, the occluded elements in the corresponding attention output are removed to obtain one or more dense matrices. For example, the first attention output, after removing the occluded elements, becomes a dense matrix F1, where the two numbers "1" and "4" on the left represent the indices of the remaining symbols. Similarly, the second attention output, after removing the occluded elements, becomes a dense matrix F. 2,1 and matrix F 2,2 The first, third, and fourth symbols are retained. In this embodiment, the size of each dense matrix is ​​2, but in other embodiments it can be any value. Since three symbols are retained in the second attention output, the resulting second matrix F... 2,2 Some elements are still obscured.

[0044] Furthermore, the size of matrix W1 is ,in The elements are positive integers. The attention output of the first head (h=1) in matrix F is located in the first and second columns, thus corresponding to the first and second rows of matrix W1 in matrix multiplication. Similarly, the attention output of the second head (h=2) in matrix F is located in the third and fourth columns, corresponding to the third and fourth rows of matrix W1 in matrix multiplication. Therefore, the elements of the first and second rows of matrix W1 are extracted to form a submatrix W. 1,1 And extract the elements of the third and fourth rows of matrix W1 to form a submatrix W. 1,2 .

[0045] Next, the dense matrix is ​​multiplied by its corresponding submatrix to obtain the dense result matrix. For example, the dense matrix F1 is multiplied by the submatrix W. 1,1 The resulting dense matrix is ​​711. Dense matrix F 2,1 Multiply by submatrix W 1,2 The resulting dense matrix is ​​712. Matrix F 2,2 Multiply by submatrix W 1,2 The resulting matrix is ​​713, in which the elements in the second row are masked.

[0046] Next, for each head, these dense result matrices are reduced to sparse result matrices based on the content of the mask. For example, dense result matrix 711 can be reduced to sparse result matrix 721, while dense result matrices 712 and 713 can be reduced to sparse result matrix 722. Alternatively, elements in the dense result matrices are filled into the sparse result matrices according to the indices of the unmasked symbols. For example, two rows from dense result matrix 711 are filled into the first and fourth rows of sparse result matrix 721, respectively; two rows from dense result matrix 712 are filled into the first and third rows of sparse result matrix 722; and the first row of matrix 713 is filled into the fourth row of sparse result matrix 722.

[0047] Sparse result matrix 721 belongs to the first head. Sparse result matrix 722 belongs to the second head. Next, element-wise addition is performed on the sparse result matrices corresponding to all heads to obtain the feedforward result. In this example, sparse result matrix 721 and sparse result matrix 722 are element-wise added to obtain the feedforward result matrix 730. This feedforward result matrix 730 is the same as the result of directly multiplying matrix F by matrix W1. If matrix F is directly multiplied by matrix W1, some elements are masked, making the related calculations invalid and wasting computational resources. Conversely, most of the multiplications in the above approach deal with dense matrices. This improves hardware utilization and accelerates the computation speed of the neural network.

[0048] Please refer to Figure 2Next, step 280 is performed, involving addition and normalization, ultimately producing matrix Y. Matrix Y can be passed to the next layer or used as the output of the neural network. The output of the neural network can be used for classification, regression, generation, and segmentation, etc., but this disclosure is not limited thereto.

[0049] In the above embodiments, the calculated mask is used within the same layer. In other embodiments, the calculated mask can be applied to the next layer (referred to as the second layer). For example, the pruned symbols calculated in the first layer are omitted from the headers of the second layer. In some embodiments, multiple layers of masks can be combined, for example, by voting to determine the masked symbols. This delays symbol pruning but avoids deleting important symbols.

[0050] Figure 8 This is a diagram illustrating the internal hardware configuration of processor 110 according to one embodiment. Please refer to... Figure 8 The processor 110 includes a controller 810, an input memory 811, a weight memory 812, a direct memory access (DMA) circuit 813, a systolic array 814, a normalized exponential function circuit 815, a normalization circuit 816, a threshold determination circuit 817, a pruning circuit 818, and an excitation function circuit 819.

[0051] The memory 120 stores the input data (including multiple symbols) and weights. The controller 810 controls the entire processor's operation, including data transfer, initiating computation, and triggering the execution of various circuits. The DMA circuit 813 reads the input data from the memory 120 into the input memory 811 and reads the weights into the weight memory 812. The systolic array 814 is used to perform matrix multiplication. The systolic array 814 includes multiple processing elements (PEs). In this embodiment, due to the use of a sparse-dense-sparse mapping, the processing elements are used more efficiently. For example, the size of the dense matrix is ​​2×2, thus ensuring the use of a 2×2 PE array. If matrix multiplication is performed on a sparse matrix, some PEs will be idle. The normalization exponential function circuit 815 is used to perform the normalization exponential function. The normalization circuit 816 is used to perform normalization. The threshold determination circuit 817 is used to calculate the importance threshold based on the dispersion of importance; for example, the threshold determination circuit 817 can calculate the variance or entropy. The pruning circuit 818 determines which symbols should be pruned. The excitation function circuit 819 executes the excitation function. However, Figure 8 This is merely an example and does not limit the internal hardware configuration of the processor 110.

[0052] Figure 9This is a flowchart illustrating a symbol pruning method according to an embodiment. Please refer to... Figure 9 In step 901, multiple query matrices, multiple key matrices, and multiple value matrices corresponding to multiple headers are calculated. In step 902, for each header, an attention matrix is ​​calculated based on the corresponding query matrix and key matrix. This attention matrix corresponds to multiple symbols and includes multiple attention values. In step 903, for each header, the importance corresponding to each symbol is calculated based on the attention value, and at least one pruned symbol is determined based on the importance, wherein the pruned symbol for the first header is different from the pruned symbol for the second header. In step 904, the pruned symbols corresponding to each header are omitted in subsequent calculations.

[0053] Although this disclosure has been provided above with reference to embodiments, it is not intended to limit this disclosure. Anyone skilled in the art can make some modifications and refinements without departing from the scope of this disclosure. Therefore, the scope of protection of this disclosure shall be determined by the appended claims.

Claims

1. A symbol pruning method, characterized in that, For use in electronic devices, the symbol pruning method includes: Calculate multiple query matrices corresponding to multiple headers and multiple key matrices; For each of the plurality of headers, an attention matrix is ​​calculated based on the corresponding query matrix and the key matrix, wherein the attention matrix corresponds to a plurality of symbols and includes a plurality of attention values; For each of the plurality of heads, the importance corresponding to each of the plurality of symbols is calculated based on the plurality of attention values, and the pruned symbols among the plurality of symbols are determined based on the importance, wherein the plurality of heads includes a first head and a second head, and the pruned symbols of the first head are different from the pruned symbols of the second head; and In subsequent calculations, the pruned symbols corresponding to each of the multiple heads are omitted.

2. The symbol pruning method according to claim 1, characterized in that, The calculation of the importance of each symbol based on the multiple attention values ​​includes: The importance is obtained by accumulating the attention values ​​corresponding to one of the plurality of symbols.

3. The symbol pruning method according to claim 2, characterized in that, The pruned symbols among the plurality of symbols, determined according to their importance, include: For one of the plurality of symbols, determine whether the corresponding importance is less than an importance threshold; if so, determine that one of the plurality of symbols belongs to the pruned symbol.

4. The symbol pruning method according to claim 3, characterized in that, Also includes: For each of the plurality of heads, the dispersion of the plurality of importance of the plurality of symbols is calculated, and the dispersion is multiplied by a constant to calculate the importance threshold corresponding to the head, wherein the importance threshold of the first head is different from the importance threshold of the second head.

5. The symbol pruning method according to claim 4, characterized in that, The degree of dispersion is referred to as the variance.

6. The symbol pruning method according to claim 3, characterized in that, Also includes: For each of the plurality of symbols, the plurality of importance corresponding to the plurality of heads are summed to obtain the layer importance; The dispersion of the importance of the plurality of symbols across the plurality of layers is calculated, and the dispersion is multiplied by a constant to calculate the importance threshold, wherein the importance threshold of the first head is the same as the importance threshold of the second head.

7. The symbol pruning method according to claim 1, characterized in that, The step of omitting the pruned symbol corresponding to each of the plurality of heads in the subsequent calculation includes: For each of the plurality of heads, a mask is generated based on the corresponding pruned symbol; and For each of the plurality of heads, the corresponding attention matrix is ​​multiplied by the corresponding value matrix using the corresponding mask to obtain an attention output, wherein the attention output includes a plurality of masked elements corresponding to the pruned symbols.

8. The symbol pruning method according to claim 7, characterized in that, Also includes: For each of the plurality of heads, remove the plurality of occluded elements in the corresponding attention output to obtain a dense matrix, multiply the dense matrix by a submatrix of the weight matrix to obtain a dense result matrix, and restore the dense result matrix to a sparse result matrix according to the corresponding mask.

9. The symbol pruning method according to claim 7, characterized in that, Also includes: The feedforward result matrix is ​​obtained by summing the multiple sparse result matrices corresponding to the multiple heads element by element.

10. The symbol pruning method according to claim 1, characterized in that, Where the plurality of queries and the plurality of key matrices belong to the first layer of the neural network, the step of omitting the pruned symbols corresponding to each of the plurality of heads in the subsequent calculations includes: In the second layer of the neural network, the corresponding pruned symbols are omitted from the plurality of heads.

11. An electronic device, characterized in that, include: Memory, used to store multiple symbols; as well as A processor, electrically connected to the memory, is used to read the plurality of symbols and execute a plurality of steps: Calculate multiple query matrices corresponding to multiple headers and multiple key matrices; For each of the plurality of headers, an attention matrix is ​​calculated based on the corresponding query matrix and the key matrix, wherein the attention matrix corresponds to the plurality of symbols and includes a plurality of attention values; For each of the plurality of heads, the importance of each symbol corresponding to the plurality of symbols is calculated based on the plurality of attention values, and the pruned symbol among the plurality of symbols is determined based on the importance, wherein the plurality of heads includes a first head and a second head, and the pruned symbol of the first head is different from the pruned symbol of the second head. as well as In subsequent calculations, the pruned symbols corresponding to each of the multiple heads are omitted.

12. The electronic device according to claim 11, characterized in that, The calculation of the importance of each symbol based on the multiple attention values ​​includes: The importance is obtained by accumulating the attention values ​​corresponding to one of the plurality of symbols.

13. The electronic device according to claim 12, characterized in that, The pruned symbols among the plurality of symbols, determined according to their importance, include: For one of the plurality of symbols, determine whether the corresponding importance is less than an importance threshold; if so, determine that one of the plurality of symbols belongs to the pruned symbol.

14. The electronic device according to claim 13, characterized in that, The plurality of steps further include: For each of the plurality of heads, the dispersion of the plurality of importance of the plurality of symbols is calculated, and the dispersion is multiplied by a constant to calculate the importance threshold corresponding to the head, wherein the importance threshold of the first head is different from the importance threshold of the second head.

15. The electronic device according to claim 14, characterized in that, The degree of dispersion is referred to as the variance.

16. The electronic device according to claim 13, characterized in that, The plurality of steps further include: For each of the plurality of symbols, the plurality of importance corresponding to the plurality of heads are summed to obtain the layer importance; The dispersion of the importance of the plurality of symbols across the plurality of layers is calculated, and the dispersion is multiplied by a constant to calculate the importance threshold, wherein the importance threshold of the first head is the same as the importance threshold of the second head.

17. The electronic device according to claim 11, characterized in that, The step of omitting the pruned symbol corresponding to each of the plurality of heads in the subsequent calculation includes: For each of the plurality of heads, a mask is generated based on the corresponding pruned symbol; and For each of the plurality of heads, the corresponding attention matrix is ​​multiplied by the corresponding value matrix using the corresponding mask to obtain an attention output, wherein the attention output includes a plurality of masked elements corresponding to the pruned symbols.

18. The electronic device according to claim 17, characterized in that, The plurality of steps further include: For each of the plurality of heads, remove the plurality of occluded elements in the corresponding attention output to obtain a dense matrix, multiply the dense matrix by a submatrix of the weight matrix to obtain a dense result matrix, and restore the dense result matrix to a sparse result matrix according to the corresponding mask.

19. The electronic device according to claim 17, characterized in that, The plurality of steps further include: The feedforward result matrix is ​​obtained by summing the multiple sparse result matrices corresponding to the multiple heads element by element.

20. The electronic device according to claim 11, characterized in that, Where the plurality of query matrices and the plurality of key matrices belong to the first layer of the neural network, the step of omitting the pruned symbols corresponding to each of the plurality of heads in the subsequent calculations includes: In the second layer of the neural network, the corresponding pruned symbols are omitted from the plurality of heads.

Images