Calculation acceleration method and device for model reasoning, medium and program product

Abstract

The embodiment of the invention discloses a model reasoning calculation acceleration method and device, a medium and a program product, and the method comprises the steps: dividing the number of heads in multi-head attention processing into each processor core group according to the number of processor core groups in a processor in the multi-head attention processing of model reasoning; when the processor core group performs operation between a query vector and a key vector in multi-head attention processing, according to the total amount of the operation tasks and the number of the slave cores in the processor core group, the operation tasks are divided into the slave cores with uniform data volume; when an operation task is jointly processed by at least two slave cores, matrix segmentation is carried out on a query vector and a key vector in the operation task, and each slave core reads a corresponding vector according to a matrix segmentation result to carry out matrix multiplication operation; and caching the first matrix multiplication operation sub-result obtained by calculation of each slave core to a high-speed cache of a specified first target slave core to generate a first matrix multiplication operation result. The calculation efficiency can be improved by balancing the data volume of each slave core.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a computing acceleration method, device, medium and program product for model reasoning. Background Art

[0002] With the advancement of computer science and artificial intelligence, large language models are becoming increasingly common in our daily lives. Within these models, the self-attention mechanism is a core structure of many language dialogue models. Reasoning in large language models, such as the Transformer model, typically involves prefilling and decoding phases. The decoding phase requires a high frequency of memory access, especially during the calculation of the self-attention module. Consequently, the calculation of the self-attention module faces a memory access bottleneck, making optimization of the self-attention module's computation crucial for accelerating model inference.

[0003] In the existing technology, operator fusion is usually used, such as fusing the calculation processes between the query vector Q, key vector K, and value vector V in the self-attention module, to optimize memory access and calculation processes and improve the inference rate.

[0004] However, the existing technical solutions do not consider the impact of hardware architecture on the inference calculation process, and thus fail to fully utilize the parallel computing advantages of multi-core processors. At the same time, they do not fully optimize data storage and fail to fully utilize hardware bandwidth. Summary of the Invention

[0005] The present invention provides a computational acceleration method, device, medium, and program product for model reasoning, which can give full play to the parallel computing advantages of multiple cores, improve the reasoning speed, and fully utilize the hardware bandwidth.

[0006] According to one aspect of the present invention, a method for accelerating computation of model reasoning is provided, the method comprising:

[0007] When performing multi-head attention processing for model inference, dividing the number of heads in the multi-head attention processing into each processor core group according to the number of processor core groups in the processor;

[0008] When the processor core group performs operations between query vectors and key vectors in the multi-head attention processing of model inference, the operations are evenly divided among the slave cores according to the total number of operations and the number of slave cores in the processor core group.

[0009] When a computing task is jointly processed by at least two slave cores, the query vector and the key vector in the computing task are matrix-sliced, and each slave core reads the corresponding query sub-vector and the key sub-vector according to the matrix-slicing result to perform matrix multiplication operation;

[0010] A first target slave core is designated in each slave core, and first matrix multiplication operator results calculated by each slave core are cached in a cache of the first target slave core to generate a first matrix multiplication operation result.

[0011] According to another aspect of the present invention, a computing acceleration device for model reasoning is provided, the device comprising:

[0012] An attention head partitioning module is used to partition the number of heads in the multi-head attention processing into respective processor core groups according to the number of processor core groups in the processor when performing multi-head attention processing for model inference;

[0013] The computation task partitioning module is used to divide the computation tasks between the query vector and the key vector in the multi-head attention processing of the model inference in the processor core group, and divide the computation tasks into each slave core with a uniform data volume according to the total number of computation tasks and the number of slave cores in the processor core group;

[0014] A matrix splitting module is used to perform matrix splitting on the query vector and key vector in a computing task when a computing task is jointly processed by at least two slave cores. Each slave core reads the corresponding query sub-vector and key sub-vector according to the matrix splitting result to perform matrix multiplication operation;

[0015] The first matrix multiplication operation result generation module is used to specify the first target slave core in each slave core, cache the first matrix multiplication operator results calculated by each slave core into the cache of the first target slave core to generate the first matrix multiplication operation result.

[0016] According to another aspect of the present invention, an electronic device is provided, comprising:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the computational acceleration method for model reasoning described in any embodiment of the present invention.

[0020] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the computational acceleration method for model reasoning described in any embodiment of the present invention when executed.

[0021] According to another aspect of the present invention, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the computational acceleration method for model reasoning according to any embodiment of the present invention.

[0022] The technical solution of the embodiment of the present invention is to divide the number of heads in the multi-head attention processing into each processor core group according to the number of processor core groups in the processor when performing multi-head attention processing for model inference; when the processor core group performs operations between query vectors and key vectors in the multi-head attention processing for model inference, the operation tasks are divided into each slave core with a uniform data amount according to the total amount of operation tasks and the number of slave cores in the processor core group; when a operation task is jointly processed by at least two slave cores, the query vector and the key vector in the operation task are matrix-sliced, and each slave core reads the corresponding query sub-vector and key sub-vector according to the matrix slice result to perform matrix multiplication operation; a first target slave core is specified in each slave core, and the first matrix multiplication operator result calculated by each slave core is cached in the cache of the first target slave core to generate the first matrix multiplication operation result, thereby solving the reasoning acceleration problem of the self-attention mechanism in the large language model. By evenly dividing the operation tasks into each slave core according to the data amount, the task amount of each slave core can be completely balanced, giving full play to the advantages of multi-core parallel computing and improving computing efficiency; by caching the operation results in the cache of the slave core, the subsequent data reading rate can be improved and the hardware bandwidth can be fully utilized.

[0023] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1a This is a flowchart of a computational acceleration method for model reasoning provided according to the first embodiment of the present invention;

[0026] Figure 1b This is a schematic diagram of the structure of a multi-core processor provided according to the first embodiment of the present invention;

[0027] Figure 1c This is a schematic structural diagram of a processor core group provided according to a second embodiment of the present invention;

[0028] Figure 1d This is a schematic diagram of memory access performance at different levels of a processor provided according to the first embodiment of the present invention;

[0029] Figure 2 This is a flowchart of a computational acceleration method for model reasoning provided according to the second embodiment of the present invention;

[0030] Figure 3 2 is a schematic diagram of the structure of a computing acceleration device for model reasoning according to a third embodiment of the present invention;

[0031] Figure 4 It is a structural diagram of an electronic device for implementing the computational acceleration method for model reasoning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0033] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] Example 1

[0035] Figure 1a This is a flowchart of a computational acceleration method for model inference provided according to Example 1 of the present invention. This embodiment is applicable to the case of accelerating inference in the self-attention module of a large language model. The method can be executed by a computational acceleration device for model inference. The computational acceleration device for model inference can be implemented in the form of hardware and / or software. The computational acceleration device for model inference can be configured in an electronic device, which can be a mobile phone or a computer.

[0036] Before explaining the computational acceleration method for model inference provided by the embodiments of the present invention, we first introduce the inference decoding computation process of the self-attention module in the large language model. Specifically, when performing self-attention computation, the following steps are usually performed:

[0037] Step a: Read the input data and input weight matrix from the device main memory, perform matrix multiplication, and obtain the merged data of the query vector (Query, Q), the key vector (Key, K), and the value vector (Value, V).

[0038] The input data can be a high-dimensional vector of shape [batch_size, 1, hidden_dim], where batch_size represents the number of data in a batch and hidden_dim represents the hidden layer dimension. The input weight matrix can be a matrix of shape [hidden_dim, (q_head_num + 2 × kv_head_num) × head_dim]. q_head_num represents the number of heads in the multi-head attention query structure, kv_head_num represents the number of heads in the key or value structure, and head_dim represents the head dimension of the multi-head attention.

[0039] The combined data of Q, K, and V, obtained by the matrix multiplication of the input data and the input weight matrix, can be represented as a matrix of shape [batch_size, 1, (q_head_num + 2 × kv_head_num) × head_dim]. The shape of the query vector Q is [batch_size, 1, q_head_num, head_dim]. The shape of the key vector K is [batch_size, 1, kv_head_num, head_dim]. The shape of the value vector V is [batch_size, 1, kv_head_num, head_dim].

[0040] Step b: Separate the data of Q, K, and V from the merged data of Q, K, and V into three parts, namely the query vector, key vector, and value vector.

[0041] Step c: perform rotational position encoding calculation on the query vector Q and key vector K in step b, store the calculation results, and continue to use the query vector Q and key vector K for identification.

[0042] Step d: Output the key vector K obtained in step c and the value vector V obtained in step b to the K and V data cache areas of the device memory.

[0043] Step e: Read all key vectors K in the K and V data cache areas, transpose the key vector K, and the transposed data is KT data.

[0044] Among them, the data shape of the key vector K in step e is [batch_size, seq_len, kv_head_num, head_dim], and the K obtained after transposition is T The data shape is [batch_size, kv_head_num, head_dim, seq_len]. seq_len represents the sequence length of the K data accumulated in the current inference and the previous inference.

[0045] Step f: Use the query vector Q from step c and K from step e T The data is matrix multiplied and multiplied by the coefficient 1 / sqrt(head_dim). The result is the self-attention score, i.e., the atten-score data.

[0046] Among them, sqrt is the square root operation.

[0047] Step g: perform activation function calculation (such as softmax function calculation) on the atten-score data in step f, and the calculation result is the attention probability result, namely the atten-prob data.

[0048] Step h: Read the value vector V in the K and V data cache areas, perform matrix multiplication on the atten-prob data in step g and the value vector V, and calculate the result as the attention context, i.e., the atten-context data.

[0049] Among them, the data shape of the attention context is [batch_size,kv_head_num,head_dim].

[0050] Step i: Perform matrix multiplication on the atten-context data in step h and the output weight matrix to obtain the final result, i.e., the output data, and write the output data back to the device main memory.

[0051] Among them, the output data shape is [batch_size,1,hidden_dim].

[0052] The technical solution in the embodiment of the present invention can deeply integrate the operations in the above steps a to i to reduce the access time of the memory between operators and improve the reasoning efficiency. Figure 1a As shown, the computational acceleration method for model reasoning provided by the embodiment of the present invention includes:

[0053] Step 110: When performing multi-head attention processing for model inference, the number of heads in the multi-head attention processing is divided into each processor core group according to the number of processor core groups in the processor.

[0054] Figure 1b FIG. 1 is a schematic diagram of the structure of a multi-core processor provided according to the first embodiment of the present invention. Figure 1b As shown, there may be multiple processor core groups in one processor.

[0055] To fully utilize the parallel performance of multiple processor cores in the processor, the heads in the multi-head attention processing are divided into different processor core groups based on the independence of the multi-head computation in the multi-head attention mechanism. When dividing the heads in the multi-head attention processing, the balance of processing tasks can be considered.

[0056] For example, if the number of processor core groups in the processor is 4 and the number of heads is 64, each processor core group can process 8 heads; if the number of heads is 10, the first two processor core groups can process 3 heads and the last two processor core groups can process 2 heads.

[0057] By dividing the number of heads in multi-head attention processing, we can fully utilize the parallel performance of the processor core group and improve the inference efficiency.

[0058] Step 120: When the processor core group performs operations between the query vector and the key vector in the multi-head attention processing of the model inference, the operation tasks are divided into each slave core with a uniform amount of data according to the total amount of operation tasks and the number of slave cores in the processor core group.

[0059] Figure 1c FIG. 1 is a schematic diagram of the structure of a processor core group provided according to the second embodiment of the present invention. Figure 1c As shown, a processor core group may contain multiple high-performance slave cores. Each slave core may have an independent cache (such as a first-level cache L1-Cache). The cache capacity of the L1-Cache ranges from several hundred KB to 1 MB.

[0060] When optimizing step f in the multi-attention mechanism, the computation tasks can be divided evenly with respect to the amount of data, thereby fully utilizing the computing resources of all slave cores. The even amount of data can mean that when dividing the computation tasks, rather than simply assigning a complete computation task to a single slave core, multiple slave cores can share the processing of the computation task.

[0061] For example, if the total number of computing tasks is 37 and the number of slave cores in a processor core group is 32, then all 32 computing tasks can be fully distributed to each slave core for calculation. The remaining 5 computing tasks cannot be evenly distributed among the slave cores. In this case, the computing tasks can be divided up by having multiple slave cores jointly handle each computing task. For example, if 8 slave cores jointly handle one computing task, then 32 slave cores can handle a total of 4 computing tasks. Finally, there is still one computing task remaining, which can be jointly handled by all 32 slave cores. This achieves the effect of equalizing the number of tasks among the slave cores, fully utilizing the parallel performance of each slave core.

[0062] Optionally, based on the total amount of computing tasks and the number of slave cores in the processor core group, the computing tasks are divided into each slave core with a uniform data volume, including: dividing the total amount of computing tasks according to the formula BHN=2 n +2 n-1 +…+2 0 Split, and divide the computing tasks into each slave core with a uniform amount of data according to the split results.

[0063] Among them, 2 n is the number of slave cores in a processor core group, BHN is the total number of computing tasks, 2 n Representation 2 n Each computing task is processed by a slave core separately; 2 n-1 Representation 2 n-1 Each of the computational tasks is performed through 2 n / 2 n-1 2 slave cores jointly process; 0 This means that one computing task is processed by 32 slave cores.

[0064] In an embodiment of the present invention, considering the independence of the calculation of each head, each batch of data can be calculated separately. Moreover, in the multi-head attention mechanism, the number of heads of Q can be equal to the number of heads of K; or, the number of heads of Q is an integer multiple of the number of heads of K. When the number of heads of Q can be equal to the number of heads of K, the heads of the query vector Q and the key vector K can be operated separately. When the number of heads of Q is an integer multiple of the number of heads of K, the query vector Q can be divided into n groups in groups of m, where m is the relationship coefficient between the number of heads of Q and the number of heads of K, and n is the number of heads of the key vector structure. The query vector Q of the i-th group is operated on the i-th key vector K. Thus, based on the independence of the calculation of each head, BHN can be expressed as batch_size×kv_head_num.

[0065] For example, when BHN is 50, it can be split into 50=2 5 +2 4 +2 1, where 32 computing tasks can be processed independently by one slave core. Each of the 16 computing tasks can be processed jointly by two slave cores, and the 16 computing tasks occupy a total of 32 slave cores. Each of the 2 computing tasks can be processed jointly by 16 slave cores, and the 2 computing tasks occupy a total of 32 slave cores.

[0066] As a result, the amount of computing data in each slave core is the same, which can give full play to the parallel performance of each slave core and improve the inference performance; avoid the imbalance of slave cores and the inference delay caused by some slave cores being busy while others are idle.

[0067] According to the computing task allocation method provided by the embodiment of the present invention, any BHN can be processed evenly in all slave cores, maximizing the utilization of the slave cores.

[0068] Step 130: When a computing task is jointly processed by at least two slave cores, the query vector and the key vector in the computing task are matrix-sliced. Each slave core reads the corresponding query sub-vector and key sub-vector according to the matrix-slicing result and performs a matrix multiplication operation.

[0069] When performing matrix partitioning on the query vector and key vector, the number of matrix partitioning blocks can be determined based on the number of slave cores used for joint processing. Based on the number of partitioning blocks, the query vector can be partitioned into query sub-vectors, and the key vector can be partitioned into key sub-vectors. A matrix multiplication operation is performed on the query sub-vectors and the key sub-vectors on each slave core to obtain a first matrix multiplication operator result.

[0070] When combined with other implementations in the embodiments of the present invention, the query subvector and key subvector can be read from the cache of the second target slave core without accessing the device main memory, which can improve data reading efficiency and thus improve inference efficiency. When not combined with other implementations in the embodiments of the present invention, the query subvector and key subvector can be read from the device main memory.

[0071] Matrix multiplication can typically be expressed as C = A × B, where A is of shape [M, K] and B is of shape [K, N]. Both matrices A and B can be partitioned according to the number of slave cores. Each slave core loads only a portion of matrix A (sub_A) and a portion of matrix B (sub_B) into the cache for computation, resulting in a portion of matrix C (pre_C). The next portion of sub_B is then loaded in a loop, and the next portion of sub_A is broadcast from the cache of other cores. The matrix multiplication is then performed on the slave cores, resulting in the next portion of pre_C. This loop continues, and each portion of pre_C is added element-by-element to produce the final matrix multiplication result.

[0072] Step 140 : Designate a first target slave core in each slave core, and cache the first matrix multiplication operator results calculated by each slave core into the cache of the first target slave core to generate a first matrix multiplication operation result.

[0073] The first target slave core can add the obtained first matrix multiplication operator results element-wise to obtain the first matrix multiplication operation result. By optimizing the operation between Q and K in step f, the parallel performance between the slave cores can be fully utilized to improve the inference speed.

[0074] Figure 1d 1 is a schematic diagram of memory access performance at different levels of a processor according to the first embodiment of the present invention. Figure 1d As shown in the figure, the memory access performance of the cache L1-cache is due to the performance of the device main memory. In the embodiment of the present invention, when reading data, Q and K are obtained from the cache, and data exchange between the slave cores is also based on the cache, avoiding access to the device main memory, thereby improving data access efficiency and further improving model inference efficiency.

[0075] In the embodiment of the present invention, when multiple slave cores jointly process a computing task, they can be executed according to the above-mentioned matrix segmentation operation method, thereby realizing parallel operation of multiple slave cores and improving inference efficiency. In addition, for the matrix multiplication operation between the input data and the input weight matrix in the above-mentioned step a, and the matrix multiplication operation between the atten-context data and the output weight matrix in step i, the above-mentioned matrix segmentation method can be used to realize parallel operation in multiple cores to further improve inference efficiency.

[0076] In specific applications, each slave core can process one kv_head or part of a kv_head in a batch. The size of the K and V data in each kv_head is related to seq_len. When seq_len is large, it may exceed the cache capacity of the slave core. Therefore, in addition to splitting the computation tasks and matrices during Q and K calculations in model inference, seq_len can also be split during the computation.

[0077] Optionally, when dividing the computing task into each slave core with a uniform amount of data, it also includes: when the sequence length scale in the computing task is greater than a preset scale parameter, the sequence length scale is split; each slave core reads the query vector and key vector in the computing task according to the sequence length scale split result to perform subsequence operations, and the subsequence operation results are cached in situ in the slave core's cache to form a complete sequence operation result.

[0078] The preset scale parameter may be 16, 32, or 64. For example, the data amount of [16, head_size] is loaded each time, and the loading is cyclic, and the operation results of each subsequence are accumulated element by element to obtain the operation result of the complete sequence.

[0079] The above-mentioned task segmentation and sequence segmentation can be used in combination. For example, for a data vector with a shape of [batch_size, kv_head_num, max_seq_len, head_dim], after combining the task segmentation and sequence segmentation, it can be adjusted to [batch_size, kv_head_num, 2 x , max_seq_len / 32 / 2 x ,32,head_dim]. Among them, here 2 x Refers to splitting the sequence into 2 x 32 means that the computing tasks are evenly distributed to 32 slave cores. Each slave core can load [max_seq_len / 32 / 2 x ,head_dim] data, load 2 in one batch x 32 cores complete batch_size batches of data loading and computation. This fully utilizes the parallel capabilities of each slave core and processes data at full access bandwidth, improving inference speed.

[0080] In this embodiment of the present invention, this sequence length segmentation method can be used when performing the Q and K operations, as well as when performing the operation on the atten-prob data and the V data in step h. The results of the Q and K operations can reside in the cache. The softmax calculation can be performed based on the results in the cache, and the results can be cached in situ in the cache. The atten-prob data and the V data can continue to be calculated based on the softmax results in the cache, avoiding the time consumption caused by accessing main memory.

[0081] The technical solution of the embodiment of the present invention is to divide the number of heads in the multi-head attention processing into each processor core group according to the number of processor core groups in the processor when performing multi-head attention processing for model inference; when the processor core group performs operations between query vectors and key vectors in the multi-head attention processing for model inference, the operation tasks are divided into each slave core with a uniform data amount according to the total amount of operation tasks and the number of slave cores in the processor core group; when a operation task is jointly processed by at least two slave cores, the query vector and the key vector in the operation task are matrix-sliced, and each slave core reads the corresponding query sub-vector and key sub-vector according to the matrix slice result to perform matrix multiplication operation; a first target slave core is specified in each slave core, and the first matrix multiplication operator result calculated by each slave core is cached in the cache of the first target slave core to generate the first matrix multiplication operation result, thereby solving the reasoning acceleration problem of the self-attention mechanism in the large language model. By evenly dividing the operation tasks into each slave core according to the data amount, the task amount of each slave core can be completely balanced, giving full play to the advantages of multi-core parallel computing and improving computing efficiency; by caching the operation results in the cache of the slave core, the subsequent data reading rate can be improved and the hardware bandwidth can be fully utilized.

[0082] Example 2

[0083] Figure 2 This is a flow chart of a computational acceleration method for model reasoning provided according to the second embodiment of the present invention. This embodiment is a further refinement of the above technical solution. The technical solution in this embodiment can be combined with various optional solutions in one or more of the above embodiments. Figure 2 As shown, the method includes:

[0084] Step 210: When performing multi-head attention processing for model inference, the number of heads in the multi-head attention processing is divided into each processor core group according to the number of processor core groups in the processor.

[0085] Step 220: When performing matrix multiplication of input data and input weight matrix in multi-head attention processing of model inference in the processor core group, the input data and input weight matrix are divided separately according to the number of slave cores in the processor core group, and the divided input sub-data and input weight sub-matrix are distributed to each slave core.

[0086] Step 220 and step 230 may be an optimization of step a. The input data and the input weight matrix may be divided evenly according to the number of cores.

[0087] Step 230: When the slave core performs a matrix multiplication operation on the input data and the input weight matrix, the corresponding input sub-data and the input weight sub-matrix are read from the main memory, and a matrix multiplication operation is performed to obtain a second matrix multiplication operation result.

[0088] The input data and input weight matrix can be read from the device main memory. When reading the input data and input weight matrix, each core can read and perform operations on the corresponding input sub-data and input weight sub-matrix based on the matrix partitioning. The parallel execution of the slave cores can improve the parallel utilization of multiple cores, thereby improving the inference rate. By partitioning the input data and input weight matrix, the data reading efficiency when accessing the device main memory can be improved, thereby increasing the inference speed. The partitioning calculation of the input data and input weight matrix is ​​the same as the partitioning calculation during the matrix multiplication operation mentioned above, and will not be repeated here.

[0089] Step 240: Store the result of the second matrix multiplication operation in the cache of the slave core.

[0090] By storing the result of the second matrix multiplication operation in the cache rather than in the device main memory, the time for writing data back to the main memory can be reduced, thereby improving inference efficiency.

[0091] Optionally, the result of the second matrix multiplier operation is stored in the cache of the slave core, including: forming a query vector from each query sub-vector in the result of the second matrix multiplier operation calculated by each slave core; forming a key vector from each key sub-vector in the result of the second matrix multiplier operation calculated by each slave core; specifying a second target slave core in each slave core, caching the query vector and the key vector in the cache of the second target slave core; and outputting the value sub-vector in the result of the second matrix multiplier operation calculated by each slave core to the key-value cache of the main memory.

[0092] When performing rotational position encoding in step c, complete Q data and K data are required. In this embodiment of the present invention, the query subvectors obtained by multiple slave kernel calculations can form a query vector; the key subvectors can form a key vector. When forming the key vector, the specific data synthesis can be performed according to the segmentation operation results of the matrix multiplication operation.

[0093] The query and key vectors can be cached in the cache of the second target slave core. The positional encoding operation, or position encoding, is then performed based on the data in the cache. Storing and retrieving data in the cache improves data access efficiency, which in turn improves model inference efficiency.

[0094] The calculation of the V data in step h may not be performed in a complete form, so each slave core may output the calculated value sub-vector to a key-value cache in the main memory, that is, a KV cache.

[0095] The query subvector is part of the query vector, the key subvector is part of the key vector, and the value subvector is part of the value vector.

[0096] By storing query vectors, key vectors, and value vectors in different forms, the reasoning rate can be improved while ensuring computational rationality.

[0097] Step 250: When the processor core group performs operations between the query vector and the key vector in the multi-head attention processing of the model inference, the operation tasks are divided into each slave core with a uniform amount of data according to the total amount of operation tasks and the number of slave cores in the processor core group.

[0098] Optionally, based on the total amount of computing tasks and the number of slave cores in the processor core group, the computing tasks are divided into each slave core with a uniform data volume, including: dividing the total amount of computing tasks according to the formula BHN=2 n +2 n-1 +…+2 0 Split, and divide the computing tasks into each slave core with a uniform amount of data according to the split result; n is the number of slave cores in a processor core group, BHN is the total number of computing tasks, 2 n Representation 2 n Each computing task is processed by a slave core separately; 2 n-1 Representation 2 n-1 Each of the computational tasks is performed through 2 n / 2 n-1 2 slave cores jointly process; 0 This means that one computing task is processed by 32 slave cores.

[0099] Step 260: When a computing task is jointly processed by at least two slave cores, the query vector and the key vector in the computing task are matrix-sliced. Each slave core reads the corresponding query sub-vector and key sub-vector according to the matrix-slicing result and performs a matrix multiplication operation.

[0100] Step 270 : Designate a first target slave core in each slave core, and cache the first matrix multiplication operator results calculated by each slave core into the cache of the first target slave core to generate a first matrix multiplication operation result.

[0101] Optionally, when dividing the computing task into each slave core with a uniform amount of data, it also includes: when the sequence length scale in the computing task is greater than a preset scale parameter, the sequence length scale is split; each slave core reads the query vector and key vector in the computing task according to the sequence length scale split result to perform subsequence operations, and the subsequence operation results are cached in situ in the slave core's cache to form a complete sequence operation result.

[0102] Step 280: When calculating the activation function in the multi-head attention processing of model inference, the calculation is performed based on the first matrix multiplication result in the cache of the slave core according to the first target, and the calculated attention probability result is cached in situ.

[0103] The result of the first matrix multiplication, the attention score, can be calculated using an activation function such as the softmax function. In-situ caching allows the output result to reuse the storage space of the input data. By reading the result of the first matrix multiplication in the cache and storing the calculated attention probability in-situ, access to the device's main memory is eliminated, thereby increasing data read speed and improving model inference performance.

[0104] Step 290: When performing operations on the attention probability result and the value vector in the multi-head attention processing of model inference, read the value sub-vector in the key-value buffer of the main memory.

[0105] Step 2100: Perform calculations based on each value sub-vector and the attention probability results in the cache of the first target slave core, and store the attention context obtained by the calculation in situ.

[0106] The calculations on the atten-prob data and the V data can be performed using the aforementioned sequence segmentation method, which will not be described in detail here. Reading and writing data in the cache can improve the model inference speed.

[0107] Efficient KV cache storage using accelerator hardware can optimize data loading and storage processes, fully utilize the accelerator's bandwidth, and reduce storage bottlenecks.

[0108] After obtaining the attention context, we can perform a matrix multiplication operation on the attention context and the output weight matrix to obtain the output results of the multiple attention heads. In this matrix multiplication operation between the attention context and the output weight matrix, we can perform the aforementioned matrix slicing operation to accelerate the inference process.

[0109] The technical solution of this embodiment of the present invention integrates all operations originally performed in steps a through i, eliminating the intermediate computational steps and the time overhead of reading and writing data from the device's main memory. After optimization, only step a requires reading data from the device's main memory, step i writes data back to the device's main memory, and steps e, f, and h read data. This significantly reduces the number of accesses to the device's main memory and improves overall inference performance.

[0110] Based on the above implementation, optionally, the method further includes: in each single-core calculation, adopting a double buffering strategy to perform data operations while reading data.

[0111] The dual-buffer strategy can mean that each slave core can be equipped with two caches. When performing calculations based on data in the first cache, the second cache can be used to read data. When performing calculations based on data in the second cache, the first cache can be used to read data. The cooperation between the first and second caches can further reduce data read time and improve inference efficiency.

[0112] The technical solution of the embodiment of the present invention can fully utilize the parallel capabilities of multiple cores by performing matrix multiplication operations under matrix segmentation between input data and input weight matrix; by storing Q, K and V in different forms respectively, it can reduce the access of data to main memory while satisfying the rationality of data operation; by performing uniform data amount segmentation operations and sequence segmentation operations on the operation tasks in Q and K operations under multiple cores, it can fully utilize the parallel computing power and bandwidth of multiple cores; by accessing data through cache during softmax calculation and operation of attention probability results and value vectors, it can make data no longer dependent on device main memory, improve data access performance, and fully utilize the access bandwidth of accelerator through KV cache. When performing matrix multiplication operations between output data and output weight matrix, data can also be matrix segmented, and the multiplication operation under matrix segmentation can further improve the efficiency of model inference.

[0113] The technical solution of the embodiments of the present invention significantly improves the inference calculation speed of the self-attention mechanism in the decoding stage and reduces inference latency. It can be applied to large-scale model inference scenarios and model training processes. The multi-core processor can be an execution medium with parallel computing core processing capabilities, such as an AI processor or a neural network processor (NPU).

[0114] Example 3

[0115] Figure 3 Schematic diagram of a computing acceleration device for model reasoning according to the third embodiment of the present invention. Figure 3 As shown, the device includes: an attention head division module 310, an operation task division module 320, a matrix segmentation module 330 and a first matrix multiplication operation result generation module 340.

[0116] An attention head partitioning module 310 is used to partition the number of heads in the multi-head attention processing into each processor core group according to the number of processor core groups in the processor when performing multi-head attention processing for model inference;

[0117] A computation task partitioning module 320 is configured to divide the computation tasks among the slave cores with a uniform data size according to the total number of computation tasks and the number of slave cores in the processor core group when performing computations between query vectors and key vectors in multi-head attention processing for model inference in the processor core group;

[0118] A matrix splitting module 330 is configured to perform matrix splitting on the query vector and key vector in a computing task when a computing task is jointly processed by at least two slave cores, and each slave core reads the corresponding query sub-vector and key sub-vector according to the matrix splitting result to perform matrix multiplication operation;

[0119] The first matrix multiplication operation result generating module 340 is used to specify a first target slave core in each slave core, cache the first matrix multiplication operator results calculated by each slave core into the cache of the first target slave core to generate a first matrix multiplication operation result.

[0120] Optionally, the computing task division module 320 includes:

[0121] The computing task is divided into sub-units, which are used to divide the total amount of computing tasks into the formula BHN=2 n +2 n-1 +…+2 0 Split, and divide the computing tasks into each slave core with a uniform data volume according to the split results;

[0122] Among them, 2 n is the number of slave cores in a processor core group, BHN is the total number of computing tasks, 2 n Representation 2 n Each computing task is processed by a slave core separately; 2 n-1 Representation 2 n-1 Each of the computational tasks is performed through 2 n / 2 n-1 2 slave cores jointly process; 0 This means that one computing task is processed by 32 slave cores.

[0123] Optionally, the device further includes:

[0124] A sequence length splitting module is used to split the sequence length scale when the sequence length scale in the computing task is greater than a preset scale parameter when the computing task is divided into each slave core with a uniform data amount;

[0125] The complete sequence operation result generation module is used for each slave core to split the result according to the sequence length scale to read the query vector and key vector in the operation task to perform subsequence operations. The operation results of the subsequence are cached in situ in the slave core's cache to form the complete sequence operation result.

[0126] Optionally, the device further includes:

[0127] An input splitting and distribution module is used to, after dividing the number of heads in the multi-head attention processing into each processor core group according to the number of processor core groups in the processor, split the input data and the input weight matrix according to the number of slave cores in the processor core group when performing matrix multiplication operation of input data and input weight matrix in the multi-head attention processing for model inference in the processor core group, and distribute the split input sub-data and input weight sub-matrix to each slave core;

[0128] a second matrix multiplier operation result determination module, configured to read the corresponding input sub-data and input weight sub-matrix from the main memory when the slave core performs a matrix multiplication operation on the input data and the input weight matrix, perform a matrix multiplication operation, and obtain a second matrix multiplier operation result;

[0129] The second matrix multiplier operation result storage module is used to store the second matrix multiplier operation result in the cache of the slave core.

[0130] Optionally, the second matrix multiplication operation result storage module includes:

[0131] a query vector generating unit, configured to combine query sub-vectors in the second matrix multiplication operation results obtained from the kernels into a query vector;

[0132] a key vector generating unit, configured to form a key vector from each key sub-vector in the second matrix multiplication operation result obtained by calculating the slave core;

[0133] A query vector and key vector caching unit, configured to specify a second target slave core in each slave core and cache the query vector and key vector in a cache of the second target slave core;

[0134] The value sub-vector storage unit is used to output the value sub-vector in the second matrix multiplication operation result calculated by each slave core to the key-value buffer of the main memory.

[0135] Optionally, the device further includes:

[0136] An attention probability result determination module is configured to, after caching the first matrix multiplication operator results calculated by each slave core into the cache of the first target slave core to generate the first matrix multiplication operation result, perform calculations based on the first matrix multiplication operation results in the cache of the first target slave core during activation function calculation in the multi-head attention processing of model inference, and cache the calculated attention probability results in situ;

[0137] The value sub-vector reading module is used to read the value sub-vector in the key-value buffer of the main memory when performing the operation of the attention probability result and the value vector in the multi-head attention processing of the model inference;

[0138] The attention context determination module is used to perform operations based on each value sub-vector and the attention probability results in the cache of the first target slave core, and store the attention context obtained by the operation in situ.

[0139] Optionally, the device further includes:

[0140] The double buffer strategy application module is used to adopt the double buffer strategy in each single-core calculation to perform data calculation while reading data.

[0141] The computing acceleration device for model reasoning provided by an embodiment of the present invention can execute the computing acceleration method for model reasoning provided by any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.

[0142] Example 4

[0143] Figure 4 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0144] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0145] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0146] The processor 11 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the computational acceleration method for model inference.

[0147] In some embodiments, the computational acceleration method for model reasoning may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the computational acceleration method for model reasoning described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the computational acceleration method for model reasoning in any other appropriate manner (e.g., by means of firmware).

[0148] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0149] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0150] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0151] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0152] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0153] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0154] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0155] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A computational acceleration method for model reasoning, characterized in that: include: When performing multi-head attention processing for model inference, dividing the number of heads in the multi-head attention processing into each processor core group according to the number of processor core groups in the processor; When the processor core group performs operations between query vectors and key vectors in the multi-head attention processing of model inference, the operations are evenly divided among the slave cores according to the total number of operations and the number of slave cores in the processor core group. When a computing task is jointly processed by at least two slave cores, the query vector and the key vector in the computing task are matrix-sliced, and each slave core reads the corresponding query sub-vector and the key sub-vector according to the matrix-slicing result to perform matrix multiplication operation; A first target slave core is designated in each slave core, and first matrix multiplication operator results calculated by each slave core are cached in a cache of the first target slave core to generate a first matrix multiplication operation result.

2. The method according to claim 1, characterized in that Based on the total number of computing tasks and the number of slave cores in the processor core group, the computing tasks are evenly distributed to each slave core with a uniform data volume, including: The total amount of computing tasks is calculated according to the formula BHN=2 n +2 n-1 +…+2 0 Split, and divide the computing tasks into each slave core with a uniform data volume according to the split results; Among them, 2 n is the number of slave cores in a processor core group, BHN is the total number of computing tasks, 2 n Representation 2 n Each computing task is processed by a slave core separately; 2 n-1 Representation 2 n-1 Each of the computation tasks is performed by 2n / 2 n-1 2 slave cores jointly process; 0 This means that one computing task is processed by 32 slave cores.

3. The method according to claim 1, characterized in that When computing tasks are divided evenly among the slave cores, the following also applies: When the sequence length scale in the operation task is greater than the preset scale parameter, the sequence length scale is split; Each slave core reads the query vector and key vector in the operation task according to the sequence length and performs subsequence operations. The operation results of the subsequence are cached in situ in the slave core's cache to form a complete sequence operation result.

4. The method according to claim 1, wherein After dividing the number of heads in the multi-head attention processing into each processor core group according to the number of processor core groups in the processor, the method further includes: When performing matrix multiplication of input data and input weight matrix in the multi-head attention processing of model inference in the processor core group, the input data and input weight matrix are split according to the number of slave cores in the processor core group, and the split input sub-data and input weight sub-matrix are distributed to each slave core; When performing a matrix multiplication operation on the input data and the input weight matrix from the core, the corresponding input sub-data and the input weight sub-matrix are read from the main memory, and the matrix multiplication operation is performed to obtain a second matrix multiplication operation result; The result of the second matrix multiplication operation is stored in the cache of the slave core.

5. The method according to claim 4, characterized in that The result of the second matrix multiplication operation is stored in the cache of the slave core, including: Combining query subvectors in the second matrix multiplication results obtained from the kernel into a query vector; Combining the key subvectors in the second matrix multiplication operation results obtained from the kernel into a key vector; Specifying a second target slave core in each slave core, and caching the query vector and the key vector in a cache of the second target slave core; Output the value sub-vectors in the second matrix multiplication operation results calculated by each slave core to the key-value buffer of the main memory.

6. The method according to claim 5, characterized in that After caching the first matrix multiplication operator results calculated by each slave core into the cache of the first target slave core to generate the first matrix multiplication operation result, the method further includes: When calculating the activation function in the multi-head attention processing of model inference, the calculation is performed based on the first matrix multiplication result in the cache of the first target slave core, and the calculated attention probability result is cached in situ; When performing the operation on the attention probability result and the value vector in the multi-head attention processing of model inference, the value sub-vector in the key-value buffer of the main memory is read; The attention context is calculated based on the attention probability results of each value sub-vector and the first target in the cache of the slave core, and the attention context obtained by the calculation is stored in situ.

7. The method according to claim 1, characterized in that Also includes: In each single-core calculation, a double buffering strategy is adopted to perform data operations while reading data.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the computational acceleration method for model reasoning according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the computational acceleration method for model reasoning according to any one of claims 1 to 7 when executed.

10. A computer program product, comprising a computer program, which, when executed by a processor, implements the computational acceleration method for model reasoning according to any one of claims 1 to 7.

Images