Operator execution method and device, storage medium and program product

Abstract

The invention provides an operator execution method and device, a storage medium and a program product, and relates to the technical field of artificial intelligence, in the method, when a second thread bundle group adopts a vector operator to carry out vector calculation on an nth batch of tasks, a first thread bundle group adopts a first matrix operator to carry out matrix calculation on an (n + 1) th batch of tasks, and a second thread bundle group adopts a second matrix operator to carry out matrix calculation on the (n + 1) th batch of tasks; meanwhile, the third thread beam group adopts a second matrix operator to perform matrix calculation on the (n-1) th batch of tasks, and the first thread beam group and the third thread beam group are both composed of tensor calculation units, so that the tensor calculation units can process two batches of tasks at the same time. The tensor calculation unit and the vector calculation unit can process different batches of tasks at the same time, so that the mutual waiting time is avoided, and the time overhead caused by processing multiple batches of tasks by the Attention operator can be reduced.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of artificial intelligence technology, and in particular, to an operator execution method, device, storage medium, and program product. Background Art

[0002] With the rapid development of artificial intelligence technology, large language models (LLMs) are widely used in natural language processing, computer vision, speech recognition and other fields. At present, large language models generally use the Transformer structure as the core infrastructure, and the core computing part of the Transformer structure is the Attention operator.

[0003] When a large language model performs reasoning on multiple batches of tasks, the Attention operator can perform matrix and vector calculations for each batch of tasks in turn. However, when there are too many batches of tasks to be reasoned, this will bring huge time overhead. How to reduce the time overhead brought by reasoning on a large number of batches of tasks is a technical problem that needs to be solved urgently. Summary of the invention

[0004] The embodiments of the present application provide an operator execution method, device, storage medium, and program product for reducing the time overhead caused by reasoning a large number of batch tasks.

[0005] On the one hand, an embodiment of the present application provides an operator execution method, which is applied to an artificial intelligence chip including S tensor computing cores and W vector computing cores, wherein the artificial intelligence chip is used to process N batch tasks using an attention mechanism operator; the attention mechanism operator at least includes a first matrix operator, a vector operator, and a second matrix operator that sequentially execute any batch of tasks in the N batch tasks; S and W are both positive integers, and N is an integer greater than 2; When the second thread warp group uses the vector operator to perform vector calculation on the nth batch task among the N batch tasks, the first thread warp group uses the first matrix operator to perform matrix calculation on the n+1th batch task among the N batch tasks, and the third thread warp group uses the second matrix operator to perform matrix calculation on the n-1th batch task among the N batch tasks; the second thread warp group includes the W vector computing cores, the first thread warp group includes S1 tensor computing cores among the S tensor computing cores, the third thread warp group includes S2 tensor computing cores among the S tensor computing cores, and there is no overlap between the S1 tensor computing cores and the S2 tensor computing cores; the value range of n is [2,N], S1 and S2 are both less than S, and the sum of S1 and S2 is less than or equal to S.

[0006] Optionally, the second warp group uses the vector operator to perform vector calculation on the nth batch of tasks in the N batches of tasks, including: When the second thread warp group receives the first instruction corresponding to the nth batch of tasks sent by the first thread warp group and the second thread warp group completes the vector calculation of the n-1th batch of tasks, the second thread warp group uses the vector operator to perform vector calculation on the nth batch of tasks; the first instruction corresponding to the nth batch of tasks is used to indicate that the first thread warp group has completed the matrix calculation of the nth batch of tasks using the first matrix operator.

[0007] Optionally, the second warp group uses the vector operator to perform vector calculation on the nth batch of tasks, including: The second warp group obtains a first intermediate result corresponding to the nth batch of tasks; the first intermediate result corresponding to the nth batch of tasks is a result of the first warp group performing matrix calculation on the nth batch of tasks using the first matrix operator; The second warp group uses the vector operator to perform vector calculation on the first intermediate result corresponding to the nth batch of tasks to obtain a second intermediate result corresponding to the nth batch of tasks.

[0008] Optionally, the first warp group uses the first matrix operator to perform matrix calculation on the (n+1)th batch of tasks in the N batches of tasks, including: After the first warp group completes matrix calculation on the nth batch of tasks using the first matrix operator, the first warp group performs matrix calculation on the (n+1)th batch of tasks using the first matrix operator.

[0009] Optionally, the third warp group uses the second matrix operator to perform matrix calculation on the n-1th batch of tasks in the N batches of tasks, including: When the third thread warp group receives the second instruction corresponding to the n-1th batch of tasks sent by the second thread warp group and the third thread warp group completes matrix calculation on the n-2th batch of tasks using the second matrix operator, the third thread warp group performs matrix calculation on the n-1th batch of tasks using the second matrix operator; the second instruction corresponding to the n-1th batch of tasks is used to indicate that the second thread warp group completes vector calculation on the n-1th batch of tasks using the vector operator.

[0010] Optionally, the third warp group uses the second matrix operator to perform matrix calculation on the n-1th batch of tasks, including: The third warp group obtains the second intermediate result corresponding to the n-1th batch of tasks; the second intermediate result corresponding to the n-1th batch of tasks is a result of the second warp group performing vector calculation on the n-1th batch of tasks using the vector operator; The third thread warp group uses the second matrix operator to perform matrix calculation on the second intermediate result corresponding to the n-1th batch of tasks to obtain the result of the n-1th batch of tasks.

[0011] Optionally, before the second warp group uses the vector operator to perform vector calculation on the nth batch of tasks in the N batches of tasks, the method further includes: The first warp group completes matrix calculation of the nth batch of tasks by using the first matrix operator, and the first warp group sends a first instruction corresponding to the nth batch of tasks to the second warp group.

[0012] On the one hand, an embodiment of the present application provides an operator execution device, including: A matrix calculation module, used for performing matrix calculation on the n+1th batch task among the N batch tasks by using the first thread bundle group using the first matrix operator when the second thread bundle group uses the vector operator to perform vector calculation on the nth batch task among the N batch tasks, and performing matrix calculation on the n-1th batch task among the N batch tasks by using the second matrix operator through the third thread bundle group; the second thread bundle group includes W vector calculation cores, the first thread bundle group includes S1 tensor calculation cores among the S tensor calculation cores, and the third thread bundle group includes S2 tensor calculation cores among the S tensor calculation cores; the value range of n is [2, N], N is an integer greater than 2, S1 and S2 are both less than S, and the sum of S1 and S2 is less than or equal to S.

[0013] Optionally, the vector calculation module is used to perform vector calculation on the nth batch of tasks by the second thread bundle group using the vector operator when the second thread bundle group receives the first instruction corresponding to the nth batch of tasks sent by the first thread bundle group and the second thread bundle group completes the vector calculation on the n-1th batch of tasks; the first instruction corresponding to the nth batch of tasks is used to indicate that the first thread bundle group has completed the matrix calculation on the nth batch of tasks using the first matrix operator.

[0014] Optionally, the vector calculation module is specifically used to: acquiring, through the second warp group, a first intermediate result corresponding to the nth batch of tasks; the first intermediate result corresponding to the nth batch of tasks is a result of matrix calculation performed by the first warp group on the nth batch of tasks using the first matrix operator; The second thread warp group uses the vector operator to perform vector calculation on the first intermediate result corresponding to the nth batch of tasks to obtain a second intermediate result corresponding to the nth batch of tasks.

[0015] Optionally, the matrix calculation module is specifically used to: When the first warp group finishes the matrix calculation on the nth batch of tasks by using the first matrix operator, the first warp group performs the matrix calculation on the (n+1)th batch of tasks by using the first matrix operator.

[0016] Optionally, the matrix calculation module is specifically used to: When the third thread warp group receives the second instruction corresponding to the n-1th batch of tasks sent by the second thread warp group and the third thread warp group completes matrix calculation on the n-2th batch of tasks using the second matrix operator, the third thread warp group performs matrix calculation on the n-1th batch of tasks using the second matrix operator; the second instruction corresponding to the n-1th batch of tasks is used to indicate that the second thread warp group completes vector calculation on the n-1th batch of tasks using the vector operator.

[0017] Optionally, the matrix calculation module is specifically used to: Acquire the second intermediate result corresponding to the n-1th batch of tasks through the third warp group; the second intermediate result corresponding to the n-1th batch of tasks is a result of the second warp group performing vector calculation on the n-1th batch of tasks using a vector operator; The results of the n-1th batch of tasks are obtained by performing matrix calculation on the second intermediate results corresponding to the n-1th batch of tasks using the second matrix operator through the third thread warp group.

[0018] Optionally, the matrix calculation module is also used to: before the second thread warp group uses the vector operator to perform vector calculation on the nth batch of tasks among the N batches of tasks, complete the matrix calculation of the nth batch of tasks by the first thread warp group using the first matrix operator, and send the first instruction corresponding to the nth batch of tasks to the second thread warp group through the first thread warp group.

[0019] On the one hand, an embodiment of the present application provides a computer device, including a memory, an artificial intelligence chip, and a computer program stored in the memory and executable on the artificial intelligence chip, wherein the artificial intelligence chip implements the steps of the above-mentioned operator execution method when executing the computer program.

[0020] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program executable by a computer device. When the computer program runs on the computer device, the computer device executes the steps of the above-mentioned operator execution method.

[0021] On the one hand, an embodiment of the present application provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer device, the computer device executes the steps of the above-mentioned operator execution method.

[0022] In an embodiment of the present application, during the process of the Attention operator processing N batches of tasks, when the second thread bundle group uses the vector operator to perform vector calculation on the nth batch of tasks, the first thread bundle group uses the first matrix operator to perform matrix calculation on the n+1th batch of tasks, and at the same time, the third thread bundle group uses the second matrix operator to perform matrix calculation on the n-1th batch of tasks. Since the first thread bundle group and the third thread bundle group are both composed of tensor calculation units, the tensor calculation unit can process two batches of tasks at the same time. In the above process, the tensor calculation unit and the vector calculation unit can process different batches of tasks at the same time, avoiding waiting time for each other, and the tensor calculation unit and the vector calculation unit are fully utilized, thereby reducing the time overhead caused by the Attention operator processing multiple batches of tasks. In addition, since each thread bundle group only performs one calculation, such as the first thread bundle group performs matrix calculation, the second thread bundle group performs vector calculation, and the third thread bundle group performs matrix calculation, the complexity of managing thread bundle groups can be effectively reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0024] Figure 1A A schematic diagram of the structure of an artificial intelligence chip provided in an embodiment of the present application; Figure 1B A schematic diagram of the structure of a computing unit provided in an embodiment of the present application; Figure 2 A schematic diagram of the structure of an Attention operator provided in an embodiment of the present application; Figure 3 A schematic diagram of the structure of the calculation process of an Attention operator provided in an embodiment of the present application; Figure 4 A schematic diagram of the structure of the calculation process of an Attention operator provided in an embodiment of the present application; Figure 5 A schematic diagram of the allocation of tensor computing cores and vector computing cores provided in an embodiment of the present application; Figure 6 A schematic diagram of the allocation of tensor computing cores and vector computing cores provided in an embodiment of the present application; Figure 7 A flowchart of an operator execution method provided in an embodiment of the present application; Figure 8 A schematic diagram of the structure of the calculation process of an Attention operator provided in an embodiment of the present application; Fig. 9 A schematic diagram of the structure of the calculation process of an Attention operator provided in an embodiment of the present application; Fig.10 A schematic diagram of the structure of an operator execution device provided in an embodiment of the present application; Fig.11 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and beneficial effects of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0026] Below, some terms in the embodiments of the present application are explained to facilitate understanding by those skilled in the art.

[0027] A warp group is composed of a group of warps. A warp is the scheduling unit when an artificial intelligence chip executes a program. Threads in a warp can execute the same instructions with different data resources.

[0028] refer to Figure 1A , which is a structural diagram of an artificial intelligence chip applicable to an embodiment of the present application. The artificial intelligence chip 100 at least includes: multiple computing units (Compute Units, referred to as CUs) 101, on-chip cache 105 and video memory 106, wherein each computing unit 101 includes: a tensor computing unit 102, a vector computing unit 103, and a register 104, wherein the tensor computing unit 102 and the vector computing unit 103 are the core of the computing unit 101.

[0029] The register 104 is a register shared by the tensor computing unit 102 and the vector computing unit 103, and the tensor computing unit 102 and the vector computing unit 103 can communicate through the register 104. Compared with the on-chip cache 105, the capacity of the register 104 is smaller than that of the on-chip cache 105, but the data exchange speed is faster than that of the on-chip cache 105.

[0030] The on-chip cache 105 is a temporary memory having a smaller capacity than the video memory 106 but a faster data exchange speed than the video memory 106 .

[0031] The video memory 106 may be a high bandwidth memory (HBM for short) or other types of memory.

[0032] If the capacity of the register 104 can satisfy the data transmission between the tensor calculation unit 102 and the vector calculation unit 103 , the data between the tensor calculation unit 102 and the vector calculation unit 103 is transmitted through the register 104 .

[0033] If the capacity of the register 104 cannot satisfy the data transmission between the tensor calculation unit 102 and the vector calculation unit 103 , the data between the tensor calculation unit 102 and the vector calculation unit 103 is transmitted through the on-chip cache 105 .

[0034] If the capacity of the on-chip cache 105 cannot satisfy the data transmission between the tensor calculation unit 102 and the vector calculation unit 103 , the data between the tensor calculation unit 102 and the vector calculation unit 103 is transmitted through the video memory 106 .

[0035] In addition to the above-mentioned structure, the artificial intelligence chip 100 in the present application may also include other structures, which are not specifically limited in the present application.

[0036] The artificial intelligence chip 100 may be: a central processing unit (CPU), a graphics processing unit (GPU), a general-purpose graphics processing unit (GPGPU), a domain-specific architecture (DSA), etc.

[0037] against Figure 1A The computing unit 101 in Figure 1BThe structure of a computing unit 101 is exemplified. The tensor computing unit 102 in the computing unit 101 includes 4 tensor computing cores (T-Cores), and the 4 tensor computing cores are all used for tensor computing. The vector computing unit 103 in the computing unit 101 includes 16 vector computing cores (V-Cores), and the 16 vector computing cores are all used for vector computing.

[0038] In practical applications, one of the structures of the Attention operator is Figure 2 As shown, the Attention operator includes a first matrix operator, a scaling operator (Scale), a mask operator (Mask), a normalization operator (SoftMax), and a second matrix operator. The first matrix operator and the second matrix operator are both matrix operators (MatMul), and the matrix operator may include at least one matrix multiply accumulation operation (Matrix Multiply Accumulate, MMA for short). The scaling operator, the mask operator, and the normalization operator are all vector operators.

[0039] In the embodiments of the present application, the process of using the Attention operator to train or reason about any batch of tasks is the same. For the sake of convenience of explanation, the following is a detailed explanation using the Attention operator to reason about any batch of tasks as an example.

[0040] When using the Attention operator to reason about any batch of tasks, the specific calculation process of the Attention operator can be referred to Figure 3 The steps shown are specifically as follows: the tensor calculation unit first uses the first matrix operator to perform matrix calculation on the query vector Q and the key vector K to obtain the correlation between the query vector Q and the key vector K, and the vector calculation unit then uses the vector operator (scaling operator, masking operator and normalization operator, respectively) to perform vector calculation on the correlation between the query vector Q and the key vector K to obtain the weight. Finally, the tensor calculation unit uses the second matrix operator to perform matrix calculation on the weight and value vector V to obtain the final output vector.

[0041] If there are multiple batches of tasks to be inferred, the Attention operator can be executed for each batch of tasks. Figure 3 In order to improve the processing efficiency of multiple batch tasks, two thread warp groups are generally used to process each batch task in parallel, where each thread warp group is responsible for the complete processing of a batch task. Therefore, each thread warp group is assigned a tensor computing unit and a vector computing unit, and each thread warp group performs matrix calculations and vector calculations.

[0042] Take two thread warp groups processing four batch tasks to be inferred as an example. The specific calculation process can be referred to Figure 4Steps shown. Figure 4 It can be seen that warp group 1 processes batch tasks 1 and batch tasks 3, and warp group 2 processes batch tasks 2 and batch tasks 4.

[0043] Taking the example of warp group 1 processing batch task 1 and warp group 2 processing batch task 2, the tensor calculation unit first uses the first matrix operator to perform matrix calculation on batch task 1. After that, the vector calculation unit uses the vector operator to perform vector calculation on batch task 1, and at the same time, the tensor calculation unit uses the first matrix operator to perform matrix calculation on batch task 2. Then, the tensor calculation unit uses the second matrix operator to perform matrix calculation on batch task 1, and at the same time, the vector calculation unit uses the vector operator to perform vector calculation on batch task 2.

[0044] From the above, it can be seen that although the two thread bundle groups can process each batch task in parallel, when thread bundle group 1 processes the batch task through the tensor computing unit, thread bundle group 2 cannot process the batch task through the tensor computing unit at the same time, that is, the tensor computing unit cannot process two batch tasks at the same time, which will cause the tensor computing unit or the vector computing unit to be idle. Since the tensor computing unit and the vector computing unit are not fully utilized, the time overhead brought by the Attention operator reasoning the above four batch tasks is relatively large. In addition, each thread bundle group is assigned a tensor computing unit and a vector computing unit, and each thread bundle group needs to perform matrix calculations and vector calculations, which will also increase the complexity of managing thread bundle groups.

[0045] In view of this, when the Attention operator processes N batches of tasks, and the Attention operator includes at least a first matrix operator, a vector operator, and a second matrix operator that execute any batch of tasks in sequence, N is an integer greater than 2. In order to increase the speed at which the Attention operator processes N batches of tasks and reduce time overhead, this application proposes an operator execution method.

[0046] In conjunction with this application Figure 1A As shown in the system architecture diagram, set Figure 1A The tensor computing unit in the artificial intelligence chip includes S tensor computing cores, and the vector computing unit includes W vector computing cores, where S and W are both positive integers. S1 tensor computing cores are selected from the S tensor computing cores and assigned to the first thread warp group, and S2 tensor computing cores are selected from the S tensor computing cores and assigned to the third thread warp group, wherein the first thread warp group is used to perform matrix calculations on batch tasks using the first matrix operator, and the third thread warp group is used to perform matrix calculations on batch tasks using the second matrix operator, there is no overlap between the S1 tensor computing cores and the S2 tensor computing cores, S1 and S2 are both less than S, and the sum of S1 and S2 is less than or equal to S.

[0047] In addition, the W vector computing cores in the artificial intelligence chip are allocated to the second thread warp group, wherein the second thread warp group is used to perform vector computing on batch tasks using vector operators.

[0048] For example, assume that the artificial intelligence chip includes 4 computing units, each of which includes 4 tensor computing cores and 16 vector computing cores. The allocation scheme of the tensor computing cores and vector computing cores in the above artificial intelligence chip can refer to Figure 5 , that is, the tensor computing cores in computing unit 1 and computing unit 2 are allocated to the first thread warp group, the tensor computing cores in computing unit 3 and computing unit 4 are allocated to the third thread warp group, and the vector computing cores in computing unit 1, computing unit 2, computing unit 3 and computing unit 4 are allocated to the second thread warp group.

[0049] The allocation scheme of tensor computing cores and vector computing cores in the above-mentioned artificial intelligence chip can also refer to Figure 6 , that is, the four tensor computing cores in each computing unit are evenly distributed to the first thread warp group and the third thread warp group, and the vector computing cores in computing unit 1, computing unit 2, computing unit 3 and computing unit 4 are distributed to the second thread warp group.

[0050] The allocation scheme of tensor computing cores and vector computing cores in the above-mentioned artificial intelligence chip can also adopt other allocation schemes, which will not be described in detail here. Since the tensor computing cores in the same computing unit are allocated to the same thread warp group, the processing efficiency of the thread warp group can be higher. Therefore, the following text will follow Figure 5 The allocation scheme shown allocates tensor computing cores and vector computing cores in the artificial intelligence chip.

[0051] The process of an operator execution method proposed in an embodiment of the present application is interactively executed by the first thread warp group, the second thread warp group, and the third thread warp group, and the tensor computing cores or vector computing cores allocated to the first thread warp group, the second thread warp group, and the third thread warp group refer to Figure 5 , the above operator execution method may include the following steps: When the second thread warp group uses a vector operator to perform vector calculation on the nth batch task among N batch tasks, the first thread warp group uses a first matrix operator to perform matrix calculation on the n+1th batch task among N batch tasks, and the third thread warp group uses a second matrix operator to perform matrix calculation on the n-1th batch task among N batch tasks; the value range of n is [2,N].

[0052] Specifically, since the second thread bundle group uses a vector operator to perform vector calculations on the nth batch of tasks, it needs to rely on the first thread bundle group to use the first matrix operator to perform matrix calculations on the nth batch of tasks. Therefore, before the second thread bundle group uses a vector operator to perform vector calculations on the nth batch of tasks, the nth batch of tasks has been matrix calculated by the first matrix operator and the calculation results have been obtained, and the n-1th batch of tasks have been vector calculated by the vector operator and the calculation results have been obtained. Therefore, when the second thread bundle group uses a vector operator to perform vector calculations on the nth batch of tasks, the first thread bundle group at this time can use the first matrix operator to perform matrix calculations on the n+1th batch of tasks, and the third thread bundle group at this time can use the second matrix operator to perform matrix calculations on the n-1th batch of tasks.

[0053] It should be understood that in the embodiment of the present application, the nth batch of tasks can be the tasks to be processed by the Attention operator, can be the tasks to be processed by the Multi-Head Attention (MHA) operator, can be the tasks to be processed by an Attention operator in the MHA operator, or can be part of the tasks to be processed by an Attention operator in the MHA operator, and is not limited here.

[0054] For the convenience of the following description, the following detailed description is given by taking the nth batch of tasks as the tasks to be processed by the Attention operator as an example.

[0055] In the above method, when the Attention operator processes N batches of tasks, when the second thread bundle group uses the vector operator to perform vector calculation on the nth batch of tasks, the first thread bundle group uses the first matrix operator to perform matrix calculation on the n+1th batch of tasks, and the third thread bundle group uses the second matrix operator to perform matrix calculation on the n-1th batch of tasks. Since the first thread bundle group and the third thread bundle group are both composed of tensor calculation units, the tensor calculation unit can process two batches of tasks at the same time. In the above process, the tensor calculation unit and the vector calculation unit can process different batches of tasks at the same time, avoiding waiting time for each other, and the tensor calculation unit and the vector calculation unit are fully utilized, thereby reducing the time overhead caused by the Attention operator processing multiple batches of tasks. In addition, since each thread bundle group only performs one calculation, such as the first thread bundle group performs matrix calculation, the second thread bundle group performs vector calculation, and the third thread bundle group performs matrix calculation, the complexity of managing the thread bundle group can be effectively reduced.

[0056] For each batch task, the execution process of the first thread bundle group, the second thread bundle group, and the third thread bundle group is described in detail below, including the following: Figure 7 The following embodiments are shown.

[0057] S701: When the first warp group is in an idle state, the first warp group uses a first matrix operator to perform matrix calculation on the nth batch of tasks.

[0058] S702, after the first thread warp group completes the matrix calculation on the nth batch of tasks using the first matrix operator, the first thread warp group sends a first instruction corresponding to the nth batch of tasks to the second thread warp group, wherein the first instruction corresponding to the nth batch of tasks is used to indicate that the first thread warp group completes the matrix calculation on the nth batch of tasks using the first matrix operator.

[0059] In the above S702, after the first warp group completes the matrix calculation on the nth batch of tasks using the first matrix operator, it can obtain the first intermediate result corresponding to the nth batch of tasks and write the first intermediate result corresponding to the nth batch of tasks into the register.

[0060] S703: If the second warp group receives the first instruction corresponding to the nth batch of tasks and the second warp group completes vector calculations on the n-1th batch of tasks, the second warp group uses a vector operator to perform vector calculations on the nth batch of tasks.

[0061] S704, after the second thread warp group completes the vector calculation on the nth batch of tasks using the vector operator, the second thread warp group sends a second instruction corresponding to the nth batch of tasks to the third thread warp group, wherein the second instruction corresponding to the nth batch of tasks is used to indicate that the second thread warp group completes the vector calculation on the nth batch of tasks using the vector operator.

[0062] In the above S704, the second thread warp group first obtains the first intermediate result corresponding to the nth batch of tasks, wherein the first intermediate result corresponding to the nth batch of tasks is the result of the first thread warp group performing matrix calculation on the nth batch of tasks using the first matrix operator; the second thread warp group then uses the vector operator to perform vector calculation on the first intermediate result corresponding to the nth batch of tasks to obtain the second intermediate result corresponding to the nth batch of tasks, and writes the second intermediate result corresponding to the nth batch of tasks into the register.

[0063] In the above S704, the second warp group may obtain the first intermediate result corresponding to the nth batch of tasks through implementation mode A1, A2 or A3.

[0064] In implementation A1, if the register into which the first intermediate result corresponding to the nth batch of tasks is written is located in the same computing unit as the second warp group, the second warp group can obtain the first intermediate result corresponding to the nth batch of tasks from the register.

[0065] In implementation A2, if the register into which the first intermediate result corresponding to the nth batch of tasks is written is located in a different computing unit from the second warp group, and the second warp group can access registers in other computing units, the second warp group can obtain the first intermediate result corresponding to the nth batch of tasks from the register.

[0066] Implementation method A3, if the register into which the first intermediate result corresponding to the nth batch of tasks is written is located in a different computing unit from the second thread warp group, and the second thread warp group cannot access the registers in other computing units, then the first thread warp group first writes the first intermediate result corresponding to the nth batch of tasks into the on-chip cache, and the second thread warp group then obtains the first intermediate result corresponding to the nth batch of tasks from the on-chip cache.

[0067] S705 , when the second warp group uses the vector operator to perform vector calculation on the nth batch of tasks, the first warp group uses the first matrix operator to perform matrix calculation on the (n+1)th batch of tasks.

[0068] In the above 705, when the second warp group uses the vector operator to perform vector calculation on the nth batch of tasks, the first warp group has completed matrix calculation on the nth batch of tasks. Therefore, the first warp group can use the first matrix operator to perform matrix calculation on the (n+1)th batch of tasks.

[0069] S706, when the second thread warp group uses the vector operator to perform vector calculation on the nth batch of tasks, if the third thread warp group receives the second instruction corresponding to the n-1th batch of tasks sent by the second thread warp group and the third thread warp group completes the matrix calculation on the n-2th batch of tasks using the second matrix operator, then the third thread warp group uses the second matrix operator to perform matrix calculation on the n-1th batch of tasks.

[0070] In the above 706, the third warp group obtains the second intermediate result corresponding to the n-1th batch of tasks from the register, wherein the second intermediate result corresponding to the n-1th batch of tasks is the result of the second warp group performing vector calculation on the n-1th batch of tasks using the vector operator; the third warp group performs matrix calculation on the second intermediate result corresponding to the n-1th batch of tasks using the second matrix operator to obtain the result of the n-1th batch of tasks, and writes the result of the n-1th batch of tasks into the register. In order to facilitate the output of the result of the n-1th batch of tasks, the third warp group may also write the result of the n-1th batch of tasks into the on-chip cache.

[0071] It should be understood that when the nth batch of tasks is a task to be processed by the Attention operator or a task to be processed by the MHA operator, the result of the nth batch of tasks in the on-chip cache can be directly used as the final result of the nth batch of tasks. When the nth batch of tasks is a task to be processed by an Attention operator in the MHA operator or a part of a task to be processed by an Attention operator in the MHA operator, the results of the nth batch of tasks in the on-chip cache can be concatenated to obtain the final result of the nth batch of tasks.

[0072] Above Figure 7 The steps shown provide a method for a first warp group, a second warp group, and a third warp group to process batches of tasks in parallel.

[0073] For ease of understanding, the following takes the Attention operator to perform reasoning on three batches of tasks as an example to discuss in detail the above operator execution method in this application. Since the calculation time of the first matrix operator, the second matrix operator, and the vector operator are not the same, for ease of discussion, this application assumes that the calculation time of the first matrix operator is the same as the calculation time of the second matrix operator. The following is a specific discussion of the above operator execution method in this application, including the following implementation methods B1 or B2, in conjunction with the relationship between the calculation times corresponding to the first matrix operator, the second matrix operator, and the vector operator.

[0074] Implementation method B1: The calculation time of the vector operator is less than or equal to the calculation time of the first matrix operator (or the second matrix operator). The Attention operator is calculated for the three batches of tasks in sequence. The specific calculation process can be referred to Figure 8 Steps shown.

[0075] Assume that the first warp group, the second warp group and the third warp group are all in an idle state at time t0, and instruction transmission between the first warp group, the second warp group and the third warp group does not take time.

[0076] The first thread bundle group uses the first matrix operator to perform matrix calculation on the query vector Q and the key vector K of the first batch of tasks at time t0, obtains the correlation between the query vector Q and the key vector K of the first batch of tasks at time t1, and sends the first instruction corresponding to the first batch of tasks to the second thread bundle group at time t1, wherein the first instruction corresponding to the first batch of tasks is used to indicate that the first thread bundle group has completed the matrix calculation on the first batch of tasks using the first matrix operator.

[0077] The second thread bundle group receives the first instruction corresponding to the first batch of tasks at time t1, and uses the vector operator to perform vector calculation on the correlation corresponding to the first batch of tasks at time t1, obtains the weight corresponding to the first batch of tasks at time t2, and sends the second instruction corresponding to the first batch of tasks to the third thread bundle group at time t2, wherein the second instruction corresponding to the first batch of tasks is used to indicate that the second thread bundle group has completed the vector calculation of the first batch of tasks using the vector operator.

[0078] When the second thread warp group uses the vector operator to perform vector calculation on the correlation corresponding to the first batch of tasks, the first thread warp group uses the first matrix operator to perform matrix calculation on the query vector Q and the key vector K of the second batch of tasks at time t1, obtains the correlation between the query vector Q and the key vector K of the second batch of tasks at time t3, and sends the first instruction corresponding to the second batch of tasks to the second thread warp group at time t3, wherein the first instruction corresponding to the second batch of tasks is used to indicate that the first thread warp group has completed the matrix calculation of the second batch of tasks using the first matrix operator.

[0079] The second thread warp group receives the first instruction corresponding to the second batch of tasks at time t3 and completes the vector calculation for the first batch of tasks at time t2. Since time t3 is greater than time t2, the second thread warp group uses the vector operator to perform vector calculation on the correlation corresponding to the second batch of tasks at time t3, obtains the weight corresponding to the second batch of tasks at time t4, and sends the second instruction corresponding to the second batch of tasks to the third thread warp group at time t4, wherein the second instruction corresponding to the second batch of tasks is used to indicate that the second thread warp group has completed the vector calculation of the second batch of tasks using the vector operator.

[0080] When the second thread warp group uses the vector operator to perform vector calculation on the correlation corresponding to the second batch of tasks, the third thread warp group uses the second matrix operator to perform matrix calculation on the weights corresponding to the first batch of tasks and the value vector V of the first batch of tasks at time t2, and obtains the output vector of the first batch of tasks at time t4. In addition, the first thread warp group uses the first matrix operator to perform matrix calculation on the query vector Q and the key vector K of the third batch of tasks at time t3, obtains the correlation between the query vector Q and the key vector K of the third batch of tasks at time t5, and sends the first instruction corresponding to the third batch of tasks to the second thread warp group at time t5, wherein the first instruction corresponding to the third batch of tasks is used to indicate that the first thread warp group has completed the matrix calculation of the third batch of tasks using the first matrix operator.

[0081] The second thread warp group receives the first instruction corresponding to the third batch of tasks at time t5 and completes the vector calculation of the second batch of tasks at time t4. Since time t5 is greater than time t4, the second thread warp group uses the vector operator to perform vector calculation on the correlation corresponding to the third batch of tasks at time t5, obtains the weight corresponding to the third batch of tasks at time t6, and sends the second instruction corresponding to the third batch of tasks to the third thread warp group at time t6, wherein the second instruction corresponding to the third batch of tasks is used to indicate that the second thread warp group has completed the vector calculation of the third batch of tasks using the vector operator.

[0082] When the second thread warp group uses the vector operator to perform vector calculation on the correlation corresponding to the third batch of tasks, the third thread warp group uses the second matrix operator to perform matrix calculation on the weights corresponding to the second batch of tasks and the value vector V of the second batch of tasks at time t4, and obtains the output vector of the second batch of tasks at time t6.

[0083] The third thread warp group receives the second instruction corresponding to the third batch of tasks at time t6 and the matrix calculation of the second batch of tasks is completed at time t6. Therefore, the third thread warp group uses the second matrix operator to perform matrix calculation on the weights corresponding to the third batch of tasks and the value vector V of the third batch of tasks at time t6, and obtains the output vector of the third batch of tasks at time t8.

[0084] Implementation method B2: the calculation time of the vector operator is longer than the calculation time of the first matrix operator (or the second matrix operator). The Attention operator performs calculations on three batches of tasks in sequence. For the specific calculation process, please refer to Fig. 9 Steps shown.

[0085] Assume that the first warp group, the second warp group and the third warp group are all in an idle state at time t0, and instruction transmission between the first warp group, the second warp group and the third warp group does not take time.

[0086] The first thread bundle group uses the first matrix operator to perform matrix calculation on the query vector Q and the key vector K of the first batch of tasks at time t0, obtains the correlation between the query vector Q and the key vector K of the first batch of tasks at time t1, and sends the first instruction corresponding to the first batch of tasks to the second thread bundle group at time t1, wherein the first instruction corresponding to the first batch of tasks is used to indicate that the first thread bundle group has completed the matrix calculation on the first batch of tasks using the first matrix operator.

[0087] The second thread bundle group receives the first instruction corresponding to the first batch of tasks at time t1, and uses the vector operator to perform vector calculation on the correlation corresponding to the first batch of tasks at time t1, obtains the weight corresponding to the first batch of tasks at time t3, and sends the second instruction corresponding to the first batch of tasks to the third thread bundle group at time t3, wherein the second instruction corresponding to the first batch of tasks is used to indicate that the second thread bundle group has completed the vector calculation of the first batch of tasks using the vector operator.

[0088] When the second thread warp group uses the vector operator to perform vector calculation on the correlation corresponding to the first batch of tasks, the first thread warp group uses the first matrix operator to perform matrix calculation on the query vector Q and the key vector K of the second batch of tasks at time t1, obtains the correlation between the query vector Q and the key vector K of the second batch of tasks at time t2, and sends the first instruction corresponding to the second batch of tasks to the second thread warp group at time t2, wherein the first instruction corresponding to the second batch of tasks is used to indicate that the first thread warp group has completed the matrix calculation of the second batch of tasks using the first matrix operator.

[0089] The second thread warp group receives the first instruction corresponding to the second batch of tasks at time t2 and completes the vector calculation for the first batch of tasks at time t3. Since time t3 is greater than time t2, the second thread warp group uses the vector operator to perform vector calculation on the correlation corresponding to the second batch of tasks at time t3, obtains the weight corresponding to the second batch of tasks at time t6, and sends the second instruction corresponding to the second batch of tasks to the third thread warp group at time t6, wherein the second instruction corresponding to the second batch of tasks is used to indicate that the second thread warp group has completed the vector calculation of the second batch of tasks using the vector operator.

[0090] When the second thread warp group uses the vector operator to perform vector calculation on the correlation corresponding to the second batch of tasks, the third thread warp group uses the second matrix operator to perform matrix calculation on the weights corresponding to the first batch of tasks and the value vector V of the first batch of tasks at time t3, and obtains the output vector of the first batch of tasks at time t5. In addition, the first thread warp group uses the first matrix operator to perform matrix calculation on the query vector Q and the key vector K of the third batch of tasks at time t2, obtains the correlation between the query vector Q and the key vector K of the third batch of tasks at time t4, and sends the first instruction corresponding to the third batch of tasks to the second thread warp group at time t4, wherein the first instruction corresponding to the third batch of tasks is used to indicate that the first thread warp group has completed the matrix calculation of the third batch of tasks using the first matrix operator.

[0091] The second thread warp group receives the first instruction corresponding to the third batch of tasks at time t4 and completes the vector calculation of the second batch of tasks at time t6. Since time t6 is greater than time t4, the second thread warp group uses the vector operator to perform vector calculation on the correlation corresponding to the third batch of tasks at time t6, obtains the weight corresponding to the third batch of tasks at time t8, and sends the second instruction corresponding to the third batch of tasks to the third thread warp group at time t8, wherein the second instruction corresponding to the third batch of tasks is used to indicate that the second thread warp group has completed the vector calculation of the third batch of tasks using the vector operator.

[0092] When the second thread warp group uses the vector operator to perform vector calculation on the correlation corresponding to the third batch of tasks, the third thread warp group uses the second matrix operator to perform matrix calculation on the weights corresponding to the second batch of tasks and the value vector V of the second batch of tasks at time t6, and obtains the output vector of the second batch of tasks at time t7.

[0093] The third thread warp group receives the second instruction corresponding to the third batch of tasks at time t8 and completes the matrix calculation for the second batch of tasks at time t7. Since time t8 is greater than time t7, the third thread warp group uses the second matrix operator to perform matrix calculation on the weights corresponding to the third batch of tasks and the value vector V of the third batch of tasks at time t8, and obtains the output vector of the third batch of tasks at time t9.

[0094] In the above-mentioned Implementation A1 or Implementation A2, any time from t0 to t9 in different implementations may be the same time or different time.

[0095] Based on the same technical concept, the present application embodiment provides a structural diagram of an operator execution device, such as Fig.10 As shown, the operator execution device 1000 includes: A matrix calculation module 1001 is used to perform matrix calculation on the n+1th batch task among the N batch tasks by using a first matrix operator through the first thread bundle group when the second thread bundle group uses a vector operator to perform vector calculation on the nth batch task among the N batch tasks, and to perform matrix calculation on the n-1th batch task among the N batch tasks by using a second matrix operator through the third thread bundle group; the second thread bundle group includes W vector calculation cores, the first thread bundle group includes S1 tensor calculation cores among the S tensor calculation cores, the third thread bundle group includes S2 tensor calculation cores among the S tensor calculation cores, and there is no overlap between the S1 tensor calculation cores and the S2 tensor calculation cores; the value range of n is [2, N], N is an integer greater than 2, S1 and S2 are both less than S, and the sum of S1 and S2 is less than or equal to S.

[0096] Optionally, the vector calculation module 1002 is used to perform vector calculation on the nth batch of tasks by the second thread bundle group using the vector operator when the second thread bundle group receives the first instruction corresponding to the nth batch of tasks sent by the first thread bundle group and the second thread bundle group completes the vector calculation on the n-1th batch of tasks; the first instruction corresponding to the nth batch of tasks is used to indicate that the first thread bundle group has completed the matrix calculation on the nth batch of tasks using the first matrix operator.

[0097] Optionally, the vector calculation module 1002 is specifically configured to: acquiring, through the second warp group, a first intermediate result corresponding to the nth batch of tasks; the first intermediate result corresponding to the nth batch of tasks is a result of matrix calculation performed by the first warp group on the nth batch of tasks using the first matrix operator; The second thread warp group uses the vector operator to perform vector calculation on the first intermediate result corresponding to the nth batch of tasks to obtain a second intermediate result corresponding to the nth batch of tasks.

[0098] Optionally, the matrix calculation module 1001 is specifically used for: When the first warp group finishes the matrix calculation on the nth batch of tasks by using the first matrix operator, the first warp group performs the matrix calculation on the (n+1)th batch of tasks by using the first matrix operator.

[0099] Optionally, the matrix calculation module 1001 is specifically used for: When the third thread warp group receives the second instruction corresponding to the n-1th batch of tasks sent by the second thread warp group and the third thread warp group completes matrix calculation on the n-2th batch of tasks using the second matrix operator, the third thread warp group performs matrix calculation on the n-1th batch of tasks using the second matrix operator; the second instruction corresponding to the n-1th batch of tasks is used to indicate that the second thread warp group completes vector calculation on the n-1th batch of tasks using the vector operator.

[0100] Optionally, the matrix calculation module 1001 is specifically used for: Acquire the second intermediate result corresponding to the n-1th batch of tasks through the third warp group; the second intermediate result corresponding to the n-1th batch of tasks is the result of the second warp group performing vector calculation on the n-1th batch of tasks using the vector operator; The results of the n-1th batch of tasks are obtained by performing matrix calculation on the second intermediate results corresponding to the n-1th batch of tasks using the second matrix operator through the third thread warp group.

[0101] Optionally, the matrix calculation module 1001 is also used to: before the second thread warp group uses the vector operator to perform vector calculation on the nth batch of tasks among the N batches of tasks, complete the matrix calculation of the nth batch of tasks by the first thread warp group using the first matrix operator, and send the first instruction corresponding to the nth batch of tasks to the second thread warp group through the first thread warp group.

[0102] Based on the same technical concept, the embodiment of the present application provides a computer device, such as Fig.11 As shown, it includes at least one artificial intelligence chip 1101 and a memory 1102 connected to the at least one artificial intelligence chip. The specific connection medium between the artificial intelligence chip 1101 and the memory 1102 is not limited in the embodiment of the present application. Fig.11 For example, the artificial intelligence chip 1101 and the memory 1102 are connected via a bus. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0103] In an embodiment of the present application, the memory 1102 stores instructions that can be executed by at least one artificial intelligence chip 1101. At least one artificial intelligence chip 1101 can execute the steps of the above-mentioned operator execution method by executing the instructions stored in the memory 1102.

[0104] Among them, the artificial intelligence chip 1101 is the control center of the computer device, which can use various interfaces and lines to connect various parts of the computer device, and realize operator execution by running or executing instructions stored in the memory 1102 and calling data stored in the memory 1102. Optionally, the artificial intelligence chip 1101 may include one or more processing units, and the artificial intelligence chip 1101 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application program, etc., and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the artificial intelligence chip 1101. In some embodiments, the artificial intelligence chip 1101 and the memory 1102 can be implemented on the same chip, and in some embodiments, they can also be implemented separately on independent chips.

[0105] The artificial intelligence chip 1101 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware processor for execution, or can be executed by a combination of hardware and software modules in the processor.

[0106] The memory 1102 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 1102 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, and the like. The memory 1102 is any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer device, but is not limited thereto. The memory 1102 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0107] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program executable by a computer device. When the program runs on the computer device, the computer device executes the steps of the above-mentioned operator execution method.

[0108] Based on the same inventive concept, an embodiment of the present application provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer device, the computer device executes the steps of the above-mentioned operator execution method.

[0109] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0110] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0111] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0112] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0113] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. An operator execution method, characterized in that: Applicable to an artificial intelligence chip including S tensor computing cores and W vector computing cores, wherein the artificial intelligence chip is used to process N batches of tasks using an attention mechanism operator; The attention mechanism operator at least includes a first matrix operator, a vector operator and a second matrix operator that sequentially execute any batch of tasks in the N batches of tasks; S and W are both positive integers, and N is an integer greater than 2; When the second thread warp group uses the vector operator to perform vector calculation on the nth batch task among the N batch tasks, the first thread warp group uses the first matrix operator to perform matrix calculation on the n+1th batch task among the N batch tasks, and the third thread warp group uses the second matrix operator to perform matrix calculation on the n-1th batch task among the N batch tasks; the second thread warp group includes the W vector computing cores, the first thread warp group includes S1 tensor computing cores among the S tensor computing cores, the third thread warp group includes S2 tensor computing cores among the S tensor computing cores, and there is no overlap between the S1 tensor computing cores and the S2 tensor computing cores; the value range of n is [2,N], S1 and S2 are both less than S, and the sum of S1 and S2 is less than or equal to S.

2. The method according to claim 1, characterized in that The second warp group uses the vector operator to perform vector calculation on the nth batch of tasks in the N batches of tasks, including: When the second thread warp group receives the first instruction corresponding to the nth batch of tasks sent by the first thread warp group and the second thread warp group completes the vector calculation of the n-1th batch of tasks, the second thread warp group uses the vector operator to perform vector calculation on the nth batch of tasks; the first instruction corresponding to the nth batch of tasks is used to indicate that the first thread warp group has completed the matrix calculation of the nth batch of tasks using the first matrix operator.

3. The method according to claim 2, characterized in that The second warp group uses the vector operator to perform vector calculation on the nth batch of tasks, including: The second warp group obtains a first intermediate result corresponding to the nth batch of tasks; the first intermediate result corresponding to the nth batch of tasks is a result of the first warp group performing matrix calculation on the nth batch of tasks using the first matrix operator; The second warp group uses the vector operator to perform vector calculation on the first intermediate result corresponding to the nth batch of tasks to obtain a second intermediate result corresponding to the nth batch of tasks.

4. The method according to claim 1, characterized in that The first warp group uses the first matrix operator to perform matrix calculation on the (n+1)th batch of tasks in the N batches of tasks, including: After the first warp group completes matrix calculation on the nth batch of tasks using the first matrix operator, the first warp group performs matrix calculation on the (n+1)th batch of tasks using the first matrix operator.

5. The method according to claim 1, characterized in that The third warp group uses the second matrix operator to perform matrix calculation on the n-1th batch of tasks in the N batches of tasks, including: When the third thread warp group receives the second instruction corresponding to the n-1th batch of tasks sent by the second thread warp group and the third thread warp group completes matrix calculation on the n-2th batch of tasks using the second matrix operator, the third thread warp group performs matrix calculation on the n-1th batch of tasks using the second matrix operator; the second instruction corresponding to the n-1th batch of tasks is used to indicate that the second thread warp group completes vector calculation on the n-1th batch of tasks using the vector operator.

6. The method according to claim 5, characterized in that The third warp group uses the second matrix operator to perform matrix calculation on the n-1th batch of tasks, including: The third warp group obtains the second intermediate result corresponding to the n-1th batch of tasks; the second intermediate result corresponding to the n-1th batch of tasks is a result of the second warp group performing vector calculation on the n-1th batch of tasks using the vector operator; The third thread warp group uses the second matrix operator to perform matrix calculation on the second intermediate result corresponding to the n-1th batch of tasks to obtain the result of the n-1th batch of tasks.

7. The method according to claim 1, characterized in that Before the second warp group uses the vector operator to perform vector calculation on the nth batch of tasks in the N batches of tasks, the method further includes: The first warp group completes matrix calculation of the nth batch of tasks by using the first matrix operator, and the first warp group sends a first instruction corresponding to the nth batch of tasks to the second warp group.

8. A computer device comprising a memory, the artificial intelligence chip, and a computer program stored in the memory and running on the artificial intelligence chip, characterized in that: When the artificial intelligence chip executes the computer program, the steps of the method as described in any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium, characterized in that: It stores a computer program executable by a computer device. When the computer program is run on the computer device, the computer device executes the steps of any method as claimed in claims 1-7.

10. A computer program product, characterized in that The computer program product comprises a computer program stored on a computer-readable storage medium, wherein the computer program comprises program instructions, and when the program instructions are executed by a computer device, the computer device is caused to execute the steps of the method according to any one of claims 1 to 7.

Images