Operator execution method, device, storage medium and program product

Abstract

The embodiment of the present application provides an operator execution method, device, storage medium and program product, which relates to the field of artificial intelligence technology. In the method, an artificial intelligence chip is used to use a multi-head attention mechanism operator to execute M1 tasks, and the multi-head attention mechanism operator includes N attention mechanism operators; the first processing unit determines K1 subtasks to be executed, and any of the K1 subtasks to be executed is a partial task of executing one of the M1 tasks using the attention mechanism operator, and the first processing unit uses the attention mechanism operator to execute the K1 subtasks to be executed respectively. In the above process, multiple subtasks to be executed belonging to the same task can be assigned to different processing units, and different processing units use the attention mechanism operator to execute the subtasks to be executed in parallel. Therefore, the time overhead caused by the processing unit using the multi-head attention mechanism operator to execute the task can be effectively reduced.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of artificial intelligence technology, and in particular, to an operator execution method, device, storage medium, and program product. Background Art

[0002] With the rapid development of artificial intelligence technology, large language models (LLMs) are widely used in natural language processing, computer vision, speech recognition and other fields. At present, large language models generally use the Transformer structure as the core infrastructure, and the core computing part of the Transformer structure is the Attention operator.

[0003] In order to improve the expressiveness of large language models, the Multi-Head Attention (MHA) operator is currently more commonly used to build large language models. However, using the Multi-Head Attention operator to reason about tasks will bring huge time overhead. How to reduce the time overhead brought about by using the Multi-Head Attention operator to reason about tasks is a technical problem that needs to be solved urgently. Summary of the invention

[0004] The embodiments of the present application provide an operator execution method, device, storage medium, and program product for reducing the time overhead caused by using a Multi-Head Attention operator for task reasoning.

[0005] On the one hand, an embodiment of the present application provides an operator execution method, which is applied to an artificial intelligence chip including S processing units, wherein the artificial intelligence chip is used to execute M1 tasks using a Multi-Head Attention operator, wherein the Multi-Head Attention operator includes N attention mechanism Attention operators; S, M1, and N are all greater than 0, including:

[0006] A first processing unit determines K1 subtasks to be executed, where the first processing unit is any processing unit among the S processing units, and any subtask to be executed among the K1 subtasks to be executed is a partial task of executing one of the M1 tasks using the Attention operator, and K1 is greater than or equal to 0;

[0007] The first processing unit uses the Attention operator to respectively execute the K1 subtasks to be executed.

[0008] Optionally, the first processing unit among the S processing units determines K1 subtasks to be executed, including:

[0009] The first processing unit obtains K1 identification records from the first record corresponding to the first processing unit, where the first record corresponding to the first processing unit is used to record the execution order of the K1 to-be-executed subtasks to be processed by the first processing unit;

[0010] The first processing unit determines the K1 subtasks to be executed according to the K1 identification records.

[0011] Optionally, the first record corresponding to the first processing unit is determined based on K2 executable subtasks corresponding to the first processing unit and a second record, any executable subtask of the K2 executable subtasks is a partial task of using the Attention operator to execute one of the M2 tasks assigned to the first processing unit, and the second record is used to record the execution status corresponding to the M2 tasks respectively; M2 is greater than or equal to M1, and K2 is greater than or equal to K1.

[0012] Optionally, the first record further includes the number of times K1 the first record can be accessed;

[0013] The first processing unit obtains K1 identification records from the first record corresponding to the first processing unit, including:

[0014] The first processing unit obtains the accessible number K1 from the first record corresponding to the first processing unit;

[0015] The first processing unit accesses the first record K1 times according to the accessible number K1, and sequentially obtains the K1 identification records.

[0016] Optionally, the K1 identification records include a first identification record, and the first identification record is any identification record among the K1 identification records;

[0017] The first processing unit determines the K1 to-be-executed subtasks according to the K1 identification records, including:

[0018] The first processing unit determines the first subtask to be executed among the K1 subtasks to be executed based on the first identification record, the number S of processing units included in the artificial intelligence chip, the position identifier of the first processing unit in the S processing units, and the number N of Attention operators.

[0019] Optionally, the first processing unit performs, according to the first identification record, the number S of processing units included in the artificial intelligence chip, the position identification of the first processing unit in the S processing units, and the number N of Attention operators, a first subtask to be executed among the K1 subtasks to be executed, including:

[0020] The first processing unit uses the product of the first identification record and the number S of processing units as a first reference value;

[0021] The first processing unit uses the sum of the first reference value and the position identifier of the first processing unit in the S processing units as the second reference value;

[0022] The first processing unit determines the first subtask to be executed according to the second reference value and the number N of Attention operators.

[0023] Optionally, the first processing unit determines the first to-be-performed subtask according to the second reference value and the number N of the Attention operators, including:

[0024] The first processing unit determines the first task from the M1 tasks according to the quotient of the second reference value and the number N of the Attention operators;

[0025] The first processing unit determines a first operator from the N Attention operators according to a remainder of the second reference value and the number N of the Attention operators;

[0026] The first processing unit determines the first subtask to be executed according to the first task and the first operator.

[0027] Optionally, the first processing unit among the S processing units determines K1 subtasks to be executed, including:

[0028] The first processing unit determines K2 executable subtasks, any executable subtask of the K2 executable subtasks is a partial task of using the Attention operator to execute one of the M2 tasks assigned to the first processing unit; K2 is greater than or equal to K1; M2 is greater than or equal to M1;

[0029] The first processing unit obtains a second record, where the second record is used to record the execution states corresponding to the M2 tasks respectively;

[0030] The first processing unit determines the K1 to-be-executed subtasks according to the K2 executable subtasks and the second record.

[0031] Optionally, the K2 executable subtasks include a first executable subtask, and the first executable subtask is any executable subtask among the K2 executable subtasks;

[0032] The first processing unit determines the K1 subtasks to be executed according to the K2 executable subtasks and the second record, including:

[0033] The first processing unit determines, from the M2 tasks, a second task corresponding to the first executable subtask;

[0034] The first processing unit determines the execution status of the second task according to the second record;

[0035] When the execution state of the second task is in an executable state, the first processing unit uses the first executable subtask as a to-be-executed subtask.

[0036] Optionally, any processing unit among the S processing units includes a tensor calculation unit and a vector calculation unit; the Attention operator includes a first matrix operator, a vector operator and a second matrix operator;

[0037] The first processing unit uses the Attention operator to respectively execute the K1 subtasks to be executed, including:

[0038] When the vector calculation unit in the first processing unit uses the vector operator to perform vector calculation on the zth subtask to be executed, the tensor calculation unit in the first processing unit uses the first matrix operator to perform matrix calculation on the z+ith subtask to be executed, or the tensor calculation unit in the first processing unit uses the second matrix operator to perform matrix calculation on the zjth subtask to be executed; z is an integer that spans [1,K1], and i and j are both integers greater than 0.

[0039] On the one hand, an embodiment of the present application provides an operator execution device, including:

[0040] A determination module, configured for a first processing unit to determine K1 to-be-executed subtasks, wherein the first processing unit is any processing unit among the S processing units, and any to-be-executed subtask among the K1 to-be-executed subtasks is a partial task of executing one of the M1 tasks using the Attention operator, and K1 is greater than or equal to 0;

[0041] An execution module is used for the first processing unit to respectively execute the K1 subtasks to be executed by using the Attention operator.

[0042] Optionally, a module is determined, specifically for:

[0043] The first processing unit obtains K1 identification records from the first record corresponding to the first processing unit, where the first record corresponding to the first processing unit is used to record the execution order of the K1 to-be-executed subtasks to be processed by the first processing unit;

[0044] The first processing unit determines the K1 subtasks to be executed according to the K1 identification records.

[0045] Optionally, the first record corresponding to the first processing unit is determined based on K2 executable subtasks corresponding to the first processing unit and a second record, any executable subtask of the K2 executable subtasks is a partial task of using the Attention operator to execute one of the M2 tasks assigned to the first processing unit, and the second record is used to record the execution status corresponding to the M2 tasks respectively; M2 is greater than or equal to M1, and K2 is greater than or equal to K1.

[0046] Optionally, the first record further includes a number of times K1 of accessibility of the first record; the determination module is specifically configured to:

[0047] The first processing unit obtains the accessible number K1 from the first record corresponding to the first processing unit;

[0048] The first processing unit accesses the first record K1 times according to the accessible number K1, and sequentially obtains the K1 identification records.

[0049] Optionally, the K1 identification records include a first identification record, and the first identification record is any identification record among the K1 identification records; the determination module is specifically configured to:

[0050] The first processing unit determines the first subtask to be executed among the K1 subtasks to be executed based on the first identification record, the number S of processing units included in the artificial intelligence chip, the position identifier of the first processing unit in the S processing units, and the number N of Attention operators.

[0051] Optionally, a module is determined, specifically for:

[0052] The first processing unit uses the product of the first identification record and the number S of processing units as a first reference value;

[0053] The first processing unit uses the sum of the first reference value and the position identifier of the first processing unit in the S processing units as the second reference value;

[0054] The first processing unit determines the first subtask to be executed according to the second reference value and the number N of Attention operators.

[0055] Optionally, a module is determined, specifically for:

[0056] The first processing unit determines the first task from the M1 tasks according to the quotient of the second reference value and the number N of the Attention operators;

[0057] The first processing unit determines a first operator from the N Attention operators according to a remainder of the second reference value and the number N of the Attention operators;

[0058] The first processing unit determines the first subtask to be executed according to the first task and the first operator.

[0059] Optionally, a module is determined, specifically for:

[0060] The first processing unit determines K2 executable subtasks, any executable subtask of the K2 executable subtasks is a partial task of using the Attention operator to execute one of the M2 tasks assigned to the first processing unit; K2 is greater than or equal to K1; M2 is greater than or equal to M1;

[0061] The first processing unit obtains a second record, where the second record is used to record the execution states corresponding to the M2 tasks respectively;

[0062] The first processing unit determines the K1 to-be-executed subtasks according to the K2 executable subtasks and the second record.

[0063] Optionally, the K2 executable subtasks include a first executable subtask, and the first executable subtask is any executable subtask among the K2 executable subtasks; the determination module is specifically used to:

[0064] The first processing unit determines, from the M2 tasks, a second task corresponding to the first executable subtask;

[0065] The first processing unit determines the execution status of the second task according to the second record;

[0066] When the execution state of the second task is in an executable state, the first processing unit uses the first executable subtask as a to-be-executed subtask.

[0067] Optionally, any processing unit among the S processing units includes a tensor calculation unit and a vector calculation unit; the Attention operator includes a first matrix operator, a vector operator and a second matrix operator; and the execution module is specifically used to:

[0068] When the vector calculation unit in the first processing unit uses the vector operator to perform vector calculation on the zth subtask to be executed, the tensor calculation unit in the first processing unit uses the first matrix operator to perform matrix calculation on the z+ith subtask to be executed, or the tensor calculation unit in the first processing unit uses the second matrix operator to perform matrix calculation on the zjth subtask to be executed; z is an integer that spans [1,K1], and i and j are both integers greater than 0.

[0069] On the one hand, an embodiment of the present application provides a computer device, including a memory, an artificial intelligence chip, and a computer program stored in the memory and executable on the artificial intelligence chip, wherein the artificial intelligence chip implements the steps of the above-mentioned operator execution method when executing the computer program.

[0070] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program executable by a computer device. When the computer program runs on the computer device, the computer device executes the steps of the above-mentioned operator execution method.

[0071] On the one hand, an embodiment of the present application provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer device, the computer device executes the steps of the above-mentioned operator execution method.

[0072] In the embodiment of the present application, when the artificial intelligence chip uses the Multi-Head Attention operator to execute M1 tasks, it is not that the M1 processing unit in the artificial intelligence chip uses the Multi-Head Attention operator to execute one task respectively, but that the M1 task is split into multiple sub-tasks to be executed, and S processing units determine the sub-tasks to be executed from the multiple sub-tasks to be executed, and then the S processing units use the Attention operator to execute the sub-tasks to be executed respectively, and finally achieve the purpose of the artificial intelligence chip to quickly execute and complete the M1 tasks. In the above process, multiple sub-tasks to be executed belonging to the same task can be assigned to different processing units, and different processing units use the Attention operator to execute the sub-tasks to be executed in parallel. Therefore, the time overhead caused by the processing unit using the Multi-Head Attention operator to execute tasks can be effectively reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0074] Figure 1 A schematic diagram of a reasoning process provided in an embodiment of the present application;

[0075] Figure 2A A schematic diagram of the structure of a processor chip provided in an embodiment of the present application;

[0076] Figure 2B A schematic diagram of the structure of a computing unit provided in an embodiment of the present application;

[0077] Figure 3 A schematic diagram of a reasoning process provided in an embodiment of the present application;

[0078] Figure 4 A schematic diagram of a reasoning process provided in an embodiment of the present application;

[0079] Figure 5 A flowchart of an operator execution method provided in an embodiment of the present application;

[0080] Figure 6 A flowchart of a method for determining K1 subtasks to be executed provided in an embodiment of the present application;

[0081] Figure 7 A schematic diagram of the structure of executable subtasks allocated to a processing unit provided in an embodiment of the present application;

[0082] Figure 8 A flowchart of a method for determining a subtask to be executed provided in an embodiment of the present application;

[0083] Fig. 9 A schematic diagram of a structure of subtasks to be executed allocated by a processing unit provided in an embodiment of the present application;

[0084] Fig.10 A schematic diagram of a structure of subtasks to be executed allocated by a processing unit provided in an embodiment of the present application;

[0085] Fig.11 A flowchart of a method for determining executable subtasks provided in an embodiment of the present application;

[0086] Fig.12A flowchart of a method for determining executable subtasks provided in an embodiment of the present application;

[0087] Fig.13 A flowchart of a method for determining K1 subtasks to be executed provided in an embodiment of the present application;

[0088] Fig.14 A flowchart of a method for determining a first record provided in an embodiment of the present application;

[0089] Fig.15 A flowchart of a method for obtaining K1 identification records provided in an embodiment of the present application;

[0090] Fig.16 A flowchart of a method for determining a first subtask to be executed provided in an embodiment of the present application;

[0091] Fig.17 A flowchart of a method for determining a first subtask to be executed provided in an embodiment of the present application;

[0092] Fig.18 A schematic diagram of the structure of an Attention operator provided in an embodiment of the present application;

[0093] Fig.19 A flowchart of a method for executing K1 subtasks to be executed provided in an embodiment of the present application;

[0094] Fig. 20 A flowchart of a method for executing K2 executable subtasks provided in an embodiment of the present application;

[0095] Fig.21 A flowchart of a method for executing K1 subtasks to be executed provided in an embodiment of the present application;

[0096] Fig. 22 A flowchart of a method for executing K1 subtasks to be executed provided in an embodiment of the present application;

[0097] Fig.23 A schematic diagram of the structure of an operator execution device provided in an embodiment of the present application;

[0098] Fig.24 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0099] In order to make the purpose, technical solution and beneficial effects of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0100] Below, some terms in the embodiments of the present application are explained to facilitate understanding by those skilled in the art.

[0101] A large language model is a deep learning model trained with a large amount of text data that can generate natural language text or understand the meaning of language text. A large language model can handle a variety of natural language tasks, such as text classification, question answering, and dialogue, and is an important path to artificial intelligence.

[0102] Prompt information, in the large language model, prompt information is a piece of prompt text used to guide the large language model to generate a specific type of output. By designing and optimizing a set of prompt information, the large language model can perform specific tasks.

[0103] When using a large language model to reason about a task, the reasoning process can be divided into two stages: the prefill stage and the decoder stage. The prefill stage includes the large language model receiving prompt information and generating the first element (Token) based on the prompt information. The Token can be a symbol used to represent Chinese characters, English words, or Chinese and English phrases. The decoder stage includes the large language model generating the next Token based on the prompt information and the generated Token until the terminator (END) is generated.

[0104] Take a large language model reasoning task as an example. The prompt information corresponding to this task is "What day is today?" The reasoning process of the large language model can be referred to Figure 1 , the large language model obtains the prompt information "What day of the week is today", and then generates the first Token "star" according to the prompt information. After that, the large language model generates the second Token "period" based on the prompt information and the first Token "star". After that, the large language model generates the third Token "three" based on the prompt information and the first Token "star", the second Token "period", and the third Token "three". Finally, the large language model generates the fourth Token "END" based on the prompt information and the first Token "star", the second Token "period", and the third Token "three". At this point, the reasoning process of the large language model for the above task ends.

[0105] In both the Prefill stage and the Decoder stage, the Multi-Head Attention operator in the large language model is used to infer the task. Since the Multi-Head Attention operator includes N Attention operators, N is greater than 0, in the Prefill stage and the Decoder stage, when N Attention operators are used to infer the task separately, it will incur huge time overhead.

[0106] In order to solve the above problems, in view of this, an embodiment of the present application provides an operator execution method.

[0107] refer to Figure 2A , which is a system architecture diagram applicable to the embodiment of the present application, the system architecture includes a central processing unit (CPU) 100 and an artificial intelligence chip 200. Among them, the CPU 100 and the artificial intelligence chip 200 can be connected by wire or wirelessly. The artificial intelligence chip 200 at least includes: a plurality of processing unit groups 201 and a plurality of video memories 202.

[0108] refer to Figure 2B The processing unit group 201 may include multiple programmable multiprocessors, such as a stream processor cluster (Stream Processors Cluster, SPC for short), each SPC includes at least one tensor calculation unit and at least one vector calculation unit. Among them, the tensor calculation unit is used to use matrix operators to perform matrix calculations, and common matrix operators such as matrix multiply accumulation (MMA for short) are not limited here. The vector calculation unit is used to use vector operators to perform vector calculations, and common vector operators such as scaling operators (Scale), mask operators (Mask), normalization operators (SoftMax), etc. are not limited here.

[0109] The video memory 202 may be a high bandwidth memory (HBM) type memory or other types of memory. The HBM type memory may include high bandwidth memory 2 (HBM2), high bandwidth memory 2E (HBM2E), high bandwidth memory 3 (HBM3), etc., which are not limited here.

[0110] It should be understood that the artificial intelligence chip 200 in the present application may include other structures in addition to the above-mentioned structure, and the present application does not make any specific limitation on this.

[0111] The artificial intelligence chip 200 can be a graphics processing unit (GPU), a general-purpose computing on graphics processing units (GPGPU), a domain specific architecture (DSA), etc., which is not limited here.

[0112] Combination Figure 2A The system architecture diagram shown can be based on Figure 2A The capacity of the video memory in the artificial intelligence chip shown in the figure can determine that the upper limit of the number of tasks that the artificial intelligence chip can execute simultaneously using a large language model is M2, where M2 is greater than 0. Figure 2A The artificial intelligence chip shown executes M2 tasks, and the CPU can set a second record for the artificial intelligence chip, wherein the second record can be used to record the execution status corresponding to the M2 tasks, wherein the execution status includes an executable state and an unexecutable state. If a task is in an executable state, the artificial intelligence chip can use a large language model to reason about the task; if a task is in an unexecutable state, the artificial intelligence chip does not need to use a large language model to reason about the task.

[0113] It should be understood that the data structure of the second record in the embodiment of the present application can be an array, a linked list, a stack, a hash table, etc., which is not limited here.

[0114] For example, Figure 2A The upper limit of the number of tasks that the artificial intelligence chip shown can execute simultaneously using the large language model is 4. The current artificial intelligence chip uses the large language model to execute a task. The prompt information corresponding to the task and the inference result output by the large language model are shown in Table 1.

[0115]

[0116] The process of the artificial intelligence chip executing task 1 can be referred to Figure 3 , the specific process is as follows:

[0117] The CPU may set a second record for the artificial intelligence chip, wherein the second record may record the execution statuses corresponding to the four tasks. Since the artificial intelligence chip only executes one task, the execution status of task 1 recorded in the second record 1 is "executable".

[0118] The AI ​​chip reads the second record 1 and determines that the execution status of Task 1 is "executable". Therefore, the AI ​​chip executes the Prefill stage, that is, generates the first Token "star" corresponding to Task 1 according to the prompt information of Task 1. The CPU determines that the first Token corresponding to Task 1 has been generated and the generated Token is not a terminator, and sets the execution status of Task 1 in the second record 1 to "executable", generating the second record 2.

[0119] The AI ​​chip reads the second record 2 and determines that the execution status of Task 1 is "executable". Therefore, the AI ​​chip executes the Decoder stage, that is, generates the second Token "Period" according to the prompt information of Task 1 and the generated Token. The CPU determines that the second Token corresponding to Task 1 has been generated and the generated Token is not a terminator, and sets the execution status of Task 1 in the second record 2 to "executable", generating the second record 3.

[0120] The AI ​​chip reads the second record 3 and determines that the execution status of Task 1 is "executable". Therefore, the AI ​​chip executes the Decoder stage, that is, generates the third Token "three" according to the prompt information of Task 1 and the generated Token. The CPU determines that the third Token corresponding to Task 1 has been generated and the generated Token is not a terminator, and sets the execution status of Task 1 in the second record 3 to "executable", generating the second record 4.

[0121] The artificial intelligence chip reads the second record 4 and determines that the execution status of task 1 is "executable". Therefore, the artificial intelligence chip executes the Decoder stage, that is, generates the fourth Token "END" according to the prompt information of task 1 and the generated Token.

[0122] Since the artificial intelligence chip generates the terminator "END" corresponding to Task 1, the reasoning process of the artificial intelligence chip for Task 1 using the large language model ends here.

[0123] For example, Figure 2A The upper limit of the number of tasks that the artificial intelligence chip shown can execute simultaneously using the large language model is 4. The current artificial intelligence chip uses the large language model to execute 5 tasks. The prompt information corresponding to the 5 tasks and the inference results output by the large language model are shown in Table 2.

[0124]

[0125] The process of the AI ​​chip executing Task 1 to Task 5 can be referred to Figure 4 , the specific process is as follows:

[0126] The CPU can set a second record for the artificial intelligence chip, wherein the second record can record the execution statuses corresponding to the four tasks. Since the artificial intelligence chip can perform reasoning on Task 1, Task 2, Task 3, and Task 4 simultaneously, the execution statuses of Task 1, Task 2, Task 3, and Task 4 recorded in the second record 1 are all "executable".

[0127] The artificial intelligence chip reads the second record 1 and determines that the execution status of Task 1, Task 2, Task 3, and Task 4 are all "executable". Therefore, the artificial intelligence chip executes the Prefill stage, that is, according to the prompt information of Task 1 "What day of the week is today", the first Token "Star" corresponding to Task 1 is generated; according to the prompt information of Task 2 "What is the temperature today", the first Token "32" corresponding to Task 2 is generated; according to the prompt information of Task 3 "Is today a holiday", the first Token "No" corresponding to Task 3 is generated; according to the prompt information of Task 4 "What is 1 plus 1 equal to?", the first Token "2" corresponding to Task 4 is generated.

[0128] Afterwards, the CPU determines that the first Token corresponding to Task 1 has been generated and the generated Token is a non-terminator, and sets the execution status of Task 1 in the second record 1 to "executable"; the CPU determines that the first Token corresponding to Task 2 has been generated and the generated Token is a non-terminator, and sets the execution status of Task 2 in the second record 1 to "executable"; the CPU determines that the first Token corresponding to Task 3 has been generated and the generated Token is a non-terminator, and sets the execution status of Task 3 in the second record 1 to "executable"; the CPU determines that the first Token corresponding to Task 4 has been generated and the generated Token is a non-terminator, and sets the execution status of Task 4 in the second record 1 to "executable". Based on the above modifications to the second record 1, the CPU can generate the second record 2.

[0129] The artificial intelligence chip reads the second record 2 and determines that the execution status of Task 1, Task 2, Task 3, and Task 4 are all "executable". Therefore, the artificial intelligence chip executes the Decoder stage, that is, according to the prompt information of Task 1 "What day is today" and the generated Token, the second Token "period" corresponding to Task 1 is generated; according to the prompt information of Task 2 "What is the temperature today" and the generated Token, the second Token "degree" corresponding to Task 2 is generated; according to the prompt information of Task 3 "Is today a holiday" and the generated Token, the second Token "yes" corresponding to Task 3 is generated; according to the prompt information of Task 4 "What is 1 plus 1 equal to" and the generated Token, the second Token "END" corresponding to Task 4 is generated.

[0130] Afterwards, when the CPU determines that the second Token corresponding to Task 4 has been generated and the generated Token is a terminator, it determines that there is a task 5 to be executed, so Task 4 in the second record 2 is replaced by Task 5, and the execution status of Task 5 in the second record 2 is set to "executable", and the execution status of Task 1, Task 2, and Task 3 in the second record 2 are all set to "non-executable". Based on the above modification of the second record 2, the CPU can generate the second record 3.

[0131] The artificial intelligence chip reads the second record 3 and determines that the execution status of Task 1, Task 2, and Task 3 are all "not executable", and the execution status of Task 5 is "executable". Therefore, the artificial intelligence chip executes the Prefill stage, that is, it generates the first Token "No" corresponding to Task 5 according to the prompt information of Task 5 "Will it rain this week?"

[0132] Afterwards, the CPU determines that the first Token corresponding to Task 5 has been generated and the generated Token is not a terminator, so the execution status of Task 5 in the second record 3 is set to "executable". In addition, the execution status of Task 1, Task 2, and Task 3 in the second record 3 can also be set to "executable". Based on the above modification of the second record 3, the second record 4 can be generated.

[0133] The artificial intelligence chip reads the second record 4 and determines that the execution status of Task 1, Task 2, Task 3, and Task 4 are all "executable". Therefore, the artificial intelligence chip executes the Decoder stage, that is, according to the prompt information of Task 1 "What day of the week is today" and the generated Token, the third Token "three" corresponding to Task 1 is generated; according to the prompt information of Task 2 "What is the temperature today" and the generated Token, the third Token "END" corresponding to Task 2 is generated; according to the prompt information of Task 3 "Is today a holiday" and the generated Token, the third Token "false" corresponding to Task 3 is generated; according to the prompt information of Task 5 "Will it rain this week" and the generated Token, the second Token "yes" corresponding to Task 5 is generated.

[0134] Afterwards, the CPU determines that the third Token corresponding to Task 1 has been generated and the generated Token is a non-terminator, and sets the execution status of Task 1 in the second record 4 to "executable"; when the CPU determines that the third Token corresponding to Task 2 has been generated and the generated Token is a terminator, it determines that there are no other tasks to be executed, so the execution status of Task 2 in the second record 4 is set to "non-executable"; the CPU determines that the third Token corresponding to Task 3 has been generated and the generated Token is a non-terminator, and sets the execution status of Task 3 in the second record 4 to "executable"; the CPU determines that the second Token corresponding to Task 5 has been generated and the generated Token is a non-terminator, and sets the execution status of Task 5 in the second record 4 to "executable". Based on the above modifications to the second record 4, the CPU can generate the second record 5.

[0135] The artificial intelligence chip reads the second record 5 and determines that the execution status of Task 1, Task 3, and Task 5 are all "executable", and the execution status of Task 2 is "not executable". Therefore, the artificial intelligence chip executes the Decoder stage, that is, according to the prompt information of Task 1 "What day of the week is today" and the generated Token, the fourth Token "END" corresponding to Task 1 is generated; according to the prompt information of Task 3 "Is today a holiday" and the generated Token, the fourth Token "period" corresponding to Task 3 is generated; according to the prompt information of Task 5 "Will it rain this week" and the generated Token, the third Token "END" corresponding to Task 5 is generated.

[0136] Afterwards, when the CPU determines that the fourth Token corresponding to Task 1 has been generated and the generated Token is a terminator, it determines that there are no other tasks to be executed, and sets the execution status of Task 1 in the second record 5 to "not executable"; when the CPU determines that the fourth Token corresponding to Task 3 has been generated and the generated Token is not a terminator, it sets the execution status of Task 3 in the second record 5 to "executable"; when the CPU determines that the third Token corresponding to Task 5 has been generated and the generated Token is a terminator, it determines that there are no other tasks to be executed, and sets the execution status of Task 5 in the second record 5 to "not executable". Based on the above modifications to the second record 5, the CPU can generate the second record 6.

[0137] The artificial intelligence chip reads the second record 6 and determines that the execution status of task 3 is "executable", and the execution status of other tasks is "not executable". Therefore, the artificial intelligence chip executes the Decoder stage, that is, it generates the fifth Token "END" corresponding to Task 3 according to the prompt information of Task 3 "Is today a holiday" and the generated Token.

[0138] Since the artificial intelligence chip generates the terminator "END" corresponding to task 3, the reasoning process of the artificial intelligence chip using the large language model for tasks 1 to 5 is completed.

[0139] In the above-mentioned Prefill stage or Decoder stage, the artificial intelligence chip can use the Multi-HeadAttention operator to perform M1 tasks respectively, where M1 is greater than 0 and M1 is less than or equal to M2. Figure 2A and 2B The system architecture diagram of the artificial intelligence chip shown in the figure, this application proposes an operator execution method, through which the operator execution method is used to introduce in detail how the artificial intelligence chip uses the Multi-Head Attention operator to execute M1 tasks respectively. The operator execution method may include the following: Figure 5 The following steps are shown:

[0140] S501: The first processing unit determines K1 subtasks to be executed, where K1 is greater than or equal to 0.

[0141] In the embodiment of the present application, the artificial intelligence chip includes S processing units, the first processing unit is any processing unit among the S processing units, and the processing unit can be Figure 2B The SPC in is not limited here.

[0142] In the embodiment of the present application, since the Multi-Head Attention operator includes N Attention operators, using the Multi-Head Attention operator to execute a task can be decomposed into using the Attention operator to execute N subtasks to be executed. Among them, the subtask to be executed is a partial task of executing a task using the Attention operator.

[0143] After acquiring M1 tasks, the artificial intelligence chip can decompose the M1 tasks into M1*N subtasks to be executed, and the S processing units in the artificial intelligence chip execute the M1*N subtasks to be executed. The number of subtasks to be executed by each processing unit can be the same or different, which is not limited here. Any subtask to be executed is executed by only one processing unit.

[0144] Taking the first processing unit as an example, the first processing unit can determine K1 subtasks to be executed from M1*N subtasks to be executed. The method for the first processing unit to determine K1 subtasks to be executed will be described in detail later, and will not be repeated here.

[0145] S502: The first processing unit uses the Attention operator to execute K1 subtasks to be executed respectively.

[0146] In an embodiment of the present application, S processing units can use the Attention operator in parallel to respectively execute the determined subtasks to be executed.

[0147] Taking the first processing unit as an example, after the first processing unit determines K1 subtasks to be executed, the first processing unit can use the Attention operator to execute the K1 subtasks to be executed in sequence and obtain the execution results of the K1 subtasks to be executed.

[0148] exist Figure 5 In the steps shown, when the artificial intelligence chip uses the Multi-Head Attention operator to execute M1 tasks, it is not that the M1 processing unit in the artificial intelligence chip uses the Multi-Head Attention operator to execute one task respectively, but that the M1 task is split into multiple subtasks to be executed, and S processing units determine the subtasks to be executed from the multiple subtasks to be executed. After that, the S processing units use the Attention operator to execute the subtasks to be executed respectively, and finally achieve the purpose of the artificial intelligence chip to quickly execute and complete the M1 tasks. In the above process, multiple subtasks to be executed belonging to the same task can be assigned to different processing units, and different processing units use the Attention operator to execute the subtasks to be executed in parallel. Therefore, the time overhead caused by the processing unit using the Multi-Head Attention operator to execute tasks can be effectively reduced.

[0149] Optionally, in the above S501, the first processing unit may determine K1 subtasks to be executed through implementation mode A1 or A2.

[0150] In embodiment A1, the first processing unit can be Figure 6 The following steps shown determine K1 subtasks to be executed.

[0151] S601, a first processing unit determines K2 executable subtasks, where K2 is greater than or equal to K1.

[0152] In an embodiment of the present application, the upper limit of the number of tasks that can be executed simultaneously by the artificial intelligence chip using a large language model is M2, and the M2 tasks can be decomposed into M2*N executable subtasks, where the executable subtask is a partial task of one of the M2 tasks using the Attention operator.

[0153] After determining the M2*N executable subtasks, the M2*N executable subtasks can be allocated to the S processing units in the artificial intelligence chip. The number of executable subtasks allocated to each processing unit can be the same or different, which is not limited here. Any executable subtask is only executed by one processing unit.

[0154] In a possible implementation, M2*N executable subtasks are sequentially assigned to S processing units. The upper limit of the number of executable subtasks assigned to each processing unit is ,in, Indicates rounding up. Taking the first processing unit among S processing units as an example, the first processing unit is assigned K2 executable subtasks, and K2 is less than or equal to .

[0155] For example, an AI chip includes 8 SPCs. The upper limit of the number of tasks that can be executed simultaneously by the AI ​​chip using a large language model is 4. The Multi-Head Attention operator includes 5 Attention operators, so the 4 tasks can be decomposed into 4*5 executable subtasks. Figure 7 The allocation method shown is as follows:

[0156] Task 1 is decomposed into 5 executable subtasks, namely T1H1, T1H2, T1H3, T1H4, and T1H5. T1H1, T1H2, T1H3, T1H4, and T1H5 are assigned to SPC1, SPC2, SPC3, SPC4, and SPC5 respectively.

[0157] Task 2 is decomposed into 5 executable subtasks, namely T2H1, T2H2, T2H3, T2H4, and T2H5. T2H1, T2H2, T2H3, T2H4, and T2H5 are assigned to SPC6, SPC7, SPC8, SPC1, and SPC2 respectively.

[0158] Task 3 is decomposed into 5 executable subtasks, namely T3H1, T3H2, T3H3, T3H4, and T3H5. T3H1, T3H2, T3H3, T3H4, and T3H5 are assigned to SPC3, SPC4, SPC5, SPC6, and SPC7 respectively.

[0159] Task 4 is decomposed into 5 executable subtasks, namely T4H1, T4H2, T4H3, T4H4, and T4H5. T4H1, T4H2, T4H3, T4H4, and T4H5 are assigned to SPC8, SPC1, SPC2, SPC3, and SPC4 respectively.

[0160] In summary, the executable subtasks assigned by SPC1 include T1H1, T2H4, and T4H2, the executable subtasks assigned by SPC2 include T1H2, T2H5, and T4H3, the executable subtasks assigned by SPC3 include T1H3, T3H1, and T4H4, the executable subtasks assigned by SPC4 include T1H4, T3H2, and T4H5, the executable subtasks assigned by SPC5 include T1H5 and T3H3, the executable subtasks assigned by SPC6 include T2H1 and T3H4, the executable subtasks assigned by SPC7 include T2H2 and T3H5, and the executable subtasks assigned by SPC8 include T2H3 and T4H1.

[0161] In order to facilitate each processing unit to more conveniently determine its corresponding executable subtask, each processing unit can set Reference identifiers. Set the reference identifier corresponding to each processing unit to i, where i is the number of references. Each processing unit can be represented by How the processing units determine the executable subtasks corresponding to each other through the reference identifiers will be described in detail later, and will not be described here in detail.

[0162] S602: The first processing unit obtains a second record.

[0163] In the embodiment of the present application, the second record can refer to the above description and will not be repeated here.

[0164] S603: The first processing unit determines K1 subtasks to be executed according to the K2 executable subtasks and the second record.

[0165] In the above S603, the K2 executable subtasks include a first executable subtask, and the first executable subtask is any executable subtask among the K2 executable subtasks.

[0166] For the first executable subtask, you can use Figure 8 The following steps shown determine whether the first executable subtask is a subtask to be executed.

[0167] S801: A first processing unit determines a second task corresponding to a first executable subtask from M2 tasks.

[0168] S802: The first processing unit determines an execution status of the second task according to the second record.

[0169] S803: When the execution state of the second task is in the executable state, the first processing unit takes the first executable subtask as the subtask to be executed.

[0170] In the embodiment of the present application, when the execution status of the second task is in an unexecutable state, the first executable subtask is not used as a to-be-executed subtask.

[0171] For example, the executable subtasks assigned to each processing unit refer to Figure 7 , the second record is shown in Table 3.

[0172]

[0173] Taking SPC1 as an example, SPC1 can determine, according to the reference identifier i, that the allocated executable subtasks include T1H1, T2H4, and T4H2.

[0174] SPC1 determines that the executable subtask T1H1 corresponds to Task 1. Through the second record shown in Table 3, it can be determined that the execution status of Task 1 is executable. Therefore, the executable subtask T1H1 is a to-be-executed subtask of SPC1.

[0175] SPC1 determines that the executable subtask T2H4 corresponds to Task 2. Through the second record shown in Table 3, it can be determined that the execution status of Task 2 is a non-executable state. Therefore, the executable subtask T2H4 is not a to-be-executed subtask of SPC1.

[0176] SPC1 determines that the executable subtask T4H2 corresponds to Task 4. Through the second record shown in Table 3, it can be determined that the execution status of Task 4 is a non-executable state. Therefore, the executable subtask T4H2 is not a to-be-executed subtask of SPC1.

[0177] In summary, SPC1 can determine that the subtasks to be executed include T1H1.

[0178] The methods for the remaining SPCs to determine their corresponding subtasks to be executed can refer to the method for SPC1 to determine the subtasks to be executed, which will not be described in detail here.

[0179] Through the above method, the subtasks to be executed determined by each SPC are as follows: Fig. 9 shown.

[0180] For example, the executable subtasks assigned to each processing unit refer to Figure 7 , the second record is shown in Table 4.

[0181]

[0182] Taking SPC1 as an example, SPC1 can determine, according to the reference identifier i, that the allocated executable subtasks include T1H1, T2H4, and T4H2.

[0183] SPC1 determines that the executable subtask T1H1 corresponds to Task 1. Through the second record shown in Table 4, it can be determined that the execution status of Task 1 is a non-executable status. Therefore, the executable subtask T1H1 is not a to-be-executed subtask of SPC1.

[0184] SPC1 determines that the executable subtask T2H4 corresponds to Task 2. Through the second record shown in Table 4, it can be determined that the execution status of Task 2 is executable. Therefore, the executable subtask T2H4 is a to-be-executed subtask of SPC1.

[0185] SPC1 determines that the executable subtask T4H2 corresponds to Task 4. Through the second record shown in Table 4, it can be determined that the execution status of Task 4 is a non-executable state. Therefore, the executable subtask T4H2 is not a to-be-executed subtask of SPC1.

[0186] In summary, SPC1 can determine that the subtasks to be executed include T2H4.

[0187] The methods for the remaining SPCs to determine their corresponding subtasks to be executed can refer to the method for SPC1 to determine the subtasks to be executed, which will not be described in detail here.

[0188] Through the above method, the subtasks to be executed determined by each SPC are as follows: Fig.10 shown.

[0189] exist Figure 8 In the steps shown, the first processing unit can first determine the second task corresponding to the first executable subtask, determine the execution status of the second task by accessing the second record, and when the execution status of the second task is executable, determine the first executable subtask as a subtask to be executed. Since the first processing unit queries the second record for each executable subtask and determines whether each executable subtask is a subtask to be executed through the second record, the accuracy and completeness of the determined subtasks to be executed can be guaranteed.

[0190] pass Figure 6 In the steps shown, the first processing unit can accurately determine K1 subtasks to be executed according to K2 executable subtasks and the second record.

[0191] Optionally, in the above S601, the first processing unit may determine K2 executable subtasks according to the reference identifier i, the number of processing units S included in the artificial intelligence chip, the position identifier of the first processing unit in the S processing units, and the number of Attention operators N. The specific implementation method for determining K2 executable subtasks may refer to implementation method B1 or B2.

[0192] In embodiment B1, the first processing unit may be Fig.11The following steps shown determine the executable subtask corresponding to the reference identifier i.

[0193] S1101, the first processing unit uses the product of the reference identifier i and the number of processing units S as a third reference value.

[0194] S1102: The first processing unit uses the sum of the third reference value and the position identifier of the first processing unit in the S processing units as a fourth reference value.

[0195] For example, refer to Figure 7 As shown, the position of SPC1 among the 8 SPCs is marked as 1.

[0196] S1103: The first processing unit determines an executable subtask corresponding to the reference identifier i according to the fourth reference value and the number N of Attention operators.

[0197] In the above S1103, the first processing unit can Fig.12 The following steps shown determine the executable subtask corresponding to the reference identifier i.

[0198] S1201: The first processing unit determines a third task from M2 tasks according to a quotient of a fourth reference value and the number N of Attention operators.

[0199] In an embodiment of the present application, the first processing unit may use the quotient of the fourth reference value and the number of Attention operators N as the fifth reference value, the sum of the fifth reference value and 1 as the sixth reference value, and select the task corresponding to the sixth reference value from M2 tasks as the third task.

[0200] For example, if the sixth reference value is 1, the first task is selected from the M2 tasks as the third task.

[0201] S1202: The first processing unit determines a second operator from N Attention operators according to a remainder of a fourth reference value and the number N of Attention operators.

[0202] In an embodiment of the present application, the first processing unit may use the remainder of the fourth reference value and the number of Attention operators N as the seventh reference value, and select an operator corresponding to the seventh reference value from the N Attention operators as the second operator.

[0203] For example, if the seventh reference value is 1, the first operator is selected from the N Attention operators as the second operator.

[0204] S1203: The first processing unit determines an executable subtask corresponding to the reference identifier i according to the third task and the second operator.

[0205] In the embodiment of the present application, the partial task of the third task executed by the first processing unit using the second operator is the executable subtask corresponding to the reference identifier i.

[0206] Through the above Fig.11 In the steps shown, the first processing unit can determine the executable subtask corresponding to the reference identifier i by referring to the identifier i, and there is no need to store the identifier of the executable subtask allocated by the first processing unit, which can save video memory capacity.

[0207] In implementation mode B2, the first processing unit may determine the executable subtask corresponding to the reference identifier i by using formula (1) and formula (2).

[0208] (1)

[0209] (2)

[0210] in, The identifier of the third task, It represents the identifier of the second operator, i represents the reference identifier, S represents the number of processing units in the artificial intelligence chip, index represents the position identifier of the first processing unit in the S processing units, and N represents the number of Attention operators.

[0211] In an embodiment of the present application, the first processing unit can determine the executable subtask corresponding to the reference identifier i through the above formulas (1) and (2), without storing the identifier of the executable subtask assigned by the first processing unit, thereby saving video memory capacity.

[0212] For example, refer to Figure 7 , the reference identifier i corresponding to SPC1 is 0, 1, 2. When the reference identifier i is 0, the formula (1) and formula (2) are used to calculate and determine , Therefore, the first executable subtask assigned by SPC1 is T1H1. When the reference identifier i is 1, the formula (1) and formula (2) are used to calculate and determine , Therefore, the second executable subtask assigned by SPC1 is T2H4. When the reference identifier i is 2, the formula (1) and formula (2) are used to calculate and determine , Therefore, the third executable subtask assigned by SPC1 is T4H2.

[0213] In embodiment A2, the first processing unit can be Fig.13 The following steps shown determine K1 subtasks to be executed.

[0214] S1301: The first processing unit obtains K1 identification records from the first record corresponding to the first processing unit.

[0215] In the embodiment of the present application, the S processing units each correspond to a respective first record, and the first records corresponding to the S processing units are each generated by a CPU.

[0216] Taking the first record corresponding to the first processing unit as an example, the first record corresponding to the first processing unit is determined based on the K2 executable subtasks corresponding to the first processing unit and the second record. The first record corresponding to the first processing unit can store K1 reference identifiers corresponding to the first processing unit, and the K1 subtasks to be executed to be processed by the first processing unit can be determined through the K1 reference identifiers corresponding to the first processing unit. The recording order of the K1 reference identifiers corresponding to the first processing unit in the first record can be used to determine the execution order of the K1 subtasks to be executed to be processed by the first processing unit.

[0217] It should be understood that the data structure of the first record in the embodiment of the present application can be an array, a linked list, a stack, a hash table, etc., which is not limited here.

[0218] The first record corresponding to the first processing unit in S1301 can be obtained by Fig.14 The following steps are shown to determine.

[0219] S1401: The CPU determines, from M2 tasks, a second task corresponding to a first executable subtask assigned by a first processing unit.

[0220] In the embodiment of the present application, the first processing unit is assigned K2 executable subtasks, wherein the first executable subtask is an executable subtask among the K2 executable subtasks.

[0221] S1402: The CPU determines an execution state of the second task according to the second record.

[0222] S1403: When the execution state of the second task is in an executable state, the CPU adds a reference identifier of the first executable subtask to a first record corresponding to the first processing unit.

[0223] By repeatedly executing the above steps S1401 to S1403 and judging the K2 executable subtasks allocated by the first processing unit, a first record including K1 reference identifiers can be obtained.

[0224] Optionally, the CPU may also determine the number K1 of reference identifiers that have been added to the first record, and add the number K1 of reference identifiers to the first record corresponding to the first processing unit.

[0225] exist Fig.14In the steps shown, the CPU can quickly determine the first record corresponding to each processing unit, so that each processing unit can subsequently execute the subtask to be executed according to the first record corresponding to each processing unit.

[0226] For example, the executable subtasks assigned to each processing unit refer to Figure 7 , the second record is shown in Table 3.

[0227] Taking SPC1 as an example, the CPU can determine, based on the reference identifier i, that the executable subtasks allocated by SPC1 include T1H1, T2H4, and T4H2.

[0228] The CPU determines that the executable subtask T1H1 corresponds to Task 1. The second record shown in Table 3 can determine that the execution state of Task 1 is executable. Therefore, the CPU adds the reference identifier 0 of the executable subtask T1H1 to the first record corresponding to SPC1.

[0229] The CPU determines that the executable subtask T2H4 corresponds to Task 2. The second record shown in Table 3 can determine that the execution status of Task 2 is an unexecutable state. Therefore, the CPU does not need to add the reference identifier 1 of the executable subtask T2H4 to the first record corresponding to SPC1.

[0230] The CPU determines that the executable subtask T4H2 corresponds to task 4. The second record shown in Table 3 can determine that the execution status of task 4 is a non-executable state. Therefore, the CPU does not need to add the reference identifier 2 of the executable subtask T4H2 to the first record corresponding to SPC1.

[0231] The CPU determines that the number of reference identifiers added to the first record is 1, so it adds 1 to the first record corresponding to SPC1.

[0232] In summary, the CPU can determine that the first record corresponding to SPC1 is as shown in Table 5, where 1 in Table 5 represents the number of times the first record corresponding to SPC1 can be accessed, and 0 in Table 5 represents the reference identifier of the subtask to be executed of SPC1, wherein the reference identifier 0 in the first record can be used to determine the subtask T1H1 to be executed of SPC1.

[0233]

[0234] The method for other SPCs to determine their respective corresponding first records can refer to the method for SPC1 to determine the first record, which will not be described in detail here.

[0235] In the above S1301, the first processing unit can Fig.15The following steps shown are to obtain K1 identification records from the first record corresponding to the first processing unit, wherein the identification record in the first record is the reference identification in the first record mentioned above.

[0236] S1501: The first processing unit obtains the accessible count K1 from the first record corresponding to the first processing unit.

[0237] S1502: The first processing unit accesses the first record K1 times according to the accessible number K1, and sequentially obtains K1 identification records.

[0238] exist Fig.15 In the steps shown, the first processing unit accesses the first record K1 times according to the accessible number K1, thereby sequentially obtaining K1 identification records, which can avoid the first processing unit from reading the first record unnecessarily. In addition, when the number of subtasks to be executed to be processed by the first processing unit is much smaller than the number of executable subtasks assigned to the first processing unit, the number of times the first processing unit accesses the first record will be much smaller than the number of times the first processing unit accesses the second record, which is convenient for improving the efficiency of the first processing unit in determining the subtasks to be executed.

[0239] S1302: The first processing unit determines K1 subtasks to be executed according to the K1 identification records.

[0240] Optionally, in the above S1302, the K1 identification records include a first identification record, and the first identification record is any identification record among the K1 identification records.

[0241] For the first identification record, the first processing unit can determine the first subtask to be executed among the K1 subtasks to be executed based on the first identification record, the number S of processing units included in the artificial intelligence chip, the position identification of the first processing unit in the S processing units, and the number N of Attention operators. The specific implementation method for determining the first subtask to be executed can refer to implementation method C1 or C2.

[0242] In embodiment C1, the first processing unit can be Fig.16 The following steps shown determine the first subtask to be performed.

[0243] S1601, the first processing unit uses the product of the first identification record and the number of processing units S as a first reference value.

[0244] S1602: The first processing unit uses the sum of the first reference value and the position identifier of the first processing unit in the S processing units as the second reference value.

[0245] For example, refer to Figure 7 As shown, the position of SPC1 among the 8 SPCs is marked as 1.

[0246] S1603: The first processing unit determines a first subtask to be executed according to the second reference value and the number N of Attention operators.

[0247] In the above S1603, the first processing unit can Fig.17 The following steps shown determine the first subtask to be performed.

[0248] S1701: The first processing unit determines a first task from M1 tasks according to a quotient of a second reference value and the number N of Attention operators.

[0249] In an embodiment of the present application, the first processing unit may use the quotient of the second reference value and the number of Attention operators N as the eighth reference value, the sum of the eighth reference value and 1 as the ninth reference value, and select the task corresponding to the ninth reference value from M1 tasks as the first task.

[0250] For example, if the ninth reference value is 1, the first task is selected from the M1 tasks as the first task.

[0251] S1702: The first processing unit determines a first operator from N Attention operators according to a remainder of a second reference value and the number N of Attention operators.

[0252] In an embodiment of the present application, the first processing unit may use the remainder of the second reference value and the number of Attention operators N as the tenth reference value, and select an operator corresponding to the tenth reference value from the N Attention operators as the first operator.

[0253] For example, if the tenth reference value is 1, the first operator is selected from the N Attention operators as the first operator.

[0254] S1703: The first processing unit determines a first subtask to be executed according to the first task and the first operator.

[0255] In the embodiment of the present application, the first processing unit uses the first operator to execute a portion of the first task, that is, the first subtask to be executed.

[0256] Through the above Fig.16 In the steps shown, the first processing unit can quickly determine the first to-be-executed subtask corresponding to the first identification record through the first identification record.

[0257] In implementation C2, the first processing unit may determine the first to-be-executed subtask corresponding to the first identification record by using formula (3) and formula (4).

[0258] (3)

[0259] (4)

[0260] in, The identifier of the first task, Represents the identifier of the first operator, k represents the first identifier record, S represents the number of processing units in the artificial intelligence chip, index represents the position identifier of the first processing unit in the S processing units, and N represents the number of Attention operators.

[0261] In an embodiment of the present application, the first processing unit can determine the first to-be-executed subtask corresponding to the first identification record by using the above formulas (3) and (4), without storing the identification of the to-be-executed subtask to be processed by the first processing unit, thereby saving video memory capacity.

[0262] For example, the first record corresponding to SPC1 refers to Table 5. SPC1 obtains the number of accessible times 1 from the first record in Table 5 and determines to access the first record once. After that, SPC1 accesses the first record and obtains the first identification record 0 from the first record. The calculation is determined by formula (3) and formula (4). , , therefore, SPC1 determines that the subtask to be executed is T1H1.

[0263] exist Fig.13 In the steps shown, the first processing unit can obtain K1 identification records from the first record corresponding to the first processing unit, and determine K1 to-be-executed subtasks to be processed by the first processing unit based on the K1 identification records, which can effectively avoid the first processing unit from accessing the second record unnecessarily.

[0264] Optionally, in the above S502, one of the structures of the Attention operator is as follows: Fig.18 As shown, the Attention operator includes a first matrix operator, a scaling operator, a mask operator, a normalization operator, and a second matrix operator. The first matrix operator and the second matrix operator are both matrix operators (MatMul), and the matrix operator may include at least one matrix multiply accumulation operation (Matrix Multiply Accumulate, MMA for short). The scaling operator, the mask operator, and the normalization operator are all vector operators.

[0265] The specific calculation process of the Attention operator can refer to the following steps: the tensor calculation unit first uses the first matrix operator to perform matrix calculation on the query vector Q and the key vector K to obtain the correlation between the query vector Q and the key vector K. The vector calculation unit then uses the scaling operator, mask operator and normalization operator in sequence to perform vector calculation on the correlation between the query vector Q and the key vector K to obtain the weight. Finally, the tensor calculation unit uses the second matrix operator to perform matrix calculation on the weight and value vector V to obtain the final output vector.

[0266] In the above S502, K1 subtasks to be executed can be executed through implementation mode D1 or D2.

[0267] In implementation mode D1, when the first processing unit adopts implementation mode A1 to determine K1 subtasks to be executed, that is, the K1 subtasks to be executed to be processed by the first processing unit are determined according to K2 executable tasks and the second record, the first processing unit can Fig.19 The following steps shown execute K1 subtasks to be executed.

[0268] S1901, the first processing unit determines, based on the K2 executable subtasks, that the vector computing unit in the first processing unit executes a first sequence of the K2 executable subtasks, and that the tensor computing unit in the first processing unit executes a second sequence of the K2 executable subtasks.

[0269] In an embodiment of the present application, the first sequence and the second sequence may refer to the following: when the vector calculation unit in the first processing unit uses a vector operator to perform vector calculation on the xth executable subtask, the tensor calculation unit in the first processing unit uses a first matrix operator to perform matrix calculation on the x+ith executable subtask, or the tensor calculation unit in the first processing unit uses a second matrix operator to perform matrix calculation on the xjth executable subtask; x is an integer ranging from [1, K2], and i and j are both integers greater than 0.

[0270] S1902: The first processing unit uses the Attention operator to execute K1 subtasks to be executed respectively according to the first sequence and the second sequence.

[0271] Specifically, the vector calculation unit in the first processing unit performs vector calculation on K1 subtasks to be executed according to the first sequence using vector operators; the tensor calculation unit in the first processing unit performs tensor calculation on K1 subtasks to be executed according to the second sequence using the first matrix operator or the second matrix operator.

[0272] For example, the executable subtasks assigned to each processing unit refer to Figure 7,Take SPC1 as an example, the executable subtasks assigned to SPC1 include T1H1, T2H4, and T4H2.

[0273] Assume that the computation time of the vector operator is less than or equal to the computation time of the first matrix operator (or the second matrix operator). The Attention operator performs computations on the three executable subtasks in sequence. For the specific computation process, please refer to Fig. 20 Steps shown.

[0274] The tensor calculation unit uses the first matrix operator to perform matrix calculation on the query vector Q and the key vector K of the executable subtask T1H1 at time t0, and obtains the correlation between the query vector Q and the key vector K of the executable subtask T1H1 at time t1.

[0275] The vector calculation unit uses a vector operator to perform vector calculation on the relevance corresponding to the executable subtask T1H1 at time t1, and obtains the weight corresponding to the executable subtask T1H1 at time t2.

[0276] When the vector calculation unit uses the vector operator to perform vector calculation on the correlation corresponding to the executable subtask T1H1, the tensor calculation unit uses the first matrix operator to perform matrix calculation on the query vector Q and the key vector K of the executable subtask T2H4 at time t1, and obtains the correlation between the query vector Q and the key vector K of the executable subtask T2H4 at time t3. And so on, no further description is given here.

[0277] When the pending subtasks to be processed by SPC1 include T1H1 and T4H2, SPC1 can refer to Fig.21 Subtasks T1H1 and T4H2 are to be executed. The detailed execution process is not described here.

[0278] Fig.19 The steps shown provide a method for executing K1 to-be-executed subtasks in parallel in a certain process.

[0279] In implementation mode D2, when the first processing unit determines K1 subtasks to be executed by using implementation mode A2, that is, the K1 subtasks to be executed to be processed by the first processing unit are determined according to the K1 identification records in the first record corresponding to the first processing unit, the first processing unit can execute the K1 subtasks to be executed in the following manner:

[0280] When the vector calculation unit in the first processing unit uses the vector operator to perform vector calculation on the zth subtask to be executed, the tensor calculation unit in the first processing unit uses the first matrix operator to perform matrix calculation on the z+ith subtask to be executed, or the tensor calculation unit in the first processing unit uses the second matrix operator to perform matrix calculation on the zjth subtask to be executed; z is an integer that spans [1,K1], and i and j are both integers greater than 0.

[0281] For example, the subtasks to be processed by SPC1 include T1H1 and T4H2. Assume that the calculation time of the vector operator is less than or equal to the calculation time of the first matrix operator (or the second matrix operator). The Attention operator calculates the two subtasks to be executed in turn. The specific calculation process can be referred to Fig. 22 Steps shown.

[0282] The tensor calculation unit uses the first matrix operator to perform matrix calculation on the query vector Q and the key vector K of the subtask T1H1 to be executed at time t0, and obtains the correlation between the query vector Q and the key vector K of the subtask T1H1 to be executed at time t1.

[0283] The vector calculation unit uses a vector operator to perform vector calculation on the relevance corresponding to the subtask T1H1 to be executed at time t1, and obtains the weight corresponding to the subtask T1H1 to be executed at time t2.

[0284] When the vector calculation unit uses the vector operator to perform vector calculation on the correlation corresponding to the to-be-executed subtask T1H1, the tensor calculation unit uses the first matrix operator to perform matrix calculation on the query vector Q and the key vector K of the to-be-executed subtask T4H2 at time t1, and obtains the correlation between the query vector Q and the key vector K of the to-be-executed subtask T4H2 at time t3. And so on, no further description is given here.

[0285] In the above implementation D2, the tensor calculation unit and the vector calculation unit in the first processing unit can execute K1 subtasks to be executed in parallel.

[0286] Based on the same technical concept, the present application embodiment provides a structural diagram of an operator execution device, such as Fig.23 As shown, the operator execution device 2300 includes:

[0287] Determining module 2301, configured for a first processing unit to determine K1 to-be-executed subtasks, where the first processing unit is any processing unit among the S processing units, any to-be-executed subtask among the K1 to-be-executed subtasks is a partial task of executing one of the M1 tasks using an Attention operator, and K1 is greater than or equal to 0;

[0288] The execution module 2302 is used for the first processing unit to respectively execute the K1 subtasks to be executed by using the Attention operator.

[0289] Optionally, the determination module 2301 is specifically configured to:

[0290] The first processing unit obtains K1 identification records from the first record corresponding to the first processing unit, where the first record corresponding to the first processing unit is used to record the execution order of the K1 to-be-executed subtasks to be processed by the first processing unit;

[0291] The first processing unit determines the K1 subtasks to be executed according to the K1 identification records.

[0292] Optionally, the first record corresponding to the first processing unit is determined based on K2 executable subtasks corresponding to the first processing unit and a second record, wherein any executable subtask of the K2 executable subtasks is a partial task of executing one of the M2 tasks using the Attention operator assigned to the first processing unit, and the second record is used to record the execution status corresponding to the M2 tasks respectively, and K2 is greater than or equal to K1.

[0293] Optionally, the first record further includes a number of times K1 of accessibility of the first record; the determination module 2301 is specifically configured to:

[0294] The first processing unit obtains the accessible number K1 from the first record corresponding to the first processing unit;

[0295] The first processing unit accesses the first record K1 times according to the accessible number K1, and sequentially obtains the K1 identification records.

[0296] Optionally, the K1 identification records include a first identification record, and the first identification record is any identification record among the K1 identification records; the determination module 2301 is specifically configured to:

[0297] The first processing unit determines the first subtask to be executed among the K1 subtasks to be executed based on the first identification record, the number S of processing units included in the artificial intelligence chip, the position identifier of the first processing unit in the S processing units, and the number N of Attention operators.

[0298] Optionally, the determination module 2301 is specifically configured to:

[0299] The first processing unit uses the product of the first identification record and the number S of processing units as a first reference value;

[0300] The first processing unit uses the sum of the first reference value and the position identifier of the first processing unit in the S processing units as the second reference value;

[0301] The first processing unit determines the first subtask to be executed according to the second reference value and the number N of Attention operators.

[0302] Optionally, the determination module 2301 is specifically configured to:

[0303] The first processing unit determines the first task from the M1 tasks according to the quotient of the second reference value and the number N of the Attention operators;

[0304] The first processing unit determines a first operator from the N Attention operators according to a remainder of the second reference value and the number N of the Attention operators;

[0305] The first processing unit determines the first subtask to be executed according to the first task and the first operator.

[0306] Optionally, the determination module 2301 is specifically configured to:

[0307] The first processing unit determines K2 executable subtasks, any executable subtask of the K2 executable subtasks is a partial task of executing one of the M2 tasks assigned to the first processing unit using the Attention operator; K2 is greater than or equal to K1; M2 is greater than or equal to M1;

[0308] The first processing unit obtains a second record, where the second record is used to record the execution states corresponding to the M2 tasks respectively;

[0309] The first processing unit determines the K1 to-be-executed subtasks according to the K2 executable subtasks and the second record.

[0310] Optionally, the K2 executable subtasks include a first executable subtask, and the first executable subtask is any executable subtask among the K2 executable subtasks; the determination module 2301 is specifically used to:

[0311] The first processing unit determines, from the M2 tasks, a second task corresponding to the first executable subtask;

[0312] The first processing unit determines the execution status of the second task according to the second record;

[0313] When the execution state of the second task is in an executable state, the first processing unit uses the first executable subtask as a to-be-executed subtask.

[0314] Optionally, any processing unit among the S processing units includes a tensor calculation unit and a vector calculation unit; the Attention operator includes a first matrix operator, a vector operator and a second matrix operator; the execution module 2302 is specifically used to:

[0315] When the vector calculation unit in the first processing unit uses the vector operator to perform vector calculation on the zth subtask to be executed, the tensor calculation unit in the first processing unit uses the first matrix operator to perform matrix calculation on the z+ith subtask to be executed, or the tensor calculation unit in the first processing unit uses the second matrix operator to perform matrix calculation on the zjth subtask to be executed; z is an integer that spans [1,K1], and i and j are both integers greater than 0.

[0316] Based on the same technical concept, the embodiment of the present application provides a computer device, such as Fig.24 As shown, at least one artificial intelligence chip 2401, a memory 2402, and a CPU 2403 are included. The artificial intelligence chip 2401, the memory 2402, and the CPU 2403 are connected. The specific connection medium between the artificial intelligence chip 2401, the memory 2402, and the CPU 2403 is not limited in the embodiment of the present application. Fig.24 For example, the artificial intelligence chip 2401 and the memory 2402 are connected via a bus. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0317] In the embodiment of the present application, the memory 2402 stores instructions that can be executed by at least one artificial intelligence chip 2401, and at least one artificial intelligence chip 2401 can execute the steps of the above-mentioned operator execution method by executing the instructions stored in the memory 2402. The memory 2402 stores instructions that can be executed by the CPU 2403, and the CPU 2403 can execute the steps of the above-mentioned operator execution method by executing the instructions stored in the memory 2402.

[0318] Among them, the artificial intelligence chip 2401 is the control center of the computer device, which can use various interfaces and lines to connect various parts of the computer device, and realize operator execution by running or executing instructions stored in the memory 2402 and calling data stored in the memory 2402.

[0319] Optionally, the artificial intelligence chip 2401 may include one or more processing units, and the artificial intelligence chip 2401 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the artificial intelligence chip 2401. In some embodiments, the artificial intelligence chip 2401 and the memory 2402 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on separate chips.

[0320] The artificial intelligence chip 2401 can be a general-purpose processor, such as a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware processor for execution, or can be executed by a combination of hardware and software modules in the processor.

[0321] The memory 2402 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 2402 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (Random Access Memory, RAM), static random access memory (Static Random Access Memory, SRAM), programmable read-only memory (Programmable Read Only Memory, PROM), read-only memory (Read Only Memory, ROM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), magnetic memory, disk, optical disk, etc. The memory 2402 is any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer device, but is not limited thereto. The memory 2402 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, used to store program instructions and / or data.

[0322] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program executable by a computer device. When the computer program runs on the computer device, the computer device executes the steps of the above-mentioned operator execution method.

[0323] Based on the same inventive concept, an embodiment of the present application provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer device, the computer device executes the steps of the above-mentioned operator execution method.

[0324] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0325] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0326] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0327] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0328] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. An operator execution method, characterized in that: Applied to an artificial intelligence chip including S processing units, the artificial intelligence chip is used to perform M1 tasks using a multi-head attention mechanism operator, the multi-head attention mechanism operator including N attention mechanism Attention operators; S, M1, and N are all greater than 0, including: A first processing unit determines K1 subtasks to be executed, where the first processing unit is any processing unit among the S processing units, and any subtask to be executed among the K1 subtasks to be executed is a partial task of executing one of the M1 tasks using the Attention operator, and K1 is greater than or equal to 0; The first processing unit uses the Attention operator to respectively execute the K1 subtasks to be executed; The first processing unit determines K1 subtasks to be executed, including: The first processing unit obtains K1 identification records from the first record corresponding to the first processing unit, where the first record corresponding to the first processing unit is used to record the execution order of the K1 to-be-executed subtasks to be processed by the first processing unit; the K1 identification records include the first identification record, and the first identification record is any identification record among the K1 identification records; The first processing unit uses the product of the first identification record and the number S of processing units as a first reference value; The first processing unit uses the sum of the first reference value and the position identifier of the first processing unit in the S processing units as the second reference value; The first processing unit determines a first subtask to be executed according to the second reference value and the number N of Attention operators.

2. The method according to claim 1, characterized in that The first record corresponding to the first processing unit is determined based on the K2 executable subtasks corresponding to the first processing unit and the second record, wherein any executable subtask of the K2 executable subtasks is a partial task of using the Attention operator to execute one of the M2 tasks assigned to the first processing unit, and the second record is used to record the execution status corresponding to the M2 tasks respectively; M2 is greater than or equal to M1, and K2 is greater than or equal to K1.

3. The method according to claim 1, characterized in that The first record also includes the number of times K1 the first record can be accessed; The first processing unit obtains K1 identification records from the first record corresponding to the first processing unit, including: The first processing unit obtains the accessible number K1 from the first record corresponding to the first processing unit; The first processing unit accesses the first record K1 times according to the accessible number K1, and sequentially obtains the K1 identification records.

4. The method according to claim 1, characterized in that The first processing unit determines the first to-be-performed subtask according to the second reference value and the number N of the Attention operators, including: The first processing unit determines the first task from the M1 tasks according to the quotient of the second reference value and the number N of the Attention operators; The first processing unit determines a first operator from the N Attention operators according to a remainder of the second reference value and the number N of Attention operators; The first processing unit determines the first subtask to be executed according to the first task and the first operator.

5. The method according to claim 1, characterized in that The first processing unit determines K1 subtasks to be executed, including: The first processing unit determines K2 executable subtasks, any executable subtask of the K2 executable subtasks is a partial task of using the Attention operator to execute one of the M2 tasks assigned to the first processing unit; K2 is greater than or equal to K1; M2 is greater than or equal to M1; The first processing unit obtains a second record, where the second record is used to record the execution statuses corresponding to the M2 tasks respectively; The first processing unit determines the K1 to-be-executed subtasks according to the K2 executable subtasks and the second record.

6. The method according to claim 5, characterized in that The K2 executable subtasks include a first executable subtask, and the first executable subtask is any executable subtask among the K2 executable subtasks; The first processing unit determines the K1 subtasks to be executed according to the K2 executable subtasks and the second record, including: The first processing unit determines, from the M2 tasks, a second task corresponding to the first executable subtask; The first processing unit determines the execution status of the second task according to the second record; When the execution state of the second task is in an executable state, the first processing unit uses the first executable subtask as a to-be-executed subtask.

7. The method according to any one of claims 1 to 6, characterized in that: Any processing unit among the S processing units includes a tensor calculation unit and a vector calculation unit; the Attention operator includes a first matrix operator, a vector operator and a second matrix operator; The first processing unit uses the Attention operator to respectively execute the K1 subtasks to be executed, including: When the vector calculation unit in the first processing unit uses the vector operator to perform vector calculation on the zth subtask to be executed, the tensor calculation unit in the first processing unit uses the first matrix operator to perform matrix calculation on the z+ith subtask to be executed, or the tensor calculation unit in the first processing unit uses the second matrix operator to perform matrix calculation on the zjth subtask to be executed; z is an integer that spans [1,K1], and i and j are both integers greater than 0.

8. A computer device comprising a memory, the artificial intelligence chip, and a computer program stored in the memory and running on the artificial intelligence chip, characterized in that: When the artificial intelligence chip executes the computer program, the steps of the method as described in any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium, characterized in that: It stores a computer program executable by a computer device. When the computer program is run on the computer device, the computer device executes the steps of any method as claimed in claims 1-7.

10. A computer program product, characterized in that The computer program product comprises a computer program stored on a computer-readable storage medium, wherein the computer program comprises program instructions, and when the program instructions are executed by a computer device, the computer device is caused to execute the steps of the method according to any one of claims 1 to 7.

Images