Operator execution method, device, storage medium and program product
By splitting and synchronizing the steps of the operator, the problem of long waiting time of the computing unit is solved, the computing efficiency and performance are improved, the parallel processing of the operator steps is realized, and the computing power of the artificial intelligence chip is fully utilized.
Patent Information
- Application Number
- CN202410823946.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-25
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2044-06-25
AI Technical Summary
During the execution of existing operators, the waiting time between computing units is long, resulting in low computing efficiency and performance, and failure to fully utilize the computing power of artificial intelligence chips.
By splitting the steps of the operator into multiple steps chunks, and completing identification misalignment synchronization, the calculation unit allows the calculation unit to immediately execute the next step based on the results of the previous step, thereby reducing the waiting time and realizing parallel processing of multiple steps .
The computing efficiency and performance of the operator is optimized, the waiting time of the computing unit is reduced, the computing power of the artificial intelligence chip is fully utilized, and the correctness of the calculation and logical completeness are ensured.
Smart Images

Figure CN118394415B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular, to an operator execution method, device, storage medium, and program product. Background Art
[0002] Artificial intelligence models usually refer to neural network models that are trained for inference and prediction, such as image inference models, speech inference models, etc. The operations of artificial intelligence models can be implemented by operators in the computation graph. Operators refer to various operations performed on tensors at each layer in the artificial intelligence model. For example, the convolution operation performed by the convolution layer of the artificial intelligence model on the input data of the artificial intelligence model is the convolution operator.
[0003] In actual applications, the execution process of an operator generally includes multiple interdependent steps. To ensure the correctness of the operator calculation, one step in the operator must be completed before the next step in the operator can be executed, which affects the operator calculation efficiency and performance. Summary of the invention
[0004] The embodiments of the present application provide an operator execution method, device, storage medium, and program product for reducing the waiting time of a computing unit, thereby improving operator computing efficiency and performance.
[0005] On the one hand, an embodiment of the present application provides an operator execution method, including:
[0006] Split the first step and the second step contained in the operator into multiple step blocks respectively;
[0007] Execute the first step block in the first step by a first computing unit, obtain a first execution result, and send a completion mark of the first step;
[0008] The completion mark of the first step triggers the second calculation unit to execute the first step block in the second step based on the first execution result to obtain a second execution result.
[0009] On the one hand, an embodiment of the present application provides an operator execution device, including:
[0010] A splitting module, used to split the first step and the second step included in the operator into multiple step blocks respectively;
[0011] An execution module, configured to execute a first step block in the first step through a first computing unit, obtain a first execution result, and send a completion mark of the first step;
[0012] A trigger module is used to trigger the second computing unit to execute the first step block in the second step based on the first execution result through the completion mark of the first step to obtain a second execution result.
[0013] Optionally, the operator further includes a third step performed by the first computing unit, and the third step includes a plurality of step blocks;
[0014] The execution module is also used for:
[0015] After sending the completion mark of the first step, the other step blocks in the first step and the first step block in the third step are executed in sequence by the first computing unit;
[0016] After executing the first step block in the third step, a completion mark of the third step is sent, and the completion mark of the third step is used to indicate that the first computing unit has executed the first step block of the third step and the steps before the third step.
[0017] Optionally, the execution module is further used for:
[0018] After sending the completion mark of the third step, the completion mark of the third step is used to trigger the second computing unit to execute the other step blocks in the second step respectively based on the execution results of the other step blocks in the first step to obtain corresponding execution results.
[0019] Optionally, the operator further includes a fourth step performed by the second computing unit, and the fourth step includes a plurality of step blocks;
[0020] The execution module is also used for:
[0021] After sending the completion mark of the third step, the completion mark of the third step is used to trigger the second computing unit to execute the first step block in the fourth step based on the execution result of the first step block in the third step to obtain the corresponding execution result.
[0022] Optionally, the operator further includes a third step performed by the first computing unit, and the third step includes a plurality of step blocks;
[0023] The execution module is also used for:
[0024] After sending the completion mark of the first step, the other step blocks in the first step and the multiple step blocks in the third step are sequentially executed by the first computing unit;
[0025] After executing multiple step blocks in the third step, a completion mark of the third step is sent, and the completion mark of the third step is used to indicate that the first computing unit has completed executing the third step and steps before the third step.
[0026] Optionally, the operator further includes a fourth step performed by the second computing unit, and the fourth step includes a plurality of step blocks;
[0027] The execution module is also used for:
[0028] After sending the completion mark of the third step, the completion mark of the third step is used to trigger the second computing unit to execute the multiple step blocks in the fourth step based on the execution results of the multiple step blocks in the third step to obtain corresponding execution results.
[0029] Optionally, the execution module is further used for:
[0030] The completion mark of the first step is used to trigger the second calculation unit to execute the first step block in the second step based on the first execution result, and after obtaining the second execution result, the completion mark of the second step is sent.
[0031] Optionally, the completion identifier includes: memory fence information and thread synchronization information.
[0032] On the one hand, an embodiment of the present application provides a computer device, including a memory, an artificial intelligence chip, and a computer program stored in the memory and executable on the artificial intelligence chip, wherein the artificial intelligence chip implements the steps of the above-mentioned operator execution method when executing the computer program.
[0033] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program executable by a computer device. When the computer program runs on the computer device, the computer device executes the steps of the above-mentioned operator execution method.
[0034] On the one hand, an embodiment of the present application provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer device, the computer device executes the steps of the above-mentioned operator execution method.
[0035] In the embodiment of the present application, the first computing unit executes the first step block in the first step of the operator, and after obtaining the first execution result, the completion mark of the first step is sent, so that the second computing unit can be triggered to start executing the first step block in the second step of the operator based on the first execution result, that is, the second computing unit does not need to wait for the first computing unit to complete the first step before starting to execute the second step. Through the above-mentioned staggered synchronization method, without changing the calculation implementation of the operator, the synchronization strategy of each step in the operator is optimized, the waiting time of the computing unit is reduced, and multiple steps in the operator are processed in parallel, thereby improving the computing efficiency and performance, while giving full play to the computing power of the chip.
[0036] Secondly, for the other step blocks except the first step block in the step, when the computing unit completes the execution of these step blocks, the completion identifier of these step blocks is sent through the completion identifier of the next step executed by the computing unit. This not only ensures the logical completeness and calculation correctness, but also reduces the waiting time between multiple computing units in complex operators, thereby improving the overall performance of the operator. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0038] Figure 1 A schematic diagram of the structure of an artificial intelligence chip provided in an embodiment of the present application;
[0039] Figure 2 A flowchart of an operator execution method provided for related technologies;
[0040] Figure 3 A schematic diagram of a method for executing an operator provided in an embodiment of the present application Figure 1 ;
[0041] Figure 4 A schematic diagram of a method for executing an operator provided in an embodiment of the present application Figure 2 ;
[0042] Figure 5 A schematic diagram of a method for executing an operator provided in an embodiment of the present application Figure 3 ;
[0043] Figure 6 A schematic diagram of a method for executing an operator provided in an embodiment of the present application Figure 4 ;
[0044] Figure 7 A schematic diagram of the structure of an operator execution device provided in an embodiment of the present application;
[0045] Figure 8 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0046] In order to make the purpose, technical solution and beneficial effects of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.
[0047] refer to Figure 1 , which is a structural diagram of an artificial intelligence chip applicable to an embodiment of the present application. The artificial intelligence chip 100 at least includes: a video memory 101, an on-chip buffer 102, and multiple computing units 103.
[0048] The video memory 101 may be a high bandwidth memory (HBM) or other types of memory. The on-chip buffer 102 is a temporary memory, which has a smaller capacity than the video memory 101 but a faster data exchange speed. The number of on-chip buffers 102 may be one or more, and one on-chip buffer 102 provides storage services for multiple computing units 103.
[0049] The computing unit 103 may be a tensor computing unit, a vector computing unit, etc. The computing unit 103 may be used to execute various types of operators.
[0050] The computing unit 103 may directly read data from the video memory 101 to perform operator calculations; or may read data from the on-chip buffer 102 to perform operator calculations.
[0051] In addition to the above-mentioned structure, the artificial intelligence chip 100 in the present application may also include other structures, which are not specifically limited in the present application.
[0052] The artificial intelligence chip 100 may be: a graphics processing unit (GPU), a general-purpose computing on graphics processing units (GPGPU), a domain specific architecture (DSA), etc.
[0053] In actual applications, according to the mathematical meaning and calculation formula of different operators (single operators or fusion operators), if the calculation formula is divided into multiple interdependent steps for execution, the next step in the operator can only be executed after one step is completed, so as to ensure the correctness of the operator calculation.
[0054] However, in a scenario where multiple steps in an operator are processed in parallel by different computing units, a computing unit (for executing the next step in the operator) needs to wait until another computing unit completes the execution of the previous step in the operator before executing the next step.
[0055] For example, see the multi-head attention operator. Figure 2 ,The steps in the multi-head attention operator include: loading the key vector (K), loading the value vector (V), normalization operation (softmax), matrix multiplication (MatrixMultiply Accumulate, MMA for short) calculation 1 and MMA calculation 2.
[0056] The steps performed by the vector computing unit include: loading a key vector (K), loading a value vector (V), and a normalization operation.
[0057] The steps performed by the tensor computing unit include: MMA calculation 1 and MMA calculation 2, wherein MMA calculation 1 partially depends on the key vector (K) loaded by the vector computing unit, that is, in addition to the key vector (K), MMA calculation 1 also depends on the cached query vector (Q).
[0058] The MMA calculation 2 partially depends on the value vector (V) loaded by the vector calculation unit, that is, in addition to the value vector (V), the MMA calculation 2 also depends on the result of the normalization operation.
[0059] In the actual calculation process, the vector calculation unit first loads the key vector (K), and after the loading is completed, it sends the first completion mark corresponding to this step. Since the query vector (Q) has been cached, the tensor calculation unit starts the cache copy action in MMA calculation 1 after receiving the first completion mark, that is, copying the key vector (K), and then performs the subsequent operations of MMA calculation 1 based on the key vector (K) and the query vector (Q) to obtain the calculation result of MMA calculation 1.
[0060] After loading the key vector (K), the vector computing unit then loads the value vector (V), and after loading is completed, sends the second completion flag corresponding to this step. After receiving the second completion flag, the tensor computing unit starts to execute the cache copy action of MMA calculation 2, that is, copying the value vector (V).
[0061] After the vector calculation unit loads the value vector (V), it performs a normalization operation, wherein the normalization operation depends on the calculation result of the MMA calculation 1.
[0062] The tensor calculation unit performs subsequent operations of the MMA calculation 2 based on the copied value vector (V) and the result of the normalization operation to obtain the calculation result of the MMA calculation 2.
[0063] It can be seen from the above scheme that before the tensor calculation unit receives the first completion mark, that is, before the vector calculation unit loads the key vector, the tensor calculation unit has not started to execute MMA calculation 1, that is, it is in a waiting state. This causes the tensor calculation unit to wait for a long time, and multiple steps in the operator are actually executed serially, which affects the operator calculation efficiency and performance, and also fails to give full play to the computing power of the chip.
[0064] In view of this, this application is based on Figure 1 The architecture diagram of the artificial intelligence chip shown in FIG. 1 provides a flow chart of an operator execution method, such as Figure 3 As shown, the process of the method is executed by an artificial intelligence chip and includes the following steps:
[0065] Step 301: split the first step and the second step included in the operator into multiple step blocks respectively.
[0066] Specifically, the first step and the second step can be any two steps in the operator that have a dependency relationship, wherein the dependency relationship can be partial dependency or full dependency, wherein full dependency means that the calculation of the second step only depends on the result (calculation result or memory access result) of the first step. Partial dependency means that the calculation of the second step depends on the result (calculation result or memory access result) of other steps in addition to the result (calculation result or memory access result) of the first step. The first step and the second step are executed by different computing units.
[0067] The steps are split into multiple step blocks according to their specific contents. The sizes of the multiple step blocks can be the same or different. The first step and the second step can be split according to the same step splitting method, or the first step and the second step can be split according to different step splitting methods.
[0068] For example, the first step is to load a 2000*2000 target matrix, and the second step is to perform matrix multiplication on the target matrix. First, the first step is divided into two step blocks, namely step block 11 and step block 12. Step block 11 is: loading the 500*500 matrix block 1 in the target matrix, and step block 12 is: loading the remaining matrix block 2 in the target matrix.
[0069] Accordingly, the second step is divided into two step blocks, namely step block 21 and step block 22. Step block 21 is: performing matrix multiplication calculation on the loaded matrix block 1; step block 22 is: performing matrix multiplication calculation on the loaded matrix block 2.
[0070] For example, the first step is to load a 2000*2000 target matrix, and the second step is to perform matrix multiplication on the target matrix. First, split the first step into four step blocks, namely step block 11, step block 12, step block 13 and step block 14. Step block 11 is: load the 500*500 matrix block 1 in the target matrix; step block 12 is: load the 500*500 matrix block 2 in the target matrix; step block 13 is: load the 500*500 matrix block 3 in the target matrix; step block 14 is: load the 500*500 matrix block 4 in the target matrix.
[0071] Accordingly, the second step is divided into four step blocks, namely step block 21, step block 22, step block 23 and step block 24. Step block 21 is: performing matrix multiplication calculation on loaded matrix block 1; step block 22 is: performing matrix multiplication calculation on loaded matrix block 2; step block 23 is: performing matrix multiplication calculation on loaded matrix block 3; step block 24 is: performing matrix multiplication calculation on loaded matrix 4.
[0072] Step 302: Execute the first step block in the first step through the first computing unit, obtain a first execution result, and send a completion mark of the first step.
[0073] Specifically, the completion mark of the first step is used to indicate that the first computing unit has completed executing the first step block in the first step and the steps before the first step.
[0074] In some embodiments, the completion flag includes: memory fence information and thread synchronization information (sync), wherein the memory fence information is used in the operator calculation process to ensure that the execution result of the previous step has been stored in the specified memory location. The thread synchronization information is also used in the operator calculation process to ensure that all threads in the previous step have completed the calculation and memory access.
[0075] Step 303: trigger the second computing unit to execute the first step block in the second step based on the first execution result through the completion mark of the first step to obtain the second execution result.
[0076] Specifically, the second step executed by the second computing unit depends on the execution result of the first step executed by the first computing unit. When the second computing unit receives the completion mark of the first step, it starts to execute the second step. Since the first computing unit currently only obtains the first execution result of the first step block in the first step, the second computing unit executes the first step block in the second step based on the first execution result to obtain the second execution result.
[0077] In some cases, if the first step block in the second step depends not only on the first execution result but also on the execution results of other step blocks, then after obtaining the first execution result and the execution results of other step blocks (i.e., all dependencies of the first step block in the second step), the first step block in the second step starts to be executed.
[0078] In some operators, the first step block in the second step can also be further split into multiple secondary step blocks. In this way, when some dependencies of the first step block in the second step are obtained (the first execution result or the execution results of other step blocks), the secondary step blocks corresponding to the partial dependencies are started to be executed; when other dependencies of the first step block in the second step are subsequently obtained (the execution results of other step blocks or the first execution result), the other secondary step blocks continue to be executed.
[0079] It should be noted that an operator in the present application is not limited to including the first step and the second step, but may also include other steps. Moreover, for each step block in any step, after all dependencies of the step block are obtained, the step block is started to be executed; of course, the step block can also be further split into multiple secondary step blocks, so that when some dependencies of the step block are obtained, the secondary step blocks corresponding to the partial dependencies are started to be executed, and when other dependencies of the step block are obtained later, other secondary step blocks are continued to be executed.
[0080] In practical applications, according to the specific situation of the operator, the secondary step blocks can be further split, and so on; this application does not make any specific restrictions on the specific level of step block splitting.
[0081] Secondly, the "first" and "second" in this application are only used to distinguish different steps, and are not used to limit the execution order of the steps. In addition, the specific type and number of steps performed by the first computing unit and the second computing unit are associated with specific operators, and this application does not make specific limitations.
[0082] For example, see Figure 4, set the first calculation unit as a vector calculation unit, and the second calculation unit as a tensor calculation unit. The first step of the operator is to load a 2000*2000 target matrix, and the second step of the operator is to perform matrix multiplication on the target matrix. The first step includes: step block 11 and step block 12, step block 11 is: load the 500*500 matrix block 1 in the target matrix, and step block 12 is: load the remaining matrix block 2 in the target matrix.
[0083] The second step includes: step block 21 and step block 22. Step block 21 is: performing matrix multiplication calculation on the loaded matrix block 1; step block 22 is: performing matrix multiplication calculation on the loaded matrix block 2.
[0084] The vector calculation unit first executes step block 11 in the first step, that is, loads matrix block 1 from the video memory, and sends the memory fence information and thread synchronization information of the first step after the loading is completed. After receiving the memory fence information and thread synchronization information, the tensor calculation unit starts to execute step block 21 in the second step, that is, performs matrix multiplication calculation on matrix block 1 loaded by the vector calculation unit.
[0085] In the embodiment of the present application, the first computing unit executes the first step block in the first step of the operator, and after obtaining the first execution result, the completion mark of the first step is sent, so that the second computing unit can be triggered to start executing the first step block in the second step of the operator based on the first execution result, that is, the second computing unit does not need to wait for the first computing unit to complete the first step before starting to execute the second step. Through the above-mentioned staggered synchronization method, without changing the calculation implementation of the operator, the synchronization strategy of each step in the operator is optimized, the waiting time of the computing unit is reduced, and multiple steps in the operator are processed in parallel, thereby improving the computing efficiency and performance, while giving full play to the computing power of the chip.
[0086] In practical applications, for other step blocks in the first step except the first step block, when the first computing unit completes executing these step blocks, it is also necessary to notify the second computing unit by sending a completion mark. Since the completion mark of the first step has been sent after the first step block is completed, the completion marks of other step blocks in the first step need to be sent through the completion mark of the next step of the first step executed by the first computing unit.
[0087] In a specific implementation, the first computing unit may send a completion mark when executing the first step block of the next step of the first step; or may send a completion mark when executing all step blocks of the next step of the first step. The following describes these two implementations:
[0088] In the first implementation mode, the operator further includes a third step executed by the first computing unit, and the third step includes a plurality of step blocks.
[0089] After the first step block in the first step is executed by the first computing unit, the other step blocks in the first step and the first step block in the third step are executed in sequence by the first computing unit. After the first step block in the third step is executed, a completion mark of the third step is sent.
[0090] Specifically, the descriptions of "first", "second", "third", etc. in this application are only used to distinguish different steps, and are not used to limit the execution order of the steps. For example, the first step, the second step, and the third step in this application do not necessarily have a strict order dependency, that is, the first step, the second step, and the third step are not necessarily executed in sequence. Similarly, the descriptions of the fourth step, the fifth step, etc. that appear later are also for distinguishing different steps, and are not used to limit the execution order of the steps, and will not be repeated later.
[0091] In the present application, the first step and the third step are steps executed in the same computing unit (ie, the first computing unit), and therefore, the execution of the first step and the third step has a sequence.
[0092] Specifically, the first computing unit first executes the first step, and after executing the first step, the first computing unit can start to execute the third step. The first computing unit does not need to wait until the second computing unit completes the second step before starting to execute the third step.
[0093] The completion mark of the third step is used to indicate that the first computing unit has completed the first step block of the third step and the steps before the third step. Therefore, the completion marks of other step blocks in the first step are sent through the completion mark of the next step (i.e., the third step) executed by the first computing unit.
[0094] In some embodiments, the completion mark of the third step triggers the second computing unit to execute other step blocks in the second step respectively based on the execution results of other step blocks in the first step to obtain corresponding execution results.
[0095] Specifically, when the second computing unit receives the completion mark of the third step, it indicates that the first computing unit has completed the steps before the third step (that is, the multiple step blocks in the first step). Therefore, the second computing unit starts to execute the other step blocks in the second step except the first step block; that is, the second computing unit executes the other step blocks in the second step based on the execution results of the other step blocks in the first step to obtain corresponding execution results.
[0096] In some cases, if other step blocks in the second step not only depend on the execution results of other step blocks in the first step, but also depend on the execution results of step blocks of other steps, then after obtaining the execution results of other step blocks in the first step and the execution results of step blocks of other steps (that is, all dependencies of other step blocks in the second step), start executing other step blocks in the second step.
[0097] In some operators, the other step blocks in the second step can also be further split into multiple secondary step blocks, so that when some dependencies of the other step blocks in the second step are obtained, the secondary step blocks corresponding to the partial dependencies are started to be executed. When other dependencies of the other step blocks in the second step are obtained later, other secondary step blocks are continued to be executed.
[0098] In addition, the second computing unit may send a completion mark of the second step after executing the first step block in the second step. The completion marks of other step blocks in the second step are sent through the completion mark of the next step executed by the second computing unit. Of course, the second computing unit may also send a completion mark of the second step after executing multiple step blocks in the second step, and this application does not make specific limitations on this.
[0099] In some embodiments, since the first computing unit sends a completion mark of the third step after executing the first step block of the third step, at this time, other steps that depend on the third step can start to be executed.
[0100] Specifically, the operator also includes a fourth step executed by the second computing unit, and the fourth step includes multiple step blocks. Through the completion mark of the third step, the second computing unit is triggered to execute the first step block in the fourth step based on the execution result of the first step block of the third step to obtain the corresponding execution result.
[0101] The third step and the fourth step may be any two steps in the operator that have a dependency relationship, wherein the dependency relationship may be partial dependency or complete dependency, and the third step and the fourth step are executed by different computing units.
[0102] When the second computing unit receives the completion mark of the third step, it starts to execute the fourth step. Since the first computing unit currently only obtains the execution result of the first step block in the third step, the second computing unit executes the first step block in the fourth step based on the execution result to obtain the corresponding execution result.
[0103] In some cases, if the first step block in the fourth step depends not only on the execution result of the first step block in the third step, but also on the execution results of other step blocks, then after obtaining the execution result of the first step block in the third step and the execution results of other step blocks (i.e., all dependencies that the first step block in the fourth step depends on), start executing the first step block in the fourth step.
[0104] In some operators, the first step block in the fourth step can also be further split into multiple secondary step blocks, so that when some dependencies of the first step block in the second step (such as the execution result of the first step block in the third step) are obtained, the secondary step blocks corresponding to the partial dependencies are started to be executed; when other dependencies of the first step block in the second step (such as the execution results of other step blocks) are subsequently obtained, the other secondary step blocks are continued to be executed.
[0105] Similarly, for other step blocks in the fourth step, after all dependencies of other step blocks in the fourth step are obtained, other step blocks in the fourth step can be started to be executed. Other step blocks in the fourth step can also be further split into multiple secondary step blocks, so that when some dependencies of other step blocks in the fourth step are obtained, the secondary step blocks corresponding to the partial dependencies are started to be executed. When other dependencies of other step blocks in the fourth step are obtained later, other secondary step blocks continue to be executed.
[0106] Since the other step blocks in the second step and the fourth step are all executed by the second computing unit, after the second computing unit receives the completion identifier of the third step sent by the first computing unit, the second computing unit determines the order in which the step blocks in the second step and the step blocks in the fourth step are executed by the second computing unit according to the type of operator.
[0107] For example, the second computing unit may first execute the first step block in the fourth step, then execute other step blocks in the second step, and then execute other step blocks in the fourth step; or it may first execute other step blocks in the second step, and then execute multiple step blocks in the fourth step. This application does not make any specific limitations on this.
[0108] In addition, the completion identifiers of the other step blocks except the first step block in the third step can be sent through the completion identifier of the next step executed by the first computing unit, and so on, until the last step executed by the first computing unit. For the last step executed by the first computing unit, the first computing unit can send the completion identifier of the step after executing multiple step blocks of the last step.
[0109] Similarly, the second computing unit may send a completion mark of the fourth step after executing the first step block in the fourth step. The completion marks of other step blocks in the fourth step are sent through the completion mark of the next step executed by the second computing unit until the last step executed by the second computing unit. For the last step executed by the second computing unit, the second computing unit may send a completion mark of the step after executing multiple step blocks of the last step. Of course, the second computing unit may also send a completion mark of the fourth step after executing multiple step blocks in the fourth step, and this application does not make specific limitations on this.
[0110] For example, see Figure 5 , set the first calculation unit as a vector calculation unit and the second calculation unit as a tensor calculation unit. The steps in the multi-head attention operator include: the first step (loading key vector), the second step (matrix multiplication calculation 1), the third step (loading value vector), the fourth step (matrix multiplication calculation 2), and the fifth step (normalization operation).
[0111] The steps performed by the vector calculation unit include: a first step, a third step and a fifth step.
[0112] The steps performed by the tensor computing unit include: a second step and a fourth step.
[0113] The vector computing unit first executes the first step block (step block M 11) of the first step, that is, loads the first vector block in the key vector. After loading the first vector block in the key vector, it sends a completion flag 1 corresponding to the first step (including memory fence information and thread synchronization information).
[0114] After receiving the completion mark 1, the tensor calculation unit starts to execute the first step block of the second step (step block M21), that is, copying the first vector block in the key vector from the vector calculation unit and performing matrix calculation based on the copied vector block and the first vector block of the cached query vector to obtain result block 21.
[0115] After loading the first vector block in the key vector, the vector computing unit executes the second step block (step block M12) of the first step, that is, loads other vector blocks in the key vector. After loading the entire key vector, the first step block (step block M31) of the third step begins to be executed, that is, loads the first vector block in the value vector. After loading the first vector block in the value vector, the completion flag 2 corresponding to the third step (including memory fence information and thread synchronization information) is sent.
[0116] After receiving the completion mark 2, the tensor calculation unit starts to execute the first step block (step block M41) of the fourth step, that is, copying the first vector block in the value vector from the vector calculation unit.
[0117] After loading the first vector block in the value vector, the vector calculation unit executes the second step block (step block M32) of the third step, that is, loads other vector blocks in the value vector. After loading the entire value vector, the first step block (step block M51) of the fifth step is started, that is, a normalization operation is performed based on the result block 21 to obtain the result block 51. Then, the completion mark 3 corresponding to the fifth step (including memory fence information and thread synchronization information) is sent.
[0118] After receiving the completion mark 3, the tensor calculation unit starts to execute the second step block (step block M42) of the fourth step, that is, performs matrix calculation based on the first vector block in the copied value vector and the result block 51 to obtain the result block 42.
[0119] Next, the tensor calculation unit starts to execute the second step block (step block M22) in the second step, that is, copying other vector blocks in the key vector from the vector calculation unit, and performing matrix calculation based on other vector blocks in the key vector and other vector blocks of the cached query vector to obtain result block 22.
[0120] Next, the tensor calculation unit starts to execute the third step block (step block M43) in the fourth step, that is, copying other vector blocks in the value vector from the vector calculation unit.
[0121] After executing the first step block (step block M51 ) of the fifth step, the vector calculation unit continues to execute the second step block (step block M52 ) of the fifth step, that is, performs a normalization operation based on the result block 22 to obtain the result block 52 .
[0122] After obtaining the result block 52 , the tensor calculation unit starts to execute the fourth step block (step block M44 ) in the fourth step, that is, performs matrix calculation based on other vector blocks in the copied value vector and the result block 52 to obtain the result block 44 .
[0123] In an embodiment of the present application, for step blocks other than the first step block in the steps, when the computing unit completes execution of these step blocks, the completion identifiers of these step blocks are sent through the completion identifier of the next step executed by the computing unit. This not only ensures logical completeness and calculation correctness, but also reduces the waiting time between multiple computing units in a complex operator, thereby improving the overall performance of the operator.
[0124] In the second implementation mode, the operator further includes a third step executed by the first computing unit, and the third step includes a plurality of step blocks.
[0125] After the first step block in the first step is executed by the first computing unit, the other step blocks in the first step and the multiple step blocks in the third step are executed in sequence by the first computing unit. After the multiple step blocks in the third step are executed, a completion mark of the third step is sent.
[0126] At this time, the completion mark of the third step is used to indicate that the first computing unit has completed the third step and the steps before the third step. The relationship between the third step and the first step and the second step has been explained above and will not be repeated here.
[0127] In some embodiments, the completion mark of the third step triggers the second computing unit to execute other step blocks in the second step respectively based on the execution results of other step blocks in the first step to obtain corresponding execution results.
[0128] Specifically, when the second computing unit receives the completion mark of the third step, it indicates that the first computing unit has completed the steps before the third step (that is, the multiple step blocks in the first step). Therefore, the second computing unit starts to execute the other step blocks in the second step except the first step block; that is, the second computing unit executes the other step blocks in the second step based on the execution results of the other step blocks in the first step to obtain corresponding execution results.
[0129] In some cases, if other step blocks in the second step not only depend on the execution results of other step blocks in the first step, but also depend on the execution results of step blocks of other steps, then after obtaining the execution results of other step blocks in the first step and the execution results of step blocks of other steps (that is, all dependencies of other step blocks in the second step), start executing other step blocks in the second step.
[0130] In some operators, the other step blocks in the second step can also be further split into multiple secondary step blocks, so that when some dependencies of the other step blocks in the second step are obtained, the secondary step blocks corresponding to the partial dependencies are started to be executed. When other dependencies of the other step blocks in the second step are obtained later, other secondary step blocks are continued to be executed.
[0131] In some embodiments, the operator also includes a fourth step performed by the second computing unit, and the fourth step includes multiple step blocks.
[0132] The completion mark of the third step triggers the second computing unit to execute the multiple step blocks in the fourth step based on the execution results of the multiple step blocks in the third step to obtain corresponding execution results.
[0133] Specifically, the relationship between the fourth step and the third step has been explained in the previous text and will not be repeated here. The fourth step executed by the second computing unit depends on the execution result of the third step executed by the first computing unit. When the second computing unit receives the completion mark of the third step, it starts to execute the fourth step. Since the first computing unit has currently obtained the execution results of multiple step blocks in the third step, the second computing unit executes multiple step blocks in the fourth step respectively based on the obtained execution results of the multiple step blocks to obtain corresponding execution results.
[0134] In some cases, for each step block in the fourth step, if the step block depends not only on the execution results of the step block in the third step, but also on the execution results of step blocks in other steps, then after obtaining the execution results of the step block in the third step and the execution results of the step blocks in other steps (that is, all dependencies of the step block), start executing the step block in the fourth step.
[0135] In some operators, for each step block in the fourth step, the step block can be further split into multiple secondary step blocks. In this way, when some dependencies of the step block are obtained, the secondary step blocks corresponding to the partial dependencies are started to be executed; when other dependencies of the step block are subsequently obtained, other secondary step blocks are continued to be executed.
[0136] Since the other step blocks in the first step and the fourth step are all executed by the second computing unit, after the second computing unit receives the completion mark of the third step sent by the first computing unit, the second computing unit determines the order in which the second computing unit executes the step blocks in the second step and the step blocks in the fourth step according to the type of operator. This application does not make any specific limitations on this.
[0137] For example, see Figure 6 , set the first calculation unit as a vector calculation unit and the second calculation unit as a tensor calculation unit. The operator includes: the first step (loading target matrix 1), the second step (matrix multiplication calculation 1 on target matrix 1), the third step (loading target matrix 2), and the fourth step (matrix multiplication calculation 2 on target matrix 2).
[0138] The steps performed by the vector calculation unit include: a first step and a third step.
[0139] The steps performed by the tensor computing unit include: a second step and a fourth step.
[0140] The vector computing unit first executes the first step block (step block M11) of the first step, that is, loads the matrix block 11 in the target matrix 1, and after loading the matrix block 11, sends the completion mark 1 corresponding to the first step (including memory fence information and thread synchronization information). After receiving the completion mark 1, the tensor computing unit starts to execute the first step block (step block M21) of the second step, that is, performs matrix multiplication calculation based on the loaded matrix block 11.
[0141] After loading matrix block 11, the vector calculation unit executes the second step block (step block M12) of the first step, that is, continues to load matrix block 12 in target matrix 1. After loading matrix block 12, the third step is started, that is, loading target matrix 2 (including matrix block 21 and matrix block 22). After loading target matrix 2, a completion mark 2 corresponding to the third step (including memory fence information and thread synchronization information) is sent.
[0142] After receiving the completion mark 2, the tensor calculation unit starts to execute the first step block (step block M41) of the fourth step, that is, to perform matrix multiplication calculation based on the loaded matrix block 21. Then, the second step block (step block M22) of the second step is executed, that is, to perform matrix multiplication calculation based on the loaded matrix block 12. Then, the second step block (step block M42) of the fourth step is executed, that is, to perform matrix multiplication calculation based on the loaded matrix block 22.
[0143] In an embodiment of the present application, for step blocks other than the first step block in the steps, when the computing unit completes execution of these step blocks, the completion identifiers of these step blocks are sent through the completion identifier of the next step executed by the computing unit. This not only ensures logical completeness and calculation correctness, but also reduces the waiting time between multiple computing units in a complex operator, thereby improving the overall performance of the operator.
[0144] Based on the same technical concept, the present application embodiment provides a structural diagram of an operator execution device, such as Figure 7 As shown, the operator execution device 700 includes:
[0145] A splitting module 701 is used to split the first step and the second step included in the operator into multiple step blocks respectively;
[0146] An execution module 702 is configured to execute a first step block in the first step through a first computing unit, obtain a first execution result, and send a completion mark of the first step;
[0147] The trigger module 703 is used to trigger the second calculation unit to execute the first step block in the second step based on the first execution result through the completion mark of the first step to obtain a second execution result.
[0148] Optionally, the operator further includes a third step performed by the first computing unit, and the third step includes a plurality of step blocks;
[0149] The execution module 702 is also used for:
[0150] After sending the completion mark of the first step, the other step blocks in the first step and the first step block of the third step are executed in sequence by the first computing unit;
[0151] After executing the first step block of the third step, a completion mark of the third step is sent, and the completion mark of the third step is used to indicate that the first computing unit has executed the first step block of the third step and the steps before the third step.
[0152] Optionally, the execution module 702 is further configured to:
[0153] After sending the completion mark of the third step, the completion mark of the third step is used to trigger the second computing unit to execute the other step blocks in the second step respectively based on the execution results of the other step blocks in the first step to obtain corresponding execution results.
[0154] Optionally, the operator further includes a fourth step performed by the second computing unit, and the fourth step includes a plurality of step blocks;
[0155] The execution module 702 is also used for:
[0156] After sending the completion mark of the third step, the completion mark of the third step is used to trigger the second computing unit to execute the first step block in the fourth step based on the execution result of the first step block of the third step to obtain the corresponding execution result.
[0157] Optionally, the operator further includes a third step performed by the first computing unit, and the third step includes a plurality of step blocks;
[0158] The execution module 702 is also used for:
[0159] After sending the completion mark of the first step, the other step blocks in the first step and the multiple step blocks of the third step are executed in sequence by the first computing unit;
[0160] After executing multiple step blocks of the third step, a completion mark of the third step is sent, and the completion mark of the third step is used to indicate that the first computing unit has executed the third step and steps before the third step.
[0161] Optionally, the operator further includes a fourth step performed by the second computing unit, and the fourth step includes a plurality of step blocks;
[0162] The execution module 702 is also used for:
[0163] After sending the completion mark of the third step, the completion mark of the third step is used to trigger the second computing unit to execute the multiple step blocks in the fourth step based on the execution results of the multiple step blocks of the third step to obtain corresponding execution results.
[0164] Optionally, the execution module 702 is further configured to:
[0165] The completion mark of the first step is used to trigger the second calculation unit to execute the first step block in the second step based on the first execution result, and after obtaining the second execution result, the completion mark of the second step is sent.
[0166] Optionally, the completion identifier includes: memory fence information and thread synchronization information.
[0167] In the embodiment of the present application, the first computing unit executes the first step block in the first step of the operator, obtains the first execution result, and sends the completion mark of the first step, so that the second computing unit can be triggered to start executing the first step block in the second step of the operator based on the first execution result, that is, the second computing unit does not need to wait for the first computing unit to complete the entire first step before starting to execute the second step. Through the above-mentioned staggered synchronization method, without changing the operator calculation implementation, the synchronization strategy of each step in the operator is optimized, the waiting time of the computing unit is reduced, and the computing efficiency and performance are improved.
[0168] Secondly, for the other step blocks except the first step block in the step, when the computing unit completes the execution of these step blocks, the completion identifier of these step blocks is sent through the completion identifier of the next step executed by the computing unit. This not only ensures the logical completeness and calculation correctness, but also reduces the waiting time between multiple computing units in complex operators, thereby improving the overall performance of the operator.
[0169] Based on the same technical concept, the embodiment of the present application provides a computer device, such as Figure 8As shown, it includes at least one artificial intelligence chip 100 and a memory 801 connected to the at least one artificial intelligence chip 100. The specific connection medium between the artificial intelligence chip 100 and the memory 801 is not limited in the embodiment of the present application. Figure 8 In the example, the artificial intelligence chip 100 and the memory 801 are connected via a bus. The bus can be divided into an address bus, a data bus, a control bus, etc.
[0170] In an embodiment of the present application, the memory 801 stores instructions that can be executed by at least one artificial intelligence chip 100. At least one artificial intelligence chip 100 can execute the steps of the above-mentioned operator execution method by executing the instructions stored in the memory 801.
[0171] Among them, the artificial intelligence chip 100 is the control center of the computer device, and various interfaces and lines can be used to connect various parts of the computer device, and the operator execution is realized by running or executing instructions stored in the memory 801 and calling data stored in the memory 801. Optionally, the artificial intelligence chip 100 may include one or more processing units, and the artificial intelligence chip 100 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application program, etc., and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the artificial intelligence chip 100. In some embodiments, the artificial intelligence chip 100 and the memory 801 can be implemented on the same chip, and in some embodiments, they can also be implemented separately on independent chips.
[0172] The artificial intelligence chip 100 can be a general-purpose processor, such as a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, which can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor.
[0173] The memory 801 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 801 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, and the like. The memory 801 is any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer device, but is not limited thereto. The memory 801 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.
[0174] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program executable by a computer device. When the computer program runs on the computer device, the computer device executes the steps of the above-mentioned operator execution method.
[0175] Based on the same inventive concept, an embodiment of the present application provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer device, the computer device executes the steps of the above-mentioned operator execution method.
[0176] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0177] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer device or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0178] These computer program instructions may also be stored in a computer readable memory capable of directing a computer device or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0179] These computer program instructions can also be loaded onto a computer device or other programmable data processing device so that a series of operation steps are executed on the computer device or other programmable device to produce a process implemented by the computer device, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0180] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.
[0181] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. An operator execution method, characterized in that: include: Split the first step and the second step contained in the operator into multiple step blocks respectively; Execute the first step block in the first step by a first computing unit, obtain a first execution result, and send a completion mark of the first step; Triggering, by the completion mark of the first step, a second computing unit to execute the first step block in the second step based on the first execution result to obtain a second execution result; The completion marks of the other step blocks in the first step are sent through the completion mark of the next step of the first step executed by the first computing unit, and the next step is a third step executed after the first computing unit completes the execution of the first step; The completion mark of the third step triggers the second calculation unit to execute other step blocks in the second step respectively based on the execution results of other step blocks in the first step to obtain corresponding execution results.
2. The method according to claim 1, characterized in that The third step includes a plurality of step blocks; after sending the completion mark of the first step, it also includes: sequentially executing other step blocks in the first step and the first step block in the third step by the first computing unit; After executing the first step block in the third step, a completion mark of the third step is sent, and the completion mark of the third step is used to indicate that the first computing unit has executed the first step block in the third step and the steps before the third step.
3. The method according to claim 2, characterized in that The operator also includes a fourth step performed by the second computing unit, wherein the fourth step includes a plurality of step blocks; After sending the completion mark of the third step, the method further includes: The completion mark of the third step triggers the second calculation unit to execute the first step block in the fourth step based on the execution result of the first step block in the third step to obtain the corresponding execution result.
4. The method according to claim 1, characterized in that The third step includes a plurality of step blocks; After sending the completion mark of the first step, the method further includes: sequentially executing other step blocks in the first step and multiple step blocks in the third step by the first computing unit; After executing multiple step blocks in the third step, a completion mark of the third step is sent, and the completion mark of the third step is used to indicate that the first computing unit has completed executing the third step and steps before the third step.
5. The method according to claim 4, characterized in that The operator also includes a fourth step performed by the second computing unit, wherein the fourth step includes a plurality of step blocks; After sending the completion mark of the third step, the method further includes: The completion mark of the third step triggers the second calculation unit to execute the multiple step blocks in the fourth step based on the execution results of the multiple step blocks in the third step to obtain corresponding execution results.
6. The method according to claim 1, characterized in that After the completion mark of the first step is used to trigger the second calculation unit to execute the first step block in the second step based on the first execution result, and the second execution result is obtained, the method further includes: Send a completion indication of the second step.
7. The method according to any one of claims 1 to 6, characterized in that: The completion identifier includes: memory fence information and thread synchronization information.
8. A computer device comprising a memory, an artificial intelligence chip, and a computer program stored in the memory and executable on the artificial intelligence chip, characterized in that: When the artificial intelligence chip executes the computer program, the steps of the method as described in any one of claims 1 to 7 are implemented.
9. A computer-readable storage medium, characterized in that: It stores a computer program executable by a computer device. When the computer program is run on the computer device, the computer device executes the steps of the method as claimed in any one of claims 1 to 7.
10. A computer program product, characterized in that The computer program product comprises a computer program stored on a computer-readable storage medium, wherein the computer program comprises program instructions, and when the program instructions are executed by a computer device, the computer device is caused to execute the steps of the method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Neural network tensor operator combination automatic tuning method and device
CN116090521A
Tensor dimension segmentation method, system, equipment and medium
CN117634711A
Data processing method and device and storage medium
CN118012628A
Computing graph compiling method and device, computing equipment and storage medium
CN118034695A
Operator execution method and device, storage medium and program product
CN118132156A
