Operator execution method, processor and computer equipment

By using a control unit to perform vector and matrix operations in batches and in parallel within a separate architecture processor, the problem of low efficiency of the Mix operator in a coupled architecture is solved, achieving efficient parallel computing and load balancing.

CN121723015APending Publication Date: 2026-03-24HUAWEI TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411336430.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-23
Publication Date
2026-03-24

AI Technical Summary

Technical Problem

In existing processors, the vector operation unit and matrix operation unit are coupled, which cannot perform vector operations and matrix operations in parallel, resulting in low execution efficiency of the Mix operator and frequent synchronous instruction calls.

Method used

The processor, which adopts a separate architecture, executes vector operation units and matrix operation units in batches and in parallel through the control unit. It performs the first and second operations on multiple matrix groups respectively, reducing synchronous instruction calls under data dependencies and optimizing the operation order and load balancing.

Benefits of technology

It improves the execution efficiency of the Mix operator, reduces the number of synchronous instruction calls and data transfer, and realizes efficient parallel execution of vector operation units and matrix operation units.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN121723015A_ABST
    Figure CN121723015A_ABST
Patent Text Reader

Abstract

The embodiment of the invention belongs to the technical field of computers, and particularly relates to an operator execution method, a processor and computer equipment. The method is applied to a processor, and the processor is provided with a control unit, a vector operation unit and a matrix operation unit. The control unit obtains a plurality of matrix groups of operators to be executed. And the control unit controls the vector operation unit and the matrix operation unit to execute the first operation and the second operation included in the operators on the plurality of matrix groups in batches so as to obtain an execution result of executing the operators on each matrix group. Wherein when the first operation and the second operation are executed for the Nth time, the at least one vector operation unit executes the first operation on the first matrix group, the at least one matrix operation unit executes the second operation on the second matrix group, the first matrix group and the second matrix group are different matrix groups, and N is greater than 1 and less than M; m is the total batch for executing the first operation and the second operation. By adopting the method, the execution efficiency of the processor on the operator can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to an operator execution method, a processor and a computer device. BACKGROUND

[0002] A large number of vector operations and matrix operations are involved in artificial intelligence (AI) calculation. A processor (such as a neural network processing unit (NPU)) can implement vector operations and matrix operations respectively by configuring a vector operation unit and a matrix operation unit.

[0003] In the related art, the vector operation unit and the matrix operation unit configured by the processor are mostly coupled architectures, that is, the vector operation unit and the matrix operation unit share part of the hardware resources and cannot implement vector operations and matrix operations in parallel. Therefore, the calculation process corresponding to the operator involving vector operations and matrix operations in the current AI calculation is mainly designed based on the processor with the coupled architecture, that is, the parallel execution of vector operations and matrix operations is not involved in the calculation process.

[0004] With the development of computing technology, the processor with a separated architecture of the vector operation unit and the matrix operation unit has begun to be applied, and the processor with the separated architecture can execute vector operations and matrix operations simultaneously. Therefore, there is still room for further optimization of the current calculation process of the operator. SUMMARY

[0005] Embodiments of the present application provide an operator execution method, a processor and a computer device, which can execute vector operations and matrix operations included in the operator in parallel, can improve the execution efficiency of the operator, and the corresponding technical solutions comprise:

[0006] In a first aspect, an execution method of an operator is provided and applied to a processor. The processor is configured with a control unit, at least one vector operation unit, and at least one matrix operation unit. The execution method of the operator includes: the control unit obtaining a plurality of matrix groups of the operator to be executed, wherein the operator includes at least one first operation and at least one second operation, the first operation is executed by the vector operation unit, and the second operation is executed by the matrix operation unit. The control unit controls the vector operation unit and the matrix operation unit to execute the first operation and the second operation included in the operator in batches to obtain an execution result of the operator for each matrix group. Wherein, when the vector operation unit and the matrix operation unit execute the first operation and the second operation for the Nth time, at least one vector operation unit executes the first operation on at least one first matrix group, and at least one matrix operation unit executes the second operation on at least one second matrix group, the first matrix group and the second matrix group are different matrix groups, N is greater than 1 and less than M, and M is the total batch number of the execution of the first operation and the second operation.

[0007] The vector operation unit and the matrix operation unit of the processor are in a separated architecture, the first operation refers to an operation executed by the vector operation unit, including vector operations and matrix operations that need to be converted into vector operations for implementation, and the second operation refers to an operation executed by the matrix operation unit, such as a matrix multiplication operation.

[0008] In the scheme shown in the present application, the control unit can control the parallel execution of the first operation and the second operation corresponding to different matrix groups in the process of controlling the vector operation unit and the matrix operation unit to execute the first operation and the second operation corresponding to each matrix group. In this way, the overall time length of the control unit for controlling the execution of the first operation and the second operation corresponding to a plurality of matrix groups can be compressed, and the execution efficiency of the operator can be improved.

[0009] In an implementable manner, the at least one first operation and the at least one second operation included in the operator are divided into different operation sets, wherein, in the execution sequence, one operation set is a first operation set including the first operation, and the other operation set is a second operation set including the second operation, and there is a data dependency relationship between the two operation sets.

[0010] In the scheme shown in the present application, the first operation and the second operation included in the operator can be divided into the first operation set and the second operation set respectively, and in the execution sequence, two adjacent calculation sets are one first calculation set and one second calculation set, and the operations included in one calculation set are executed by the same vector operation unit or the same matrix operation unit. In this way, the control unit controls the execution of the operator, and after the execution of the currently executable first operation, the second operation is executed again, and after the execution of the currently executable second operation, the first operation is executed again. In this way, the number of times of alternating execution of the first operation and the second operation can be reduced, on the one hand, the number of synchronization instructions required between the vector operation unit and the matrix operation unit can be reduced, thereby improving the execution efficiency of the operator, on the other hand, the data transfer amount between the vector operation unit and the matrix operation unit can be reduced, thereby improving the execution efficiency of the operator.

[0011] In an implementable manner, the control unit controls the vector operation unit and the matrix operation unit to execute the first operation set and the second operation set included in the operator in batches on the plurality of matrix groups to obtain the execution result of executing the operator on each group of matrix groups. Wherein, when the vector operation unit and the matrix operation unit execute the first operation set and the second operation set for the Nth time, at least one vector operation unit executes the first operation set on at least one first matrix group, and at least one matrix operation unit executes the second operation set on at least one second matrix group. The number of the first operation set and the second operation set executed in parallel by the vector operation unit and the matrix operation unit is X, wherein X is greater than 1 and less than or equal to Y, Y is the minimum value in N and Z, and Z is the number of the first operation set and the second operation set divided by the operator.

[0012] In the scheme shown in the present application, the control unit controls the vector operation unit and the matrix operation unit to execute the first operation set and the second operation set corresponding to each group of matrix blocks respectively, and according to the data dependency relationship, the first operation set and the second operation set corresponding to different matrix groups are controlled to be executed in parallel with the maximum parallelism X. In this way, the overall time length of the control unit controlling the execution of the first operation set and the second operation set corresponding to the plurality of groups of matrix blocks can be compressed, thereby improving the execution efficiency of the operator.

[0013] In an implementable manner, the control unit controls the vector operation unit and the matrix operation unit to execute the next batch of operation sets in parallel after each parallel execution of the operation set is completed in response to the synchronization instruction.

[0014] In the scheme shown in the present application, the control unit can control the vector operation unit and the matrix operation unit to execute the operation set in parallel according to the synchronization instruction, so that the number of calls of the synchronization instruction can be reduced, and the execution efficiency of the operator can be further improved.

[0015] In an implementable manner, the processor is configured with a plurality of computing modules, each computing module including a control unit, a vector operation unit and a matrix operation unit. The control unit controls the vector operation unit and the matrix operation unit to perform the first operation and the second operation included in the operator on the plurality of matrix groups in batches, including: the control unit in each computing module, the control unit acquires m matrix groups, and controls the vector operation unit and the matrix operation unit included in the same computing module to sequentially perform the first operation and the second operation corresponding to the m matrix groups, wherein the value of m is determined by the ratio of the first number of the plurality of matrix groups to the second number of the plurality of computing modules. In this way, the number of matrix groups required to be executed by each computing module is basically the same, which can realize load balancing of the plurality of computing modules inside the processor, and further improve the execution efficiency of the operator.

[0016] In an implementable manner, the matrix group includes a matrix block of matrix partitioning of the operator, and the matrix block included in the matrix group is a matrix block participating in the first operation or the second operation.

[0017] In the scheme shown in the present application, the matrix required to execute the operator can be blocked to obtain the matrix group, and the matrix block included in each matrix group is a matrix block participating in the first operation or the second operation. In this way, on the one hand, the invalid calculation in the process of executing the operator on the matrix can be reduced, and on the other hand, the calculation amount can be determined according to the number of matrix groups, and then the load balancing of the plurality of computing modules can be realized according to the calculation amount.

[0018] In a second aspect, a processor is provided, which is configured with a control unit, at least one vector operation unit and at least one matrix operation unit. The control unit is configured to: acquire a plurality of matrix groups to be executed on an operator, wherein the operator includes at least one first operation and at least one second operation, the first operation is executed by the vector operation unit, and the second operation is executed by the matrix operation unit. The control unit controls the vector operation unit and the matrix operation unit to perform the first operation and the second operation included in the operator on the plurality of matrix groups in batches to obtain an execution result of executing the operator on each matrix group. Wherein, when the vector operation unit and the matrix operation unit perform the first operation and the second operation for the Nth time, at least one vector operation unit performs the first operation on at least one first matrix group, and at least one matrix operation unit performs the second operation on at least one second matrix group, the first matrix group and the second matrix group are different matrix groups, N is greater than 1 and less than M, and M is the total batch number of performing the first operation and the second operation.

[0019] In an implementable manner, the at least one first operation and the at least one second operation included in the operator are divided into different operation sets, and there is a data dependency relationship between two operation sets adjacent in the execution sequence, one of the two operation sets being a first operation set including the first operation and the other being a second operation set including the second operation.

[0020] In an implementable manner, the control unit is configured to: control the vector operation unit and the matrix operation unit to perform, in batches, the first operation set and the second operation set included in the operator on the plurality of matrix groups to obtain an execution result of the operator executed on each matrix group. In the Nth time of performing the first operation set and the second operation set by the vector operation unit and the matrix operation unit, at least one vector operation unit performs the first operation set on at least one first matrix group, and at least one matrix operation unit performs the second operation set on at least one second matrix group. The number of the first operation set and the second operation set performed in parallel by the vector operation unit and the matrix operation unit is X, where X is greater than 1 and less than or equal to Y, Y being the minimum value in N and Z, and Z being the number of the first operation set and the second operation set divided by the operator.

[0021] In an implementable manner, the control unit is further configured to: in response to the synchronization instruction, control the vector operation unit and the matrix operation unit to perform, in parallel, the next batch of operation sets after each of the operation sets performed in parallel is completed.

[0022] In an implementable manner, the processor is configured with a plurality of computing modules, and each computing module includes a control unit, a vector operation unit, and a matrix operation unit. For the control unit in each computing module, the control unit is configured to: obtain m matrix groups, and control the vector operation unit and the matrix operation unit included in the same computing module to sequentially perform the first operation and the second operation corresponding to the m matrix groups, where the value of m is determined by the ratio of the first number of the plurality of matrix groups to the second number of the plurality of computing modules.

[0023] In an implementable manner, the matrix group includes a matrix block divided by the matrix of the operator, and the matrix block included in the matrix group is a matrix block participating in the first operation or the second operation.

[0024] In a third aspect, a computer device is provided, which includes a memory and at least one processor as described in the second aspect above. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 FIG. 1 is a structural schematic diagram of a processor provided by an embodiment of the present application;

[0026] Figure 2 FIG. 2 is a flowchart of an execution method of an operator provided by an embodiment of the present application;

[0027] Figure 3 is an execution flow schematic diagram of an operator provided by an embodiment of the present application;

[0028] Figure 4 is an execution flow schematic diagram of an operation set provided by an embodiment of the present application;

[0029] Figure 5 is an execution flow schematic diagram of an operation set provided by an embodiment of the present application;

[0030] Figure 6 is an execution flow schematic diagram of an operation set provided by an embodiment of the present application;

[0031] Figure 7 is a schematic diagram of Flash Attention calculation provided by an embodiment of the present application;

[0032] Figure 8 is a schematic diagram of Flash Attention calculation provided by an embodiment of the present application;

[0033] Figure 9 is a schematic diagram of Flash Attention calculation provided by an embodiment of the present application;

[0034] Figure 10 is a schematic diagram of a computer device structure provided by an embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0036] Operator (OP): a deep learning algorithm is composed of a plurality of calculation units, which are referred to as operators. Among them, an operator including only matrix operation can be referred to as a cube operator, an operator including only vector operation can be referred to as a vector operator, and an operator including both matrix operation and vector operation can be referred to as a mix operator.

[0037] NPU is a processor specially designed for accelerating neural network calculation, which can execute the above operators. The NPU can include a plurality of AI computing modules (AI cores), and each AI core can include a control unit (such as a Scalar unit), a vector operation unit, a matrix operation unit and a storage unit. Among them, the vector operation unit can be used to execute vector operation, and the matrix operation unit can be used to execute matrix operation.

[0038] Currently, most NPUs are coupled architectures, that is, the vector operation unit and the matrix operation unit included in the AI core of the NPU are coupled in the hardware architecture, and there are some shared hardware resources. In this way, the vector operation unit and the matrix operation unit included in the AI core cannot perform parallel vector operation and matrix operation. The current development of Mix operators is mainly based on NPUs with coupled architectures, so in the process of executing the Mix operator, the current AI core needs to sequentially execute the vector operation and the matrix operation included in the Mix operator. In this way, when the input of the Mix operator is multiple matrix groups, the AI core needs to sequentially execute each vector operation and each matrix operation in the Mix operator corresponding to each matrix group, thereby causing low execution efficiency of the Mix operator.

[0039] In some embodiments, the control unit needs to call a synchronization instruction to control the sequential execution of the vector operation and the matrix operation by the vector operation unit and the matrix operation unit. In this way, in the process of controlling the vector operation unit and the matrix operation unit to execute the Mix operator, a large number of synchronization instructions need to be called to ensure the sequential execution of the vector operation and the matrix operation in the Mix operator, thereby also affecting the execution efficiency of the Mix operator.

[0040] With the development of computing technology, NPUs with separated architectures have begun to be applied, and the vector operation unit and the matrix operation unit included in the NPU with separated architectures can perform parallel vector operation and matrix operation. Therefore, there is still room for further optimization in the execution method of developing Mix operators based on NPUs with coupled architectures.

[0041] Figure 1 is a structural schematic diagram of a processor provided by an embodiment of the present application. Referring to Figure 1 The processor 100 includes a control unit 110, a vector operation unit 120, and a matrix operation unit 130. The vector operation unit 120 and the matrix operation unit 130 included in the processor 100 can be separated architectures, that is, while the vector operation unit 120 is executing vector operation, the matrix operation unit 130 can execute matrix operation.

[0042] The processor 100 can be used to execute a Mix operator, and the Mix operator can be a fast attention (FlashAttention). In the embodiment of the present application, the processor 100 is not limited to an NPU, but can also be a central processing unit (CPU), a graphics processing unit (GPU), etc.

[0043] The processor 100 provided by the embodiments of the present application can be used to execute the execution method of the operator provided by the embodiments of the present application, and can realize accelerated execution of the Mix operator. For example, the control unit 110 of the processor 100 can obtain a plurality of matrix groups corresponding to the Mix operator to be executed, and control the vector operation unit and the matrix operation unit to perform vector operations and matrix operations involved in different matrix groups in parallel, thereby realizing compression of the time for the vector operation unit and the matrix operation unit to execute the Mix operator, and improving the execution efficiency of the Mix operator.

[0044] Figure 2 is a flowchart of an execution method of an operator provided by the embodiments of the present application. The method can be controlled and executed by the control unit included in the processor shown in the present application, and the method is described in detail with reference to Figure 2 The method comprises the following steps.

[0045] In step 201, the control unit obtains a plurality of matrix groups of an operator to be executed. The operator includes at least one first operation and at least one second operation. The first operation is executed by the vector operation unit, and the second operation is executed by the matrix operation unit.

[0046] The operator to be executed is a Mix operator. The first operation refers to an operation executed by the vector operation unit, including vector operations and matrix operations that need to be converted into vector operations to be implemented. The second operation refers to an operation executed by the matrix operation unit, such as a matrix multiplication operation. The number of the first operation and the second operation included in the operator is not limited. In terms of execution sequence, the first operation can be executed after the second operation is executed, or the second operation can be executed after the first operation is executed. There is a data dependency relationship between the first operation and the second operation.

[0047] In an example, the plurality of matrix groups can be matrix groups of operators executed respectively, and each matrix group can include at least two input matrices of the operator to be executed. In another example, the plurality of matrix groups can be obtained after block processing of at least two input matrices of the operator to be executed, and each matrix group can include matrix blocks obtained after block processing of each input matrix. The execution result of the plurality of matrix groups executing the operator can constitute the execution result of the operator on the at least two input matrices.

[0048] In step 202, the control unit controls the vector operation unit and the matrix operation unit to execute the first operation and the second operation included in the operator in batches on the plurality of matrix groups, to obtain the execution result of the operator on each matrix group.

[0049] The first operation and the second operation are performed by the at least one vector operation unit on at least one first matrix group and by the at least one matrix operation unit on at least one second matrix group, the first matrix group and the second matrix group are different matrix groups, N is greater than 1 and less than M, and M is a total batch number of the first operation and the second operation.

[0050] In an implementation, there is a data dependency relationship between the first operation and the second operation corresponding to the same matrix group, that is, output data of the first operation is input data of the second operation, or output data of the second operation is input data of the first operation. Therefore, the first operation and the second operation corresponding to the same matrix group cannot be executed in parallel. In this way, whether the first operation and the second operation included in each matrix group are executed one by one or the first operation and the second operation included in multiple matrix groups are executed synchronously, the vector operation unit or the matrix operation unit is in an idle state. For example, when the first operation of multiple matrix groups is executed synchronously, multiple matrix operation units are in an idle state, and when the second operation of multiple matrix groups is executed synchronously, multiple vector operation units are in an idle state. Therefore, in the embodiment of the present application, the control unit can control the vector operation unit and the matrix operation unit to execute the first operation and the second operation corresponding to different matrix groups in batches. That is, when the control unit controls the vector operation unit to execute the first operation corresponding to one matrix group (the first matrix group), the control unit controls the matrix operation unit to execute the second operation corresponding to another matrix group (the second matrix group) at the same time. In this way, the vector operation unit and the matrix operation unit of the processor can be in a working state at the same time, and the efficiency of the processor in executing the operator can be improved.

[0051] Because there is a data dependency relationship between the first operation and the second operation, in the process of executing the first operation and the second operation corresponding to each matrix group in batches, the operation executed in the first batch can be the first operation (which can be the first operation or the second operation) corresponding to one or more matrix groups included in the operator. In the subsequent batches, the next operation corresponding to the operation executed on the matrix group in the previous batch can be executed, and the same operation as in the previous batch can be executed on other matrix groups. In this way, parallel execution of different operations of different matrix groups can be realized at least after the first batch. The different operations corresponding to each matrix group are described exemplarily as follows:

[0052] In the control of the first batch of operations, the first operation corresponding to the first matrix group can be executed first. In the control of the second batch of operations, the second operation of the first matrix group and the first operation of the second matrix group can be controlled to be executed in parallel. In the control of the third batch of operations, the third operation of the first matrix group, the second operation of the second matrix group, and the first operation of the third matrix group can be controlled to be executed in parallel. By analogy, the first operation and the second operation of different order positions in the operators corresponding to different matrix groups can be controlled to be executed in parallel at the same time.

[0053] Figure 3 is an execution flow diagram of an operator provided by an embodiment of the present application. In Figure 3 , the execution order of the operations that the operator can include can be the first operation 1, the second operation 1, the first operation 2, and the second operation 2. As Figure 3 indicated, in the process of controlling the matrix operation unit and the vector operation unit to execute the operations corresponding to each matrix group in batches, the control unit can control the vector operation unit to execute the first operation 1 of the first matrix group in the first batch of operations.

[0054] In the first batch of operations, the vector operation unit can be controlled to execute the first operation 1 of the first matrix group.

[0055] In the second batch of operations, the vector operation unit can be controlled to execute the first operation 1 of the second matrix group. In addition, since the first operation 1 of the first matrix group has been completed, the matrix operation unit can also be controlled to execute the second operation 1 of the first matrix group in parallel.

[0056] In the third batch of operations, the vector operation unit can be controlled to execute the first operation 1 of the third matrix group. In addition, since the first operation 1 of the second matrix group has been completed and the second operation 2 of the first matrix group has been completed, the matrix operation unit can also be controlled to execute the second operation 1 of the second matrix group in parallel, and the vector operation unit can be controlled to execute the first operation 2 of the first matrix group in parallel.

[0057] In the fourth batch of operations, the vector operation unit can be controlled to execute the first operation 1 of the fourth matrix group. In addition, since the first operation 1 of the third matrix group has been completed, since the second operation 1 of the second matrix group has been completed, and since the first operation 2 of the first matrix group has been completed, the matrix operation unit can also be controlled to execute the second operation 1 of the third matrix group in parallel, the vector operation unit can be controlled to execute the first operation 2 of the second matrix group in parallel, and the matrix operation unit can be controlled to execute the second operation 2 of the first matrix group in parallel.

[0058] In the operation of the 5th batch to the nth batch, the vector operation unit can be controlled to perform the first operation 1 of the mth matrix group, the matrix operation unit can be controlled to perform the second operation 1 of the (m-1)th matrix group, the matrix operation unit can be controlled to perform the first operation 2 of the (m-2)th matrix group, and the matrix operation unit can be controlled to perform the second operation 2 of the (m-3)th matrix group. Wherein, n is the number of matrix groups, and m is the current execution batch.

[0059] In the operation of the (n+1)th batch, since the first operation 1 corresponding to each matrix group is completed, the matrix operation unit can be controlled to perform the second operation 1 of the nth matrix group, the matrix operation unit can be controlled to perform the first operation 2 of the (n-1)th matrix group, and the matrix operation unit can be controlled to perform the second operation 2 of the (n-2)th matrix group.

[0060] In the operation of the (n+2)th batch, since the first operation 1 and the second operation 1 corresponding to each matrix group are completed, the matrix operation unit can be controlled to perform the first operation 2 of the nth matrix group, and the matrix operation unit can be controlled to perform the second operation 2 of the (n-1)th matrix group.

[0061] In the operation of the (n+3)th batch, since the first operation 1, the second operation 1 and the first operation 2 corresponding to each matrix group are completed, the matrix operation unit can be controlled to perform the second operation 2 of the nth matrix group.

[0062] From the above Figure 3 It can be seen that the execution method of the operator provided in the present application can realize the parallel execution of the first operation and the second operation included in the operator by the vector operation unit and the matrix operation unit, can compress the overall time length of the execution of the first operation and the second operation on the plurality of matrix groups respectively, and further realize the improvement of the execution efficiency of the operator.

[0063] In an implementable manner, in order to further improve the execution efficiency of the operator, the execution order of the first operation and the second operation in the operator can be adjusted, that is, at least one first operation and at least one second operation included in the operator can be divided into a plurality of operation sets which are executed in sequence before the operator is executed. Wherein, the first operation or the second operation included in the same operation set is executed by the same vector operation unit and matrix operation unit. The rules for dividing the operation set can include:

[0064] Rule 1: only the first operation or only the second operation is included in the same operation set (the operation set including only the first operation can be referred to as the first operation set, and the operation set including only the second operation can be referred to as the second operation set);

[0065] Rule 2: in the execution order, the output data of one operation set is the input data of the other operation set;

[0066] Rule 3, in the execution sequence, two adjacent operation sets, one operation set is the first operation set, the other operation set is the second operation set.

[0067] After the first operation and the second operation included in the operator are divided into operation sets according to the above rules, the flow of the control unit executing the first operation and the second operation corresponding to the same matrix group becomes:

[0068] In the case where the first operation of the operator is the first operation, the control unit first controls the vector operation unit to execute each first operation satisfying the execution condition, and then controls the matrix operation unit to execute each second operation satisfying the execution condition, and then controls the vector operation unit to execute each first operation satisfying the execution condition, and so on, to alternately execute the first operation and the second operation until each first operation and each second operation included in the operator is executed.

[0069] In the case where the first operation of the operator is the second operation, the control unit first controls the vector operation unit to execute each second operation satisfying the execution condition, and then controls the matrix operation unit to execute each first operation satisfying the execution condition, and then controls the vector operation unit to execute each second operation satisfying the execution condition, and so on, to alternately execute the second operation and the first operation until each first operation and each second operation included in the operator is executed.

[0070] The execution condition is that the input data of the first operation or the second operation is stored in the memory of the processor, and the input data of the first operation or the second operation can be a matrix block stored in the memory, or the output data of the previous first operation or second operation corresponding to the first operation or the second operation stored in the memory. The alternately executed first operation and second operation belong to different operation sets, and different first operation sets and second operation sets corresponding to the same matrix group can be in different execution batches respectively. Hereinafter, the execution of the operation set provided by the embodiment of the present application is illustrated by taking Figure 4 and Figure 5 as examples.

[0071] Figure 4 is a schematic diagram of the division of the operation set provided by the embodiment of the present application. As shown in Figure 4As shown, the operator includes sequentially executed cube_0, cube_1, vector_0, cube_2, vector_1, cube_3 and cube_4, wherein vector_0 and vector_1 are first operations, and cube_0, cube_1, cube_2, cube_3 and cube_4 are second operations. Cube_0 and vector_0 have a data dependency relationship, cube_1 and vector_0 have a data dependency relationship, vector_0 and cube_2 have a data dependency relationship, vector_0 and vector_1 have a data dependency relationship, vector_1 and cube_3 have a data dependency relationship, and cube_3 and cube_4 have a data dependency relationship.

[0072] After the plurality of first operations and second operations are divided according to the above rules, three operation sets, namely cube_block0, vector_block0 and cube_block1, are sequentially executed, wherein cube_block0 includes cube_0 and cube_1, vector_block0 includes vector_0 and vector_1, and cube_block1 includes cube_2, cube_3 and cube_4. That is, the execution order of the first operations and the second operations included in the operator is changed to cube 0, cube 1, vector 0, vector 1, cube 2, cube 3 and cube 4.

[0073] After the operation sets are thus divided, for the execution process of the operator corresponding to the same group of matrix blocks, the number of switching between the vector operation unit and the matrix operation unit can be reduced. The switching between the vector operation unit and the matrix operation unit refers to that the vector operation unit executes the first operation and then the matrix operation unit executes the second operation, or the matrix operation unit executes the second operation and then the vector operation unit executes the first operation. Reducing the number of switching between the vector operation unit and the matrix operation unit can reduce the number of data transfer between different operation units and the number of synchronous instruction calls, and can improve the execution efficiency of the operator. As shown in FIG. 3, before the operation sets are divided, cube_0 to cube_4 need to be switched at least 4 times, and after the operation sets are divided, cube_0 to cube_4 need to be switched 2 times. Figure 4 As shown, before the operation sets are divided, cube_0 to cube_4 need to be switched at least 4 times, and after the operation sets are divided, cube_0 to cube_4 need to be switched 2 times.

[0074] In the embodiments of the present application, after the first operation set and the second operation set are divided from the first operation and the second operation included in the operator, the control unit can also control the parallel execution of the first operation set and the second operation set corresponding to different groups of matrix blocks according to the manner of step 202. That is, after the first operation set and the second operation set are divided from the operator, the processing of step 202 can be replaced by:

[0075] The control unit controls the vector operation unit and the matrix operation unit to perform the first operation set and the second operation set included in the operator on the plurality of matrix groups in batches to obtain the execution result of the operator executed on each group of matrix groups. Wherein, when the vector operation unit and the matrix operation unit perform the first operation set and the second operation set for the Nth time, at least one vector operation unit performs the first operation set on at least one first matrix group, and at least one matrix operation unit performs the second operation set on at least one second matrix group.

[0076] Since there is a data dependency relationship between the first operation set and the second operation set corresponding to the same group of matrix blocks, the first operation set and the second operation set corresponding to the same matrix group cannot be executed in parallel. Therefore, in the embodiments of the present application, the control unit can control the vector operation unit and the matrix operation unit to execute the first operation set and the second operation set corresponding to different matrix groups respectively. That is, the control unit controls the vector operation unit to execute the first operation set corresponding to one matrix group (the first matrix group) while controlling the matrix operation unit to execute the second operation set corresponding to another matrix group (the second matrix group) at the same time. In this way, the vector operation unit and the matrix operation unit of the processor can be in working state at the same time, which can improve the efficiency of the processor executing the operator.

[0077] In an example, when the vector operation unit and the matrix operation unit perform the first operation set and the second operation set for the Nth time, the number of the first operation set and the second operation set executed in parallel by the vector operation unit and the matrix operation unit is X, wherein X is greater than 1 and less than or equal to Y, Y is the minimum value in N and Z, and Z is the number of the first operation set and the second operation set divided from the operator.

[0078] Due to the existence of data dependency relationship, multiple operation sets cannot be executed in parallel for the same matrix group. Therefore, in each execution batch, the operation sets executed in parallel are different operation sets corresponding to different matrix groups. Therefore, the maximum parallel degree of the operation sets executed is theoretically Z or an integer multiple of Z. In addition, since the input data of part of the operation sets of part of the matrix groups has not been calculated in the process of executing the first operation set and the second operation set for the first Z-1 times, the maximum parallel degree of the operation sets executed in the process of executing the first operation set and the second operation set for the first Z-1 times is the number of execution batches N, or an integer multiple of N.

[0079] In an example, the method of controlling the vector operation unit and the matrix operation unit to perform the first operation set and the second operation set corresponding to each group of matrix blocks can include:

[0080] The control unit controls the vector operation unit or the matrix operation unit to sequentially perform the first operation set corresponding to each group of matrix blocks. During each execution of the first operation set, if the input data of the other operation set after the first operation set of the matrix group has been calculated, the first operation set and the other operation set of the other group are executed in parallel. Figure 5 is a schematic diagram of an execution flow of an operation set provided by an embodiment of the present application. In Figure 5 , the execution order of the operation set that the operator can include can be the first operation set 1, the second operation set 1, the first operation set 2, and the second operation set 2. As Figure 5 indicated, the control unit controls the matrix operation unit and the vector operation unit to perform each operation set corresponding to each matrix group in batches, including:

[0081] In the first batch operation, the vector operation unit can be controlled to perform the first operation set 1 of the first matrix group.

[0082] In the second batch operation, the vector operation unit can be controlled to perform the first operation set 1 of the second matrix group. In addition, since the first operation set 1 of the first matrix group has been completed, the matrix operation unit can also be controlled to perform the second operation set 1 of the first matrix group in parallel.

[0083] In the third batch operation, the vector operation unit can be controlled to perform the first operation set 1 of the third matrix group. In addition, since the first operation set 1 of the second matrix group has been completed, and the second operation set 2 of the first matrix group has been completed, the matrix operation unit can also be controlled to perform the second operation set 1 of the second matrix group in parallel, and the vector operation unit can be controlled to perform the first operation set 2 of the first matrix group in parallel.

[0084] In the fourth batch operation, the vector operation unit can be controlled to perform the first operation set 1 of the fourth matrix group. In addition, since the first operation set 1 of the third matrix group has been completed, since the second operation set 1 of the second matrix group has been completed, and since the first operation set 2 of the first matrix group has been completed, the matrix operation unit can also be controlled to perform the second operation set 1 of the third matrix group in parallel, the vector operation unit can be controlled to perform the first operation set 2 of the second matrix group in parallel, the matrix operation unit can be controlled to perform the second operation set 2 of the first matrix group in parallel, and the vector operation unit can be controlled to perform the first operation set 1 of the first matrix group in parallel.

[0085] In the operation of the 5th batch to the nth batch, the vector operation unit can be controlled to perform the first operation set 1 of the mth matrix group, the matrix operation unit can be controlled to perform the second operation set 1 of the (m-1)th matrix group, the matrix operation unit can be controlled to perform the first operation set 2 of the (m-2)th matrix group, and the matrix operation unit can be controlled to perform the second operation set 2 of the (m-3)th matrix group. Wherein, n is the number of matrix groups, and m is the current execution batch.

[0086] In the operation of the (n+1)th batch, since the first operation set 1 corresponding to each matrix group is completed, the matrix operation unit can be controlled to perform the second operation set 1 of the nth matrix group, the matrix operation unit can be controlled to perform the first operation set 2 of the (n-1)th matrix group, and the matrix operation unit can be controlled to perform the second operation set 2 of the (n-2)th matrix group.

[0087] In the operation of the (n+2)th batch, since the first operation set 1 and the second operation set 1 corresponding to each matrix group are completed, the matrix operation unit can be controlled to perform the first operation set 2 of the nth matrix group, and the matrix operation unit can be controlled to perform the second operation set 2 of the (n-1)th matrix group.

[0088] In the operation of the (n+3)th batch, since the first operation set 1, the second operation set 1 and the first operation set 2 corresponding to each matrix group are completed, the matrix operation unit can be controlled to perform the second operation set 2 of the nth matrix group.

[0089] It can be seen that, by using the operator execution method provided in the embodiments of the present application, the n matrix groups can be divided into n+Z execution batches, wherein Z is the number of operation sets divided for the operator. In each execution batch, the matrix operation unit and the vector operation unit can be controlled to execute the operation set included in the operator according to the maximum parallelism, so that the execution time of the operator can be compressed, and the execution efficiency of the operator can be improved.

[0090] In an implementable manner, the control unit can add a synchronization instruction between each batch of the operator, and after each batch is calculated, the next batch of calculation is executed. In this way, the number of uses of the synchronization instruction can be reduced, and the execution efficiency can be improved. Moreover, before the control unit controls the vector unit or the matrix operation unit to execute the first operation set or the second operation set, the control unit can allocate storage space for the output data obtained by executing the first operation set and the second operation set in the processor each time, for example, two storage spaces, including L2 Cache0 and L2 Cache1, are allocated to store the output data of the vector operation unit executing the first operation set and the output data of the matrix operation unit executing the second operation set, respectively.

[0091] Figure 6 is Figure 4 On the basis of dividing the operation set, the flowchart of executing the operator for multiple matrix groups. As shown in FIG. 4, the operator is divided into two operation sets, and the two operation sets are executed for the matrix groups.Figure 6 As shown, the execution of the whole operator is divided into n+2 times, including:

[0092] loopO, cube_blockO of the first group of matrices is executed by a cube operation unit. The output data obtained is stored to L2 CacheO as the input of the next operation. After the operation included in loopO is completed, the operation of loop 1 can be executed by calling the synchronization instruction.

[0093] loop 1, cube_blockO of the second group of matrices is executed by a cube operation unit, and vector_blockO of the first group of matrices is executed by a vector operation unit. The output data obtained by the vector operation unit is stored to L2 CacheO, and the output data obtained by the cube operation unit is stored to L2 Cache 1 as the input of the next operation. After the operation included in loop 1 is completed, the operation of loop 2 can be executed by calling the synchronization instruction.

[0094] loop 2, cube_blockO of the third group of matrices is executed by a cube operation unit, vector_blockO of the second group of matrices is executed by a vector operation unit, and cube_blockl of the first group of matrices is executed by a cube operation unit. The output data obtained by the vector operation unit is stored to L2 Cache 1, and the output data obtained by the cube operation unit is stored to L2 CacheO as the input of the next operation. After the operation included in loop 2 is completed, the operation of loop 3 can be executed by calling the synchronization instruction.

[0095] loop 3 to loop n-1, similar to loop 2, cube_blockO of the i-th group of matrices is executed by a cube operation unit, vector_blockO of the (i-1)-th group of matrices is executed by a vector operation unit, and cube_blockl of the (i-2)-th group of matrices is executed by a cube operation unit. Wherein, i is the number of times of the operation currently executed. After each operation is completed, the output data obtained by the vector operation unit and the cube operation unit is stored to L2 CacheO and L2 Cache 1 respectively as the input of the next operation. After each operation is completed, the operation of the next loop can be executed by calling the synchronization instruction.

[0096] loop n, the cube_block0 of the n-1th group of matrices is executed by a cube operation unit, and the vector_block0 of the nth group of matrices is executed by a vector operation unit. The output data obtained by the vector operation unit is stored to the L2Cache0, and the output data obtained by the cube operation unit is stored to the L2Cache1 as the input of the next operation. After the operation included in the loop n is completed, the operation of the loop n+1 can be called by invoking a synchronization instruction.

[0097] loop n+1, the cube_block1 of the nth group of matrices is executed by a cube operation unit to obtain the result data of the last operation set of the last group of matrices.

[0098] In the loop n, the cube_block0 of the n-1th group of matrices is executed by a cube operation unit, and the vector_block0 of the nth group of matrices is executed by a vector operation unit. The output data obtained by the vector operation unit is stored to the L2Cache0, and the output data obtained by the cube operation unit is stored to the L2Cache1 as the input of the next operation. After the operation included in the loop n is completed, the operation of the loop n+1 can be called by invoking a synchronization instruction. Figure 6 In the loop n, the cube_block0 of the n-1th group of matrices is executed by a cube operation unit, and the vector_block0 of the nth group of matrices is executed by a vector operation unit. The output data obtained by the vector operation unit is stored to the L2Cache0, and the output data obtained by the cube operation unit is stored to the L2Cache1 as the input of the next operation. After the operation included in the loop n is completed, the operation of the loop n+1 can be called by invoking a synchronization instruction.

[0099] In an implementable manner, the processor provided in the present application is configured with a plurality of computing modules, which can be referred to as AI cores. The AI cores can include Scalar units, Vector operation units, cube operation units and storage units. The plurality of AI cores can process the operations of a plurality of groups of matrices in parallel, and the number m of groups of matrices processed by each AI core can be equal or approximately equal. For example, the plurality of groups of matrices to be executed by an operator can be evenly distributed to each AI core, so as to realize load balancing of the AI cores and improve the efficiency of the processor in executing the operator.

[0100] In an example, the matrices included in each group of matrices are effective matrix blocks obtained by dividing at least two input matrices corresponding to an operator to be executed.

[0101] In some embodiments, when allocating matrix groups to AI cores, the distinction between valid and invalid matrix blocks within a matrix group is not made. This can lead to inconsistent numbers of matrix groups containing valid matrix blocks in each AI core's allocation. Furthermore, since matrix groups containing invalid matrix blocks do not participate in actual computation, the actual computational load for each AI core may vary, potentially hindering load balancing and reducing processor efficiency. However, in this embodiment, multiple matrix groups containing valid matrix blocks can be determined first, and then matrix groups can be allocated to each AI core based on the number of such groups. This ensures a consistent computational load across AI cores, achieving load balancing. The positions of valid matrix blocks participating in the first or second operation within the matrix of the operator to be executed can be set individually based on the specific operator. The following section details the matrix group allocation and operator execution steps using FlashAttention computation as an example:

[0102] Flash Attention is a novel attention algorithm that is I / O-aware, fast, and memory-efficient. In large model training, Flash Attention plays an irreplaceable role in saving memory and improving performance, making it a crucial mix operator in AI algorithms. Figures 7 to 9 This is a schematic diagram of FlashAttention calculation provided in an embodiment of this application.

[0103] like Figure 7 As shown, in addition to the first and fourth steps, which involve dividing the query matrix, key matrix, and value matrix into query_block, key_block, and value_block, the Flash Attention process also includes the second step, which involves performing matrix multiplication on query_block and key_block through the matrix operation unit to obtain score_block; the third step, which involves performing softmax processing on score_block through the vector operation unit; the fifth step, which involves performing matrix multiplication on score_block and value_block through the vector operation unit to obtain res_block; and the sixth step, which involves performing accumulation processing on res_block and value_block through the vector operation unit.

[0104] Therefore, Flash Attention comprises four alternating first and second operations with dependencies. The matrix multiplication operations in the second and fifth steps can be represented as cube1 and cube2, respectively; the softmax processing in the third step and the accumulation processing in the sixth step can be represented as vector1 and vector2, respectively. Based on the data dependencies between cube1, cube2, vector1, and vector2, they can be divided into four operation sets, each containing either a matrix operation or a vector operation.

[0105] Depend on Figure 7 As can be seen, the query matrix, key matrix, and value matrix are the input matrices for Flash Attention. After dividing the query matrix, key matrix, and value matrix into blocks, multiple matrix groups can be formed. Each matrix group contains a query_block, a key_block, and a value_block. The multiple matrix blocks within a matrix group can be used as inputs to Flash Attention, and the functions cube1, cube2, vector1, and vector2 of Flash Attention are executed sequentially.

[0106] Figure 8 This is a schematic diagram of an effective matrix block provided in an embodiment of this application. For example... Figure 8 As shown, in the score matrix obtained by performing matrix multiplication on the query and key matrices in the second step, the score_blocks located on the diagonal and below the diagonal are the matrix blocks involved in the operation. Therefore, the query_block and key_block used to calculate the score_block in the query and key matrices, and the value_block calculated with the score_block in the value matrix, are valid matrix blocks. The query_block and key_block used to calculate the same score_block, and the score_blocks calculated with the same score_block, can form a matrix group. See also... Figure 8The score matrix can include 11 score_blocks participating in the computation. Corresponding to these 11 score_blocks, the query matrix, key matrix, and value matrix for performing Flash Attention can be divided into 11 matrix groups, each containing a query_block, a key_block, and a value_block. In one example, with processors including AI core0 and AI core1, AI core0 can perform Flash Attention on 5 matrix groups, and AI core1 can perform Flash Attention on 6 matrix groups.

[0107] Figure 9 This is a flowchart of the execution of Flash Attention provided in the embodiments of this application, such as... Figure 8 As shown, the calculation steps for FlashAttention can include:

[0108] loop0 allows the cube operation unit to execute the first matrix group, cube1.

[0109] loop1 can execute the second matrix group cube1 by a cube operation unit, and execute the first matrix group vector1 by a vector operation unit;

[0110] loop2 can execute the third matrix group cube1 by one cube operation unit, execute the second matrix group vector1 by one vector operation unit, and execute the first matrix group cube2 by one cube operation unit;

[0111] loop3 can be executed by one cube operation unit to execute the fourth matrix group cube1, another vector operation unit to execute the third matrix group vector1, another cube operation unit to execute the second matrix group cube2, and another vector operation unit to execute the first matrix group vector2.

[0112] Loops 4 through n-1 are similar to loop 2. One cube operation unit executes cube1 of the j-th matrix group, another vector operation unit executes vector1 of the (j-1)-th matrix group, another cube operation unit executes cube2 of the (j-2)-th matrix group, and another vector operation unit executes vector2 of the (j-3)-th matrix group; where j is the current execution step number.

[0113] Loop n can be executed by one vector operation unit to execute vector1 of the nth matrix group, one cube operation unit to execute cube2 of the (n-1)th matrix group, and another vector operation unit to execute vector2 of the (n-2)th matrix group;

[0114] Loop n+1 can execute cube2 of the nth matrix group by a cube operation unit, and execute vector2 of the (n-1)th matrix group by a vector operation unit.

[0115] loop n+2 allows a vector2 of the nth matrix group to be executed by a single vector operation unit.

[0116] In each execution step, the output data of the last non-operator computation set within the computation set can be stored in L2Cache0 and L2Cache1, serving as input data for the next computation set. If the last computation set of an operator exists within each execution step, its corresponding output data can be stored in another location as the computation result of a set of matrix blocks. Furthermore, the sequential execution of adjacent computation steps can be controlled by the control unit calling synchronization instructions.

[0117] As can be seen, by adopting the embodiments of this application, the time for performing Flash Attention on multiple matrix groups can be reduced, thereby improving the execution efficiency of Flash Attention.

[0118] Based on the same inventive concept, embodiments of this application also provide a processor, which can perform as follows: Figure 1 As shown, the processor is configured with a control unit, at least one vector operation unit, and at least one matrix operation unit. The control unit is used to: acquire multiple matrix groups of operators to be executed, wherein the operator includes at least one first operation and at least one second operation, the first operation being executed by the vector operation unit and the second operation by the matrix operation unit; and control the vector operation unit and the matrix operation unit to execute the first and second operations included in the operator on the multiple matrix groups in batches, obtaining the execution result of the operator for each matrix group. Specifically, when the vector operation unit and the matrix operation unit execute the first and second operations in the Nth iteration, at least one vector operation unit executes the first operation on at least one first matrix group, and at least one matrix operation unit executes the second operation on at least one second matrix group. The first and second matrix groups are different matrix groups, N is greater than 1 and less than M, and M is the total number of batches executing the first and second operations.

[0119] In one feasible approach, at least one first operation and at least one second operation included in the operator are divided into different operation sets, wherein two operation sets that are adjacent in execution order are, respectively, a first operation set including the first operation and a second operation set including the second operation, and the two operation sets have a data dependency relationship.

[0120] In one feasible implementation, the control unit is used to: control the vector operation unit and the matrix operation unit to execute the first operation set and the second operation set included in the operator on multiple matrix groups in batches, and obtain the execution result of the operator on each matrix group. Specifically, when the vector operation unit and the matrix operation unit execute the first operation set and the second operation set for the Nth time, at least one vector operation unit executes the first operation set on at least one first matrix group, and at least one matrix operation unit executes the second operation set on at least one second matrix group. The number of first operation sets and second operation sets executed in parallel by the vector operation unit and the matrix operation unit is X, where X is greater than 1 and less than or equal to Y, Y is the minimum value between N and Z, and Z is the number of first operation sets and second operation sets divided by the operator.

[0121] In one possible implementation, the control unit is also configured to: in response to a synchronization instruction, control the vector operation unit and the matrix operation unit to run the next batch of operation sets in parallel after each set of operations executed in parallel has been completed.

[0122] In one possible implementation, the processor is configured with multiple computing modules, each including a control unit, a vector operation unit, and a matrix operation unit. For the control unit in each computing module, the control unit is used to: acquire m matrix groups, and control the vector operation units and matrix operation units within the same computing module to sequentially execute the first and second operations corresponding to the m matrix groups, wherein the value of m is determined by the ratio of the first number of matrix groups to the second number of computing modules.

[0123] In one possible implementation, the matrix group includes matrix blocks partitioned by the matrix operator, wherein the matrix blocks included in the matrix group are matrix blocks that participate in the first operation or the second operation.

[0124] The processor implementation of the operator computation provided in this application belongs to the same inventive concept as the operator execution method shown in the above method embodiments. The specific implementation process is detailed in the above method embodiments and will not be repeated here. Using the processor provided in this application, the control unit can control the first and second operations of some matrix groups to be executed in parallel while controlling the vector operation unit and matrix operation unit to execute the first and second operations corresponding to each group of matrix blocks. This can compress the overall time of the first and second operations corresponding to multiple groups of matrix blocks, thereby improving the execution efficiency of the operator.

[0125] This application also provides a computer device. Figure 10 This is a schematic diagram of a computer device provided in an embodiment of this application, such as... Figure 10 As shown, computer device 1000 may include: bus 1002, processor 1004, and memory 1006. Optionally, computer device 1000 may also include communication interface 1008. The processor 1004, memory 1006, and communication interface 1008 communicate via bus 1002. Computer device 200 may be a server or terminal device.

[0126] It should be understood that this application does not limit the number of processors and memories in the computer device 1000. The computer device 1000 can be a terminal or a server. When the computer device 1000 is a terminal, it includes, but is not limited to, desktop computers, mobile phones, laptops, tablets, etc. When the computer device 1000 is a server, it can be a single server or a server cluster composed of multiple servers. It can be a physical machine or a virtual machine, container, etc., virtualized through virtualization technology.

[0127] Bus 1002 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 10 The bus 1002 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 1002 may include a path for transmitting information between various components of the computer device 1000 (e.g., memory 1006, processor 1004, communication interface 1008).

[0128] Processor 1004 can be used as follows Figure 1The processor 100 shown includes a control unit 110, a vector operation unit 120, and a matrix operation unit 130. The processor 100 can be used to execute the operator execution method provided in the embodiments of this application. The processor 1004 is not limited to an NPU, central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP), or digital signal processor (DSP), etc.

[0129] The memory 1006 may include volatile memory, such as random access memory (RAM). The memory 1006 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD). All of the above-mentioned memories can be referred to as global memory.

[0130] The communication interface 1008 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computer device 1000 and other devices or communication networks.

[0131] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computer device or stored on any usable medium. When the computer program product is run on at least one computer device, it causes the at least one computer device to execute the operator execution method provided in this application embodiment.

[0132] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computer device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct a computer device to execute the execution method of the operator provided in this application embodiment.

[0133] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items that have substantially the same function and purpose. It should be understood that there is no logical or temporal dependency between "first" and "second," nor does it limit the quantity or order of execution. It should also be understood that although the following description uses the terms "first," "second," etc., to describe various elements, these elements should not be limited by the terms. These terms are merely used to distinguish one element from another. For example, without departing from the scope of various examples, a first operation can be referred to as a second operation, and similarly, a second operation can be referred to as a first operation. Both first and second operations can be collectively referred to as operations, and in some cases, they can be separate and distinct operations.

[0134] In this application, the term "at least one" means one or more, and the term "multiple" means two or more.

[0135] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for executing an operator, characterized in that, The method is applied to a processor, the processor being configured with a control unit, at least one vector operation unit, and at least one matrix operation unit, the method comprising: The control unit acquires multiple matrix groups of operators to be executed, wherein the operator includes at least one first operation and at least one second operation, the first operation is executed by the vector operation unit, and the second operation is executed by the matrix operation unit; The control unit controls the vector operation unit and the matrix operation unit to perform the first and second operations included in the operator on multiple matrix groups in batches, obtaining the execution result of the operator on each matrix group; wherein, When the vector operation unit and the matrix operation unit perform the first operation and the second operation for the Nth time, the at least one vector operation unit performs the first operation on at least one first matrix group, and the at least one matrix operation unit performs the second operation on at least one second matrix group. The first matrix group and the second matrix group are different matrix groups. N is greater than 1 and less than M, and M is the total batch of the first operation and the second operation.

2. The method according to claim 1, characterized in that, The operator includes at least one first operation and at least one second operation, which are divided into different operation sets. In two operation sets that are adjacent in execution order, one operation set is a first operation set including the first operation and the other operation set is a second operation set including the second operation. The two operation sets have a data dependency relationship.

3. The method according to claim 2, characterized in that, The control unit controls the vector operation unit and the matrix operation unit to perform the first operation and the second operation included in the operator on multiple matrix groups in batches, obtaining the execution result of the operator on each matrix group; wherein, when the vector operation unit and the matrix operation unit perform the first operation and the second operation for the Nth time, the at least one vector operation unit performs the first operation on at least one first matrix group, and the at least one matrix operation unit performs the second operation on at least one second matrix group, wherein the first matrix group and the second matrix group are different matrix groups, N is greater than 1 and less than M, and M is the total batch of the first operation and the second operation, including: The control unit controls the vector operation unit and the matrix operation unit to execute the first operation set and the second operation set included in the operator on multiple matrix groups in batches, and obtains the execution result of the operator on each matrix group; wherein... When the vector operation unit and the matrix operation unit execute the first operation set and the second operation set for the Nth time, the at least one vector operation unit executes the first operation set on at least one first matrix group, and the at least one matrix operation unit executes the second operation set on at least one second matrix group; The number of the first and second operation sets executed in parallel by the vector operation unit and the matrix operation unit is X, wherein X is greater than 1 and less than or equal to Y, Y is the minimum value between N and Z, and Z is the number of the first and second operation sets divided by the operator.

4. The method according to claims 1 to 3, characterized in that, The method further includes: The control unit responds to the synchronization command and, after each parallel execution of a set of operations is completed, controls the vector operation unit and the matrix operation unit to perform the next batch of operations in parallel.

5. The method according to any one of claims 1 to 4, characterized in that, The processor is configured with multiple computing modules, each computing module including the control unit, the vector operation unit and the matrix operation unit; The control unit controls the vector operation unit and the matrix operation unit to perform the first operation and the second operation included in the operator on multiple matrix groups in batches, including: The control unit in each computing module acquires m matrix groups and controls the vector operation unit and the matrix operation unit included in the same computing module to sequentially execute the first operation and the second operation corresponding to the m matrix groups, wherein the value of m is determined by the ratio of the first number of multiple matrix groups to the second number of multiple computing modules.

6. The method according to any one of claims 1 to 5, characterized in that, The matrix group includes effective matrix blocks of the matrix partitioning of the execution operator. The effective matrix blocks are the matrix blocks of the execution operator that participate in the first operation or the second operation.

7. A processor for executing operators, characterized in that, The processor is configured with a control unit, at least one vector operation unit, and at least one matrix operation unit. The control unit is used for: Obtain multiple matrix groups of operators to be executed, wherein the operators include at least one first operation and at least one second operation, the first operation is executed by the vector operation unit, and the second operation is executed by the matrix operation unit; The vector operation unit and the matrix operation unit are controlled to perform the first operation and the second operation included in the operator on multiple matrix groups in batches, so as to obtain the execution result of the operator on each matrix group; wherein, When the vector operation unit and the matrix operation unit perform the first operation and the second operation for the Nth time, the at least one vector operation unit performs the first operation on at least one first matrix group, and the at least one matrix operation unit performs the second operation on at least one second matrix group. The first matrix group and the second matrix group are different matrix groups. N is greater than 1 and less than M, and M is the total batch of the first operation and the second operation.

8. The processor according to claim 7, characterized in that, The operator includes at least one first operation and at least one second operation, which are divided into different operation sets. In two operation sets that are adjacent in execution order, one operation set is a first operation set including the first operation and the other operation set is a second operation set including the second operation. The two operation sets have a data dependency relationship.

9. The processor according to claim 8, characterized in that, The control unit is used for: The control unit controls the vector operation unit and the matrix operation unit to execute the first operation set and the second operation set included in the operator on multiple matrix groups in batches, and obtains the execution result of the operator on each matrix group; wherein... When the vector operation unit and the matrix operation unit execute the first operation set and the second operation set for the Nth time, the at least one vector operation unit executes the first operation set on at least one first matrix group, and the at least one matrix operation unit executes the second operation set on at least one second matrix group; The number of the first and second operation sets executed in parallel by the vector operation unit and the matrix operation unit is X, wherein X is greater than 1 and less than or equal to Y, Y is the minimum value between N and Z, and Z is the number of the first and second operation sets divided by the operator.

10. The processor according to claims 7 to 9, characterized in that, The control unit is also used for: In response to the synchronization command, after each parallel execution of a set of operations is completed, the vector operation unit and the matrix operation unit are controlled to proceed to the next batch of operations in parallel.

11. The processor according to any one of claims 7 to 10, characterized in that, The processor is configured with multiple computing modules, each computing module including the control unit, the vector operation unit and the matrix operation unit; For each computing module, the control unit is used to: Obtain m matrix groups, and control the vector operation unit and the matrix operation unit included in the same calculation module to sequentially execute the first operation and the second operation corresponding to the m matrix groups, wherein the value of m is determined by the ratio of the first number of multiple matrix groups to the second number of multiple calculation modules.

12. The processor according to any one of claims 7 to 11, characterized in that, The matrix group includes effective matrix blocks of the matrix partitioning of the execution operator. The effective matrix blocks are the matrix blocks of the execution operator that participate in the first operation or the second operation.

13. A computer device, characterized in that, The computer device includes a memory and at least one processor as described in claims 7 to 12.