Operator scheduling method and device, equipment, storage medium and program product

Abstract

The invention discloses an operator scheduling method and device, equipment, a storage medium and a program product. The operator scheduling method comprises the following steps: dividing a target task flow into a plurality of sub-task flows; wherein the target task flow comprises a plurality of operators, each subtask flow comprises an operator slice corresponding to at least two operators respectively, and the execution sequence of each operator slice in the subtask flow is the same as the data flow sequence of the target task flow; generating target time scheduling information of each sub-task flow; wherein the target time scheduling information comprises the scheduling time of each operator slice in the sub-task flow, the scheduling time of a plurality of operator slices of different types is the same, and the scheduling time of the operator slices of the same type is different; and scheduling the operator slices in each sub-task flow according to the target time scheduling information. For a neural network accelerator adopting an airspace computing architecture, operator scheduling is carried out according to the mode, and the utilization rate of an intra-kernel arithmetic unit can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an operator scheduling method, device, equipment, storage medium and program product. Background Art

[0002] In the accelerator of the spatial computing architecture, the processor uses a two-dimensional matrix multiplication array composed of MAC (Multiply Accumulate) units as the computing core. Each MAC unit is responsible for performing multiplication and accumulation operations. This pulsating data flow calculation method enables the weight data and partial sum data to be reused through direct transmission between ALUs (Arithmetic Logic Units). At the same time, in order to reduce data interaction with the host CPU (Central Processing Unit), the accelerator integrates a normalization / pooling unit and an activation unit that completes nonlinear activation operations.

[0003] However, when reasoning about neural networks, compared with general-purpose GPUs (Graphics Processing Units), the spatial computing architecture has poor flexibility and cannot support the internal acceleration tasks of operators that can be completed by GPUs in many scenarios. It can only execute task flows sequentially, causing some computing units to be idle, resulting in a huge waste of hardware resources. Summary of the invention

[0004] The main technical problem solved by the present application is to provide an operator scheduling method, device, equipment, storage medium and program product, which can realize parallel computing of the in-core computing units for the neural network accelerator using the spatial domain computing architecture, thereby improving the utilization rate of the in-core computing units.

[0005] To solve the above technical problems, a technical solution adopted in the present application is: to provide an operator scheduling method, the method comprising: dividing a target task flow into multiple sub-task flows; wherein the target task flow includes multiple operators, each sub-task flow includes an operator slice corresponding to at least two operators respectively, and the execution order of each operator slice in the sub-task flow is the same as the data flow order of the target task flow; generating target time scheduling information for each sub-task flow; wherein the target time scheduling information includes the scheduling time of each operator slice in the sub-task flow, there are multiple different types of operator slices with the same scheduling time, and the scheduling time of the same type of operator slices is different; scheduling the operator slices in each sub-task flow according to the target time scheduling information.

[0006] Optionally, the target task flow is divided into multiple sub-task flows, including: performing loop block processing on each operator to obtain a first loop body corresponding to each operator; wherein the first loop body includes multiple first iteration loops, and each first iteration loop includes an operator slice of the operator; merging each first loop body to obtain at least one second loop body; wherein the second loop body includes multiple second iteration loops, and each second iteration loop includes an operator slice corresponding to at least two operators; and treating each second iteration loop as a sub-task flow.

[0007] Optionally, each first loop body is merged to obtain at least one second loop body, including: merging the first loop bodies that meet the preset merging conditions to obtain at least one second loop body; wherein the preset merging conditions include: the loop parameters of each first loop body are the same, and there is a data dependency relationship between the operators corresponding to each first loop body; the loop parameters include the index upper boundary, index lower boundary and index iteration step of the outer loop.

[0008] Optionally, generating target time scheduling information for each subtask flow includes: generating several first candidate scheduling mapping relationships; wherein the first candidate scheduling mapping relationships include several candidate scheduling times and operator slice sets corresponding to each candidate scheduling time, each operator slice belonging to the same subtask flow is located in a different operator slice set, the types of each operator slice in the same operator slice set are different, and the maximum parallel number of operator slices of each first candidate scheduling mapping relationship is greater than a first preset number and less than or equal to the number of types of computing units required for the target task flow; selecting a second candidate scheduling mapping relationship that meets the requirements from the several first candidate scheduling mapping relationships; for each subtask flow, using the candidate scheduling time corresponding to each operator slice of the subtask flow in the second candidate scheduling mapping relationship as the target time scheduling information of the subtask flow.

[0009] Optionally, generating a plurality of first candidate scheduling mapping relationships includes: generating a third candidate scheduling mapping relationship; and adjusting the third candidate scheduling mapping relationship to obtain a plurality of first candidate scheduling mapping relationships.

[0010] Optionally, the maximum parallel number of operator slices of the third candidate scheduling mapping relationship is equal to the number of types of computing units required for the target task flow; and / or, adjusting the third candidate scheduling mapping relationship includes: performing at least one of the following adjustment operations on the third candidate scheduling mapping relationship: adjusting the candidate scheduling time corresponding to at least one operator slice in the third candidate scheduling mapping relationship; adjusting the maximum parallel number of operator slices of the third candidate scheduling mapping relationship.

[0011] Optionally, selecting a second candidate scheduling mapping relationship that meets the requirements from several first candidate scheduling mapping relationships includes: determining the estimated performance parameters corresponding to each first candidate scheduling mapping relationship; wherein the estimated performance parameters of the first candidate scheduling mapping relationship include at least one of the maximum parallel number of operator slices, the total number of operator slice scheduling times, and the total estimated execution time of the first candidate scheduling mapping relationship; for each first candidate scheduling mapping relationship, comprehensively determining the first evaluation value of the first candidate scheduling mapping relationship by combining the estimated performance parameters of the first candidate scheduling mapping relationship; and selecting a second candidate scheduling mapping relationship from several first candidate scheduling mapping relationships based on the first evaluation value of each first candidate scheduling mapping relationship.

[0012] Optionally, based on the first evaluation value of each first candidate scheduling mapping relationship, a second candidate scheduling mapping relationship is selected from several first candidate scheduling mapping relationships, including: selecting a first candidate scheduling mapping relationship whose first evaluation value meets the requirements from several first candidate scheduling mapping relationships as the second candidate scheduling mapping relationship; or, screening out multiple fourth candidate scheduling mapping relationships whose first evaluation values ​​meet preset conditions from several first candidate scheduling mapping relationships; performing at least one run test on each fourth candidate scheduling mapping relationship to obtain at least one total test execution time corresponding to each fourth candidate scheduling mapping relationship; for each fourth candidate scheduling mapping relationship, determining the second evaluation value of the fourth candidate scheduling mapping relationship based on at least one total test execution time of the fourth candidate scheduling mapping relationship; and selecting a fourth candidate scheduling mapping relationship whose second evaluation value meets the requirements from multiple fourth candidate scheduling mapping relationships as the second candidate scheduling mapping relationship.

[0013] Optionally, a second candidate scheduling mapping relationship that meets the requirements is selected from several first candidate scheduling mapping relationships, including: performing at least one run test on each first candidate scheduling mapping relationship to obtain at least one total test execution time corresponding to each first candidate scheduling mapping relationship; for each first candidate scheduling mapping relationship, based on at least one total test execution time of the first candidate scheduling mapping relationship, determining a third evaluation value of the first candidate scheduling mapping relationship; and selecting a first candidate scheduling mapping relationship whose third evaluation value meets the requirements from several first candidate scheduling mapping relationships as the second candidate scheduling mapping relationship.

[0014] To solve the above technical problems, another technical solution adopted in the present application is: to provide an operator scheduling device, the device comprising: a division module, used to divide the target task flow into multiple sub-task flows; wherein the target task flow includes multiple operators, each of the sub-task flows includes an operator slice corresponding to at least two of the operators, and the execution order of each of the operator slices in the sub-task flow is the same as the data flow order of the target task flow; a generation module, used to generate target time scheduling information for each of the sub-task flows; wherein the target time scheduling information includes the scheduling time of each of the operator slices in the sub-task flow, there are multiple different types of operator slices with the same scheduling time, and the scheduling time of the operator slices of the same type is different; a scheduling module, used to schedule the operator slices in each of the sub-task flows according to the target time scheduling information.

[0015] To solve the above technical problems, another technical solution adopted in the present application is: to provide an electronic device, including a memory and a processor coupled to each other, the memory storing program instructions; the processor is used to execute the program instructions stored in the memory to implement the scheduling method of the above operator.

[0016] To solve the above technical problems, another technical solution adopted in the present application is: to provide a computer-readable storage medium / computer program product, wherein the computer-readable storage medium is used to store program instructions, and the program instructions can be executed by a processor to implement the scheduling method of the above operator; the computer program product includes a computer program, and the computer program can be executed by a processor to implement the scheduling method of the above operator.

[0017] The above scheme divides the target task flow into multiple subtask flows, wherein the target task flow includes multiple operators, each subtask flow includes an operator slice corresponding to at least two operators, and the execution order of each operator slice in the subtask flow is the same as the data flow order of the target task flow; generates target time scheduling information for each subtask flow, and schedules the operator slices in each subtask flow according to the target time scheduling information of each subtask flow, wherein the target time scheduling information of the subtask flow includes the scheduling time of each operator slice in the subtask flow, and there are multiple different types of operator slices with the same scheduling time, and the scheduling time of the same type of operator slices is different. For a neural network accelerator using a spatial domain computing architecture, since different types of operator slices require different computing units in the kernel to execute, scheduling in the above manner can execute multiple different types of operator slices in parallel at the same scheduling time, realize parallel computing of computing units in the kernel, and thus improve the utilization rate of computing units in the kernel. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1It is a flowchart of an embodiment of a scheduling method for operators provided in this application;

[0019] Figure 2 It is a flowchart of another embodiment of the operator scheduling method provided by the present application;

[0020] Figure 3 It is a flowchart of an embodiment of a method for dividing subtask flows provided by the present application;

[0021] Figure 4 It is a schematic diagram of the principle of generating a third candidate scheduling mapping relationship for a scaled dot product attention sub-network provided by the present application;

[0022] Figure 5 is a schematic diagram of a first candidate scheduling mapping relationship of the first type provided by the present application;

[0023] Figure 6 is a schematic diagram of a first candidate scheduling mapping relationship belonging to the second type provided by the present application;

[0024] Figure 7 It is a schematic diagram of the framework of an embodiment of a scheduling device for operators provided by the present application;

[0025] Figure 8 It is a schematic diagram of a framework of an embodiment of an electronic device provided by the present application;

[0026] Fig. 9 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and effect of the present application clearer and more specific, the present application is further described in detail below with reference to the accompanying drawings and examples.

[0028] It should be noted that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The term "many" means two or more than two; the term "several" means at least one. In addition, the terms "first", "second", etc. in the specification and claims of this article and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0029] Neural network algorithms are widely used in image recognition, speech recognition, natural language processing and other fields. Common neural networks include CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), DNN (Deep Neural Networks), etc., which are essentially a combination of a large number of linear algebra operations (such as matrix or vector multiplication and addition). Therefore, the hardware needs to have efficient linear algebra operations. At present, many computing chips, such as CPU, GPU, FPGA (Field Programmable Gate Array) and ASIC (Application Specific Integrated Circuit) neural network accelerators have been used to accelerate the calculation of neural networks. Among them, the spatial domain computing architecture is a typical representative of ASIC neural network accelerators. Its design on dedicated hardware can effectively improve the execution performance of computing tasks and control energy consumption.

[0030] The computational model of a neural network is usually modeled as a DFG (Data Flow Graph), where each node in the DFG corresponds to an operator, representing a computational closure such as matrix multiplication, and the edges describe the data dependencies between operators. Compilers usually treat operators in the DFG as opaque library functions and schedule them to accelerators for separate execution, relying on another layer of schedulers (usually implemented in hardware) to implement coarse-grained parallel computation of operators. This strategy generates a lot of scheduling overhead and usually fails to fully utilize available hardware resources.

[0031] Based on the above DFG representation, two levels of parallel scheduling can be generated at both compile time and runtime. Among them, the first level is inter-operator parallelism, where operators without data dependencies in DFG can run in parallel. The second level is intra-operator parallelism. Operators such as matrix multiplication have inherent internal data parallelism, which can be accelerated in parallel using hardware accelerators (such as GPUs) that can perform parallel computing. In order to utilize the two levels of parallelism, a two-layer scheduling method is currently used. The scheduler between operators at the DFG layer issues operators to be executed according to the data dependencies between operators, and the scheduler within the operator maps them to the parallel execution units in the accelerator. This layered design has a fundamental impact on the system architecture of the existing neural network tool chain. Taking the DNN network as an example, the DFG layer scheduler is usually implemented in deep learning frameworks such as TensorFlow or ONNXRuntime; the operator layer scheduler is usually hidden in the backend of operator libraries such as cuDNN and MKL-DNN, or directly implemented in hardware. Although this two-layer scheduling method is adopted by existing neural network frameworks and accelerators, it will bring many backend hardware limitations.

[0032] During network reasoning, the compiler will divide the entire network according to the consumption of computing resources by the neural network, so that the computing tasks of different sub-networks can be run on different cores (computing resources), thereby improving the network parallelism. However, when performing network reasoning on hardware using a spatial domain computing architecture, for computing tasks executed by a single core, since the calculation order of each operator in the task flow is executed serially, only one computing unit on the same core will be running at a time, resulting in a waste of hardware resources.

[0033] See also Figure 1 , Figure 1 It is a flow chart of an embodiment of the scheduling method of the operator provided in the present application. It should be noted that if there are substantially the same results, the method of the present application is not based on Figure 1 The process sequence shown is limited.

[0034] like Figure 1 As shown, the method comprises the following steps:

[0035] S11: Divide the target task flow into multiple subtask flows.

[0036] The target neural network includes several subnetworks, and the target task flow is one of the several subnetworks of the target neural network. Exemplarily, the target neural network is a CNN, RNN, DNN or other types of neural networks, which is not specifically limited in this embodiment.

[0037] The target task flow includes multiple operators. Operators are linear algebra operations, that is, processing of structured data (such as matrices, tensors, etc.). For example, operators can be matrix multiplication (Matmul for short), multiplication (Mul for short), matrix addition (Matrix Addition) operation, normalization operation (Softmax), etc.

[0038] It can be understood that the premise of forming a task flow is that there is a data dependency relationship between operators in the task flow. Therefore, in this embodiment, there is a data dependency relationship between multiple operators in the target task flow.

[0039] For example, if the target task flow is a scaled dot-product attention subnetwork, then the multiple operators of the target task flow include a first matrix multiplication operation, a multiplication operation, a normalization operation, and a second matrix multiplication operation. The first operation result is obtained by performing a first matrix multiplication operation on the key value Q and the key value K; the first operation result is the operand of the multiplication operation, and the second operation result is obtained by performing a multiplication operation (corresponding to the scaling step in the scaled dot-product attention); the second operation result is the operand of the normalization operation, and the third operation result is obtained by performing a normalization operation on the second operation result; the third operation result is the operand of the second matrix multiplication operation, and the third operation result and the key value V are subjected to a second matrix multiplication operation to obtain the final operation result. That is, there is a data dependency relationship between the first matrix multiplication operation, the multiplication operation, the normalization operation, and the second matrix multiplication operation, and the first matrix multiplication operation, the multiplication operation, the normalization operation, and the second matrix multiplication operation need to be performed in sequence according to the data flow order.

[0040] However, if the operators in the target task flow are called sequentially according to the data flow order of the target task flow, only one computing unit on the same core will be running at a time, and the other computing units will be idle, resulting in a waste of hardware resources.

[0041] In this embodiment, considering that each operator in the target task flow is a linear algebra operation and there is a data dependency relationship between the operators in the target task flow, the target task flow is divided into data flow dimensions and each operator is further divided into operator slices, so that each operator slice can be scheduled in parallel, thereby improving the utilization rate of the computing unit.

[0042] Specifically, the target task stream is divided into multiple subtask streams. Each subtask stream includes an operator slice corresponding to at least two operators, and the operator slice of the operator is a sub-operation corresponding to the operator. The operator slices belonging to the same operator in different subtask streams are different, and the operator slices belonging to the same operator in different subtask streams together constitute a complete operator. For example, if the operator is a 64*64 matrix multiplication operation, the operator can be divided into 64 1*64 matrix multiplication operations (that is, the operator is divided into 64 operator slices), and the 64 1*64 matrix multiplication operations are respectively located in different subtask streams.

[0043] The execution order of each operator slice in the subtask flow is the same as the data flow order of the target task flow. Assume that the target task flow includes operators a, b, c, and d, and there is a data dependency relationship between operators a, b, c, and d, and the data flow order of operators a, b, c, and d is: a→b→c→d.

[0044] In one example, after the target task flow is divided, subtask flow 1, subtask flow 2 and subtask flow 3 are obtained. Subtask flow 1 includes operator slices a1, b1, c1 and d1, subtask flow 2 includes operator slices a2, b2, c2 and d2, and subtask flow 3 includes operator slices a3, b3, c3 and d3. a1, a2 and a3 are three operator slices of operator a, b1, b2 and b3 are three operator slices of operator b, c1, c2 and c3 are three operator slices of operator c, and d1, d2 and d3 are three operator slices of operator d. In this example, the execution order of the operator slices in subtask flow 1 is a1→b1→c1→d1, the execution order of the operator slices in subtask flow 2 is a2→b2→c2→d2, and the execution order of the operator slices in subtask flow 3 is a3→b3→c3→d3.

[0045] In another example, after the target task flow is divided, subtask flow 1, subtask flow 2, subtask flow 3, subtask flow 4 and subtask flow 5 are obtained, subtask flow 1 includes operator slices a1 and b1, subtask flow 2 includes operator slices a2 and b2, subtask flow 3 includes operator slices c1 and d1, subtask flow 4 includes operator slices c2 and d2, and subtask flow 5 includes operator slices c3 and d3. a1 and a2 are two operator slices of operator a, b1 and b2 are two operator slices of operator b, c1, c2 and c3 are three operator slices of operator c, and d1, d2 and d3 are three operator slices of operator d. In this example, the execution order of the operator slices in subtask flow 1 is a1→b1, the execution order of the operator slices in subtask flow 2 is a2→b2, the execution order of the operator slices in subtask flow 3 is c1→d1, the execution order of the operator slices in subtask flow 4 is c2→d2, and the execution order of the operator slices in subtask flow 5 is c3→d3. It should be noted that in this example, after the operator slices in subtask flow 1 and subtask flow 2 are scheduled for execution, the operator slices in subtask flow 3, subtask flow 4, and subtask flow 5 are scheduled to comply with the semantics of the original target task flow.

[0046] S12: Generate target time scheduling information for each subtask flow.

[0047] Among them, the target time scheduling information of the subtask flow includes the scheduling time of each operator slice in the subtask flow. There are multiple operator slices of different types with the same scheduling time, that is, there are multiple operator slices of different types that can be scheduled at the same time, so that different types of operator slices can be executed in parallel by different computing units at the same time. In addition, the scheduling times of operator slices of the same type are different, that is, operator slices of the same type are not scheduled at the same time, so that different computing units of the same type of operator slices have conflicts.

[0048] S13: Scheduling the operator slices in each subtask flow according to the target time scheduling information of each subtask flow.

[0049] Specifically, the target time scheduling information of all subtask flows includes several scheduling times in total, each scheduling time corresponds to at least one different type of operator slice, and according to the time sequence of the several scheduling times, at least one different computing unit required for each scheduling time is called to execute at least one different type of operator slice corresponding to each scheduling time.

[0050] Take the first example in step S11 as an example, in which the target task flow is divided into subtask flow 1, subtask flow 2 and subtask flow 3. Assume that the target time scheduling information of these three subtask flows includes six scheduling times (or execution stages), and the six scheduling times are arranged in chronological order as stage1, stage2, stage3, stage4, stage5 and stage6. In addition, the scheduling time of operator slice a1 is stage1, the scheduling time of operator slices b1 and a2 is stage2, the scheduling time of operator slices c1, b2 and a3 is stage3, the scheduling time of operator slices d1, c2 and b3 is stage4, the scheduling time of operator slices d2 and c3 is stage5, and the scheduling time of operator slice d3 is stage6. The operator slices of each scheduling time are scheduled in the time sequence of stage1 to stage6, so that operator slices b1 and a2 are executed in parallel at stage2, operator slices c1, b2 and a3 are executed in parallel at stage3, operator slices d1, c2 and b3 are executed in parallel at stage4, and operator slices d2 and c3 are executed in parallel at stage5.

[0051] In this embodiment, the target task flow is divided into multiple subtask flows, wherein the target task flow includes multiple operators, each subtask flow includes an operator slice corresponding to at least two operators, and the execution order of each operator slice in the subtask flow is the same as the data flow order of the target task flow; the target time scheduling information of each subtask flow is generated, and the operator slices in each subtask flow are scheduled according to the target time scheduling information of each subtask flow, wherein the target time scheduling information of the subtask flow includes the scheduling time of each operator slice in the subtask flow, and there are multiple different types of operator slices with the same scheduling time, and the scheduling time of the same type of operator slices is different. For a neural network accelerator using a spatial domain computing architecture, since different types of operator slices require different computing units in the kernel to execute, scheduling in the above manner can execute multiple different types of operator slices in parallel at the same scheduling time, realize parallel computing of computing units in the kernel, and thus improve the utilization rate of computing units in the kernel.

[0052] See also Figure 2 , Figure 2 It is a flow chart of another embodiment of the operator scheduling method provided by the present application. It should be noted that if there are substantially the same results, the method of the present application is not based on Figure 2 The process sequence shown is limited. Figure 2 As shown, the method comprises the following steps:

[0053] S21: Divide the target task flow into multiple subtask flows.

[0054] The target task flow includes multiple operators, and each subtask flow includes an operator slice corresponding to at least two operators. The relevant contents of the target task flow, subtask flow, operator and operator slice can refer to the aforementioned step S11, which will not be repeated here.

[0055] Figure 3 FIG. 1 is a flow chart of an embodiment of a method for dividing subtask flows provided by the present application. Figure 3 As shown, the method comprises the following steps:

[0056] S301: Perform loop block processing on each operator in the target task flow to obtain a first loop body corresponding to each operator.

[0057] MLIR (Multi-Level Intermediate Representation) is a compiler infrastructure for building reusable and extensible intermediate representations. MLIR's Linalg IR describes the task flow graph structure as consisting of multiple linalg operations (linear algebra operations), and one linalg operation corresponds to an operator of the target task flow. In step S301, by performing loop tiling processing on each operator, the first loop body corresponding to each operator can be obtained. Each first loop body describes the memory access track inside the corresponding operator and a more detailed operation representation. For example, Linalg's matmul operation (matrix multiplication operation) can be represented by the two operations arith::mul and arith::add.

[0058] The first loop body corresponding to the operator includes multiple first iteration loops, and each first iteration loop includes an operator slice of the operator.

[0059] S302: Merge the first loop bodies of each operator to obtain at least one second loop body.

[0060] In this embodiment, after the first loop body of each operator is obtained, it is further considered to merge the first loop bodies of each operator to construct multiple subtask flows.

[0061] Specifically, in step S302, first loop bodies meeting the preset merging condition are merged to obtain at least one second loop body.

[0062] The preset merging conditions include: the loop parameters of each first loop body are the same, and there is a data dependency relationship between the operators corresponding to each first loop body. The loop parameters include the index upper boundary, index lower boundary and index iteration step of the outer loop.

[0063] Each second loop body obtained by merging includes multiple second iteration loops, and each second iteration loop includes an operator slice corresponding to at least two operators. There is a data dependency relationship between the operators corresponding to the second loop bodies, and there is a data dependency relationship between the operator slices included in each second iteration loop in the second loop body.

[0064] It should be noted that, considering that it may be impossible to merge all the multiple operators of the target task flow into the same second loop body, the number of the second loop bodies in this embodiment may be more than one. For example, the target task flow includes operators a, b, c and d, and there is a data dependency relationship between operators a, b, c and d, and the data flow order of operators a, b, c and d is: a→b→c→d, but since only the first loop body corresponding to operator a and the first loop body corresponding to operator b meet the aforementioned preset merging conditions, the first loop body corresponding to operator c and the first loop body corresponding to operator d meet the aforementioned preset conditions, and the first loop bodies corresponding to operators a, b, c and d do not meet the aforementioned preset conditions, therefore, the first loop bodies corresponding to operators a, b, c and d respectively cannot be merged into one second loop body, but the first loop body corresponding to operator a and the first loop body corresponding to operator b are merged into one second loop body, and the first loop body corresponding to operator c and the first loop body corresponding to operator d are merged into another second loop body.

[0065] S303: Treat each second iteration loop of each second loop body as a subtask flow.

[0066] Specifically, the number of divided subtask flows is the same as the number of second iterative loops, and one second iterative loop in the second loop body corresponds to one subtask flow.

[0067] In the Linalg SSA (Static Single Assignment) IR in MLIR, the data dependencies in the loop body are represented by the def-use chain, and the def (definition) and use (use) of each variable are explicitly recorded. Therefore, the data dependencies between the operator slices in the second loop body and the boundaries of each subtask flow can be obtained through the SSA IR.

[0068] Through the aforementioned steps S301 to S303, the target task flow can be divided into a plurality of sub-task flows. In addition, by merging operators that can be merged into the same second loop body, the overall memory access times can be reduced and the cache hit rate can be improved.

[0069] It should be noted that there is a data dependency relationship between the operator slices included in each second iteration loop in the same second loop body. If the back-end instruction for IR mapping of the second loop body is executed, that is, the second iteration loops of the second loop body are executed in sequence according to the loop order of each second iteration loop in the second loop body, then all operator slices of the same second loop body are still executed in series, resulting in that the various computing units in the core still cannot run in parallel, that is, there is still a problem of low computing unit utilization. Therefore, in this embodiment, steps S22 to S24 are further performed to generate target time scheduling information for each subtask flow, realize parallel computing of computing units in the core, and thus improve the utilization of computing units in the core.

[0070] S22: Generate a number of first candidate scheduling mapping relationships.

[0071] Each first candidate scheduling mapping relationship is a mapping relationship between each operator slice and the candidate scheduling time. Specifically, the first candidate scheduling mapping relationship includes a number of candidate scheduling times and a set of operator slices corresponding to each candidate scheduling time. The operator slices belonging to the same subtask flow are respectively located in different operator slice sets, that is, the operator slices belonging to the same subtask flow are scheduled for execution at different candidate scheduling times. The types of operator slices in the same operator slice set are different, that is, different types of operator slices are executed corresponding to the same candidate scheduling time. The maximum parallel number of operator slices of each first candidate scheduling mapping relationship is greater than the first preset number and is less than or equal to the number of types of computing units required by the target task flow.

[0072] Each candidate scheduling time of the first candidate scheduling mapping relationship corresponds to the execution of at least one different type of operator slice, and the maximum parallel number of operator slices of the first candidate scheduling mapping relationship refers to the maximum number of operator slices executed in parallel at each candidate scheduling time. For example, the first candidate scheduling mapping relationship includes 4 candidate scheduling times, namely stage1, stage2, stage3 and stage4, stage1 corresponds to the execution of operator slice a1, stage2 corresponds to the execution of operator slice b1 and operator slice a2, stage3 corresponds to the execution of operator slice c1 and operator slice b2, and stage4 corresponds to the execution of operator slice c2, then the maximum parallel number of operator slices of the first candidate scheduling mapping relationship is 2.

[0073] The first preset number is greater than or equal to 1 and less than the number of types of computing units required by the target task flow. Exemplarily, the first preset number is 1, that is, there is at least one candidate scheduling time in the first candidate scheduling mapping relationship corresponding to the parallel execution of multiple (two or more than two) different types of operator slices.

[0074] The number of types of computing units required by the target task flow is the same as the number of types of operators corresponding to the target task flow. For example, if the target task flow includes operator a, operator b, and operator c, and the types of operators a, b, and c are all different, then the number of types of computing units required by the target task flow is 3. For another example, if the target task flow includes operator a, operator b, and operator c, but the type of operator a is the same as the type of operator c and the type of operator b is different from operators a and c, then the number of types of computing units required by the target task flow is 2.

[0075] It is understandable that if the maximum parallel number of operator slices of the first candidate scheduling mapping relationship is the same as the number of types of computing units required by the target task flow, then when the first candidate scheduling mapping relationship is used for operator scheduling, all computing units in the core can run simultaneously in certain execution stages. The first candidate scheduling mapping relationship is equivalent to an SPB (Software Pipeline Block) scheduling table, and the maximum parallel number of operator slices corresponding to the first candidate scheduling mapping relationship is the deepest pipeline depth of the SPB scheduling table.

[0076] In this embodiment, the step of generating a plurality of first candidate scheduling mapping relationships includes: generating a third candidate scheduling mapping relationship; and adjusting the third candidate scheduling mapping relationship to obtain a plurality of first candidate scheduling mapping relationships. The generated plurality of first candidate scheduling mapping relationships may include the first generated third candidate scheduling mapping relationship.

[0077] The maximum parallel number of operator slices of the third candidate scheduling mapping relationship is greater than the first preset number and less than or equal to the number of types of computing units required by the target task flow. Exemplarily, the number of types of computing units required by the target task flow is 3, and the maximum parallel number of operator slices of the third candidate scheduling mapping relationship is greater than 1 and less than 3. For example, the maximum parallel number of operator slices of the third candidate scheduling mapping relationship is 2, or the maximum parallel number of operator slices of the third candidate scheduling mapping relationship is the same as the number of types of computing units required by the target task flow, that is, 3.

[0078] In one embodiment, multiple subtask flows of the target task flow are added to the subtask flow set, and generating the third candidate scheduling mapping relationship may include the following sub-steps:

[0079] Sub-step 1: generating an initial candidate scheduling mapping relationship, wherein the initial candidate scheduling mapping relationship includes a plurality of candidate scheduling times and a set of operator slices corresponding to each candidate scheduling time.

[0080] It should be noted that the operator slice sets corresponding to each candidate scheduling time in the initial candidate scheduling mapping relationship are all empty sets.

[0081] Sub-step 2: select a sub-task flow from the sub-task flow set, and add the operator slices included in the selected sub-task flow to the target queue according to the data flow order.

[0082] In one example, a subtask flow is arbitrarily selected from the subtask flow set.

[0083] In another example, a subtask flow is selected from the subtask flow set according to the loop order of the subtask flow in the corresponding second loop body.

[0084] Furthermore, after a subtask flow is selected, the operator slices included in the selected subtask flow are added to the target queue in the order of data flow. For example, the selected subtask flow includes operator slices a1, b1, c1 and d1, and the data flow order of operator slices a1, b1, c1 and d1 is: a1→b1→c1→d1, then a1 is located at the head of the target queue, and d1 is located at the tail of the target queue. It should be noted that after a subtask flow is selected from the subtask flow set, the selected subtask flow is not included in the subtask flow set.

[0085] Sub-step three: determine the initial target scheduling time corresponding to the selected sub-task flow.

[0086] The initial target scheduling time corresponding to the subtask flow can be the first candidate scheduling time of several candidate scheduling times, or can also be other candidate scheduling times of several candidate scheduling times. The initial target scheduling times corresponding to different subtask flows can be the same or different, and this embodiment does not specifically limit this.

[0087] Sub-step 4: Determine whether there is an operator slice of the same type as the target operator slice in the operator slice set of the target scheduling time. If there is no operator slice of the same type as the target operator slice in the operator slice set of the target scheduling time, execute sub-step 5; if there is an operator slice of the same type as the target operator slice in the operator slice set of the target scheduling time, execute sub-step 6.

[0088] The target operator slice is the operator slice at the head of the target queue. If the operator slice set at the target scheduling time contains an operator slice of the same type as the target operator slice, it indicates that there is a computing unit conflict; if the operator slice set at the target scheduling time does not contain an operator slice of the same type as the target operator slice, it indicates that there is no computing unit conflict.

[0089] Sub-step five: remove the target operator slice from the target queue and add it to the operator slice set of the target scheduling time.

[0090] It can be understood that, after the target operator slice is removed from the target queue, the next operator slice of the target operator slice in the target queue becomes the new target operator slice.

[0091] After sub-step five is executed, sub-step seven is further executed.

[0092] Sub-step six, taking the next candidate scheduling time of the target scheduling time as the new target scheduling time, and re-executing sub-step four.

[0093] Sub-step 7: Determine whether the total number of operator slices in the target queue is less than the second preset number. If the total number of operator slices in the target queue is greater than or equal to the second preset number, execute sub-step 8; if the total number of operator slices in the target queue is less than the second preset number, execute sub-step 9.

[0094] The second preset number is 1. When the total number of operator slices in the target queue is less than 1, it indicates that all operator slices in the target queue have been added to the corresponding operator slice set.

[0095] Sub-step eight, taking the next candidate scheduling time of the target scheduling time as the new target scheduling time, and re-executing sub-step four.

[0096] Sub-step nine: determine whether the number of sub-task flows in the sub-task flow set is less than a second preset number. If the number of sub-task flows in the sub-task flow set is greater than or equal to the second preset number, re-execute sub-step two until the number of sub-task flows in the latest sub-task flow set is less than the second preset number.

[0097] When the number of subtask flows in the subtask flow set is less than the second preset number, it means that all operator slices of the subtask flow set have been added to the corresponding operator slice set. At this time, each candidate scheduling time and the operator slice set of each candidate scheduling time constitute a third candidate scheduling mapping relationship.

[0098] By first generating a third candidate scheduling mapping relationship and then adjusting a plurality of first candidate scheduling mapping relationships based on the third candidate scheduling mapping relationship, it is not necessary to generate each first candidate scheduling mapping relationship separately, thereby improving the generation efficiency of the plurality of first candidate scheduling mapping relationships.

[0099] In one implementation, the step of adjusting the third candidate scheduling mapping relationship includes: adjusting the candidate scheduling time of at least one operator slice in the third candidate scheduling mapping relationship.

[0100] In another embodiment, the step of adjusting the third candidate scheduling mapping relationship includes: adjusting the maximum parallel number of operator slices of the third candidate scheduling mapping relationship.

[0101] In one example, the maximum parallel number of operator slices of the third candidate scheduling mapping relationship is greater than the first preset number and less than the number of types of computing units required by the target task flow. At this time, the maximum parallel number of operator slices of the third candidate scheduling mapping relationship can be increased or decreased.

[0102] In another example, the maximum parallel number of operator slices of the third candidate scheduling mapping relationship is equal to the number of types of computing units required by the target task flow. In this case, the maximum parallel number of operator slices of the third candidate scheduling mapping relationship can be reduced.

[0103] In another embodiment, the step of adjusting the third candidate scheduling mapping relationship includes: adjusting the candidate scheduling time of at least one operator slice in the third candidate scheduling mapping relationship, and adjusting the maximum parallel number of operator slices in the third candidate scheduling mapping relationship.

[0104] The following takes the target task flow as the scaled dot product attention sub-network as an example to illustrate the process of generating several first candidate scheduling mapping relationships.

[0105] Figure 4 : is a schematic diagram of the principle of generating the third candidate scheduling mapping relationship for the scaled dot product attention sub-network provided by this application. Figure 4 As shown in the figure, MLIR's Linalg IR describes module A (i.e., the scaled dot product attention subnetwork) as module B (i.e., the target task flow corresponding to the scaled dot product attention subnetwork). Module B includes four defined linalgoperations, corresponding to the first matrix multiplication operation, multiplication operation, Softmax operation, and second matrix multiplication operation of the scaled dot product attention subnetwork.

[0106] Furthermore, after the first matrix multiplication operation, the multiplication operation, the Softmax operation, and the second matrix multiplication operation in the B module are subjected to loop tiling and merging, the C module (i.e., the second loop body corresponding to the scaled dot product attention sub-network) is obtained. The C module includes 64 second iteration loops, each of which includes an operator slice of the first matrix multiplication operation, an operator slice of the multiplication operation, an operator slice of the Softmax operation, and an operator slice of the second matrix multiplication operation, i.e., each second iteration loop is part of the scaled dot product attention sub-network.

[0107] Furthermore, a second iteration loop in the C module is used as a subtask flow of the scaled dot product attention subnetwork, thereby obtaining multiple subtask flows corresponding to the scaled dot product attention subnetwork.

[0108] Further, after performing the processing of sub-steps 1 to 9 in step S22 on the multiple sub-task flows of the scaled dot product attention sub-network, a third candidate scheduling mapping relationship corresponding to the scaled dot product attention sub-network can be obtained. The third candidate scheduling mapping relationship can be described as Figure 4 The D module in .

[0109] Furthermore, after adjusting the third candidate scheduling mapping relationship of the scaled dot product attention sub-network, several first candidate scheduling mapping relationships corresponding to the scaled dot product attention sub-network can be obtained. Since the scaled dot product attention sub-network includes a first matrix multiplication operation, a multiplication operation, a Softmax operation, and a second matrix multiplication operation, and the operation types of the first matrix multiplication operation and the second matrix multiplication operation are the same, that is, the number of types of operation units required for the scaled dot product attention sub-network is 3, the several first candidate scheduling mapping relationships corresponding to the scaled dot product attention sub-network can be divided into a first type (Triple Buffer) and a second type (Double Buffer). Among them, the maximum parallel number of operator slices in the first candidate scheduling mapping relationship belonging to the first type is 3, and the maximum parallel number of operator slices in the first candidate scheduling mapping relationship belonging to the second type is 2.

[0110] Figure 5 It is a schematic diagram of the first candidate scheduling mapping relationship of the first type provided in the present application. Figure 6 It is a schematic diagram of the first candidate scheduling mapping relationship belonging to the second type provided in the present application. Figure 5 and Figure 6 In , the square represents the first matrix multiplication operation or the second matrix multiplication operation, the circle represents the multiplication operation, and the triangle represents the Softmax operation. And, Figure 5 and Figure 6 The middle vertical axis represents the candidate scheduling time of the first candidate scheduling mapping relationship, and the horizontal axis represents at least one operator slice corresponding to each candidate scheduling time (ie, the aforementioned operator slice set corresponding to each candidate scheduling time).

[0111] It should be noted that Figure 5 In the example, only the first candidate scheduling mapping relationship of the first type is SPBI, SPBII, SPBIII and SPBIV. Figure 6 Only the first candidate scheduling mapping relationships of the second type are SPBV, SPBVI, SPBVII and SPBVIII for illustrative illustration, and they are not listed one by one here.

[0112] It should be noted that when Figure 5 The candidate scheduling time includes 2 or 3 types of operator slices, Figure 6When the candidate scheduling time includes two types of operator slices, the arrangement order of the operator slices corresponding to the candidate scheduling time does not represent the execution order of the operator slices under the candidate scheduling time. That is, when the candidate scheduling time corresponds to multiple different types of operator slices, the operator slices corresponding to the candidate scheduling time are executed in parallel, and there is no order of execution.

[0113] Since each operator slice of each candidate scheduling time is executed in parallel, in order to ensure that the data can be accessed and read and written independently, a shared memory on the kernel can be divided. Specifically, the first candidate scheduling mapping relationship belonging to the first type corresponds to dividing the shared memory into 3 parts to meet the peak memory budget of the scaled dot product attention subnetwork; the first candidate scheduling mapping relationship belonging to the second type corresponds to dividing the shared memory into 2 parts to meet the peak memory budget of the scaled dot product attention subnetwork. The maximum parallel number of operator slices corresponding to the former (for the convenience of description, the former in the following text refers to the first candidate scheduling mapping relationship belonging to the first type) is greater than the maximum parallel number of operator slices corresponding to the latter (for the convenience of expression, the latter in the following text refers to the first candidate scheduling mapping relationship belonging to the second type). It can be understood that since the total amount of data corresponding to the scaled dot product attention subnetwork is unchanged, the number of subtask flows corresponding to the former is greater than the number of subtask flows corresponding to the latter. For example, when the number of subtask flows corresponding to the former is 6, the number of subtask flows corresponding to the latter should be 4. Moreover, the data processing amount of each operator slice of the former is less than that of each operator slice of the latter, that is, the total number of operator slices of the former is more than the total number of operator slices of the latter. Therefore, the total number of calls of the operator slices of the former is higher than the total number of calls of the operator slices of the latter.

[0114] S23: Select a second candidate scheduling mapping relationship that meets the requirements from a plurality of first candidate scheduling mapping relationships.

[0115] In one embodiment, selecting a second candidate scheduling mapping relationship that meets the requirements from a plurality of first candidate scheduling mapping relationships includes the following steps:

[0116] Step 1: determine the estimated performance parameters corresponding to each first candidate scheduling mapping relationship.

[0117] Among them, the estimated performance parameters of the first candidate scheduling mapping relationship include at least one of the maximum parallel number of operator slices, the total scheduling number of operator slices, and the total estimated execution time. For example, the estimated performance parameters of the first candidate scheduling mapping relationship may include only the maximum parallel number of operator slices, or only the total scheduling number of operator slices, or only the total estimated execution time. For another example, the estimated performance parameters of the first candidate scheduling mapping relationship may include both the maximum parallel number of operator slices and the total scheduling number of operator slices, or both the maximum parallel number of operator slices and the total estimated execution time, or both the total scheduling number of operator slices and the total estimated execution time. For another example, the estimated performance parameters of the first candidate scheduling mapping relationship may include both the maximum parallel number of operator slices, the total scheduling number of operator slices, and the total estimated execution time.

[0118] It can be understood that the more the maximum parallel number of operator slices, the more the corresponding maximum parallel number of computing units, and the more the total scheduling times of the corresponding operator slices (that is, the more the calling overhead of the operator slices); the fewer the maximum parallel number of operator slices, the fewer the corresponding maximum parallel number of computing units, and the fewer the total scheduling times of the corresponding operator slices (that is, the lower the calling overhead of the operator slices). The total estimated execution time corresponding to the first candidate scheduling mapping relationship is the sum of the estimated execution times of all candidate scheduling times in the first candidate scheduling mapping relationship, and the estimated execution time of each candidate scheduling time is the maximum estimated execution time of the estimated execution times of each operator slice corresponding to the candidate scheduling time. The total estimated execution times of operator slices corresponding to different first candidate scheduling mapping relationships are different.

[0119] Step 2: for each first candidate scheduling mapping relationship, each estimated performance parameter of the first candidate scheduling mapping relationship is comprehensively considered to determine a first evaluation value of the first candidate scheduling mapping relationship.

[0120] In one example, the estimated performance parameters of each first candidate scheduling mapping relationship only include one of the maximum parallel number of operator slices, the total number of scheduling times of operator slices, and the total estimated execution time. For each first candidate scheduling mapping relationship, the estimated performance parameters of the first candidate scheduling mapping relationship can be directly used as the first evaluation value of the first candidate scheduling mapping relationship.

[0121] Alternatively, in this example, for each first candidate scheduling mapping relationship, the score value corresponding to the estimated performance parameter of the first candidate scheduling mapping relationship can be determined first, and then the score value can be used as the first evaluation value of the first candidate scheduling mapping relationship. For example, in a case where the estimated performance parameter of the first candidate scheduling mapping relationship only includes the maximum parallel number of operator slices, the score value corresponding to the maximum parallel number of operator slices of the first candidate scheduling mapping relationship can be determined based on the first corresponding relationship between the maximum parallel number of operator slices and the score value. In the first corresponding relationship, the maximum parallel number of operator slices is proportional to the score value, that is, the more the maximum parallel number of operator slices, the higher the corresponding score value, and the lower the maximum parallel number of operator slices, the lower the corresponding score value. In the case where the estimated performance parameters of the first candidate scheduling mapping relationship only include the total scheduling times of the operator slices, the score value corresponding to the total scheduling times of the operator slices of the first candidate scheduling mapping relationship can be determined according to the second corresponding relationship between the total scheduling times of the operator slices and the score value. In the second corresponding relationship, the total scheduling times of the operator slices and the score value are inversely proportional, that is, the more the total scheduling times of the operator slices, the lower the corresponding score value, and the fewer the total scheduling times of the operator slices, the higher the corresponding score value. In the case where the estimated performance parameters of the first candidate scheduling mapping relationship only include the total estimated execution time, the score value corresponding to the total estimated execution time of the first candidate scheduling mapping relationship can be determined according to the third corresponding relationship between the total estimated execution time and the score value. In the third corresponding relationship, the total estimated execution time and the score value are inversely proportional, that is, the longer the total estimated execution time, the lower the corresponding score value, and the shorter the total estimated execution time, the higher the corresponding score value.

[0122] In another example, the estimated performance parameters of each first candidate scheduling mapping relationship include two or all of the maximum number of parallel operator slices, the total number of operator slice scheduling times, and the total estimated execution time. For each first candidate scheduling mapping relationship, the score values ​​corresponding to each of the estimated performance parameters of the first candidate scheduling mapping relationship can be determined first, and then the score values ​​of each of the first candidate scheduling mapping relationships can be weighted to obtain a first evaluation value of the first candidate scheduling mapping relationship. Among them, the score value corresponding to the maximum number of parallel operator slices, the score value corresponding to the total number of operator slice scheduling times, and the score value corresponding to the total estimated execution time of the first candidate scheduling mapping relationship can be determined based on the aforementioned first corresponding relationship, second corresponding relationship, and third corresponding relationship, respectively.

[0123] For example, the estimated performance parameters of the first candidate scheduling mapping relationship include the maximum parallel number of operator slices and the total scheduling times of operator slices. First, determine the score corresponding to the maximum parallel number of operator slices and the score corresponding to the total scheduling times of operator slices of the first candidate scheduling mapping relationship, and then weight the score corresponding to the maximum parallel number of operator slices and the score corresponding to the total scheduling times of operator slices of the first candidate scheduling mapping relationship to obtain the first evaluation value of the first candidate scheduling mapping relationship. Among them, the weighting coefficient corresponding to the maximum parallel number of operator slices and the weighting coefficient corresponding to the total scheduling times of operator slices can be set according to actual needs, such as according to the chip performance of the neural network accelerator. For example, the computing performance of the neural network accelerator is higher and the memory access performance is lower. The weighting coefficient corresponding to the maximum parallel number of operator slices can be set to a higher value, and the weighting coefficient corresponding to the total scheduling times of operator slices can be set to a lower value.

[0124] For another example, the performance parameters of the first candidate scheduling mapping relationship include the maximum parallel number of operator slices, the total number of operator slice scheduling times, and the total estimated execution time. First, determine the score corresponding to the maximum parallel number of operator slices, the score corresponding to the total number of operator slice scheduling times, and the score corresponding to the total estimated execution time of the first candidate scheduling mapping relationship, and then weight the score corresponding to the maximum parallel number of operator slices, the score corresponding to the total number of operator slice scheduling times, and the score corresponding to the total estimated execution time of the first candidate scheduling mapping relationship, so as to obtain the first evaluation value of the first candidate scheduling mapping relationship. Among them, the weighting coefficient corresponding to the maximum parallel number of operator slices, the weighting coefficient corresponding to the total number of operator slice scheduling times, and the weighting coefficient of the total estimated execution time can be set according to actual needs.

[0125] Step three: selecting a second candidate scheduling mapping relationship from a plurality of first candidate scheduling mapping relationships based on the first evaluation value of each first candidate scheduling mapping relationship.

[0126] In one example, step three includes: selecting a first candidate scheduling mapping relationship whose first evaluation value meets the requirements from a plurality of first candidate scheduling mapping relationships as a second candidate scheduling mapping relationship.

[0127] For example, the estimated performance parameters of each first candidate scheduling mapping relationship only include one of the maximum number of parallel operators, the total number of operator slice scheduling times, and the total estimated execution time, and the estimated performance parameters of the first candidate scheduling mapping relationship are directly used as the first evaluation value of the first candidate scheduling mapping relationship. In the case where the estimated performance parameters only include the maximum number of parallel operators, the first candidate scheduling mapping relationship with the highest first evaluation value (i.e., the largest maximum number of parallel operators) can be used as the second candidate scheduling mapping relationship. In the case where the estimated performance parameters only include the total number of operator slice scheduling times or the total estimated execution time, the first candidate scheduling mapping relationship with the lowest first evaluation value (i.e., the smallest total number of operator slice scheduling times or the shortest total estimated execution time) can be used as the second candidate scheduling mapping relationship.

[0128] For another example, the first evaluation value of each first candidate scheduling mapping relationship is obtained based on the score value of each estimated performance parameter, and the first candidate scheduling mapping relationship with the highest first evaluation value can be used as the second candidate scheduling mapping relationship.

[0129] In another example, step three may include the following sub-steps:

[0130] Sub-step 1: Screening out a plurality of fourth candidate scheduling mapping relationships whose first evaluation values ​​meet preset conditions from a plurality of first candidate scheduling mapping relationships.

[0131] For example, each first candidate scheduling mapping relationship may be sorted based on the size of the first evaluation value of each first candidate scheduling mapping relationship, such as sorting them in descending order according to the first evaluation value; and then a plurality of fourth candidate scheduling mapping relationships that meet the preset ranking (i.e., a plurality of fourth candidate scheduling mapping relationships that meet the preset conditions) are screened out from the plurality of first candidate scheduling mapping relationships. For example, the number of the plurality of first candidate scheduling mapping relationships is 10, and the top 3 fourth candidate scheduling mapping relationships may be selected from the 10 first candidate scheduling mapping relationships.

[0132] Sub-step two: performing at least one running test on each fourth candidate scheduling mapping relationship to obtain at least one total test execution time corresponding to each fourth candidate scheduling mapping relationship.

[0133] Specifically, at least one run test is performed on each fourth candidate scheduling mapping relationship, and the total test execution time of each fourth candidate scheduling mapping relationship under each run test is obtained, thereby obtaining at least one total test execution time corresponding to each fourth candidate scheduling mapping relationship. Exemplarily, each fourth candidate scheduling mapping relationship may be run tested once, twice, five times, etc., and this embodiment does not specifically limit this.

[0134] Sub-step three: for each fourth candidate scheduling mapping relationship, based on at least one total test execution time of the fourth candidate scheduling mapping relationship, determine a second evaluation value of the fourth candidate scheduling mapping relationship.

[0135] For example, each fourth candidate scheduling mapping relationship includes a total test execution time. For each fourth candidate scheduling mapping relationship, a total test execution time of the fourth candidate scheduling mapping relationship may be used as the second evaluation value of the fourth candidate scheduling mapping relationship.

[0136] For another example, each fourth candidate scheduling mapping relationship includes multiple total test execution times. For each fourth candidate scheduling mapping relationship, the average value of the multiple total test execution times of the fourth candidate scheduling mapping relationship can be used as the second evaluation value of the fourth candidate scheduling mapping relationship.

[0137] For another example, each fourth candidate scheduling mapping relationship includes multiple total test execution times. For each fourth candidate scheduling mapping relationship, the average value of the multiple total test execution times of the fourth candidate scheduling mapping relationship can be determined first, and then the standard deviation corresponding to the multiple total test execution times of the fourth candidate scheduling mapping relationship can be determined based on the average value, and the standard deviation corresponding to the multiple total test execution times of the fourth candidate scheduling mapping relationship can be used as the second evaluation value of the fourth candidate scheduling mapping relationship.

[0138] Sub-step four: selecting a fourth candidate scheduling mapping relationship whose second evaluation value meets the requirement from multiple fourth candidate scheduling mapping relationships as a second candidate scheduling mapping relationship.

[0139] Specifically, a fourth candidate scheduling mapping relationship with the smallest second evaluation value is selected from a plurality of fourth candidate scheduling mapping relationships as the second candidate scheduling mapping relationship.

[0140] In another embodiment, selecting a second candidate scheduling mapping relationship that meets the requirements from a plurality of first candidate scheduling mapping relationships may include the following steps:

[0141] Step 1: Perform at least one running test on each first candidate scheduling mapping relationship to obtain at least one total test execution time corresponding to each first candidate scheduling mapping relationship.

[0142] Specifically, at least one run test is performed on each first candidate scheduling mapping relationship, and the total test execution time of each first candidate scheduling mapping relationship under each run test is obtained, thereby obtaining at least one total test execution time corresponding to each first candidate scheduling mapping relationship. Exemplarily, each first candidate scheduling mapping relationship may be run tested once, twice, five times, etc., and this embodiment does not specifically limit this.

[0143] Step 2: for each first candidate scheduling mapping relationship, based on at least one total test execution time of the first candidate scheduling mapping relationship, determine a third evaluation value of the first candidate scheduling mapping relationship.

[0144] For the content related to determining the third evaluation value of the first candidate scheduling mapping relationship, reference may be made to the aforementioned determination of the second evaluation value of the fourth candidate scheduling mapping relationship, which will not be elaborated herein.

[0145] Step three: Select a first candidate scheduling mapping relationship whose third evaluation value meets the requirements from among a plurality of first candidate scheduling mapping relationships as a second candidate scheduling mapping relationship.

[0146] Specifically, a first candidate scheduling mapping relationship with the smallest third evaluation value is selected from a plurality of first candidate scheduling mapping relationships as the second candidate scheduling mapping relationship.

[0147] In this embodiment, the several first candidate scheduling mapping relationships generated in step S22 can be attached as attribute information to the IR BLOCK to which the target task flow belongs, and then after step S23, the second candidate scheduling mapping relationship with the best performance is determined from the several first candidate scheduling mapping relationships for subsequent operator scheduling.

[0148] Optionally, in this embodiment, step S22 and step S23 can also be replaced by: directly generating a third candidate scheduling mapping relationship as the second candidate scheduling mapping relationship. In this embodiment, the maximum parallel number of operator slices of the third candidate scheduling mapping relationship can be equal to the number of types of computing units required by the target task flow to achieve the maximum parallelism of the computing units. The relevant content of generating the third candidate scheduling mapping relationship can refer to sub-steps 1 to 9 in step S22, and the detailed description is omitted here.

[0149] S24: For each subtask flow, the candidate scheduling time corresponding to each operator slice of the subtask flow in the second candidate scheduling mapping relationship is used as the target time scheduling information of the subtask flow.

[0150] S25: Scheduling the operator slices in each subtask flow according to the target time scheduling information of each subtask flow.

[0151] The relevant contents of step S25 can refer to the aforementioned step S13 and will not be repeated here.

[0152] In this embodiment, in view of the characteristics of the spatial domain computing architecture, the data dependencies between operators in the target task flow are statically analyzed during the compilation period, and the target task flow is divided in the data flow dimension to obtain multiple subtask flows corresponding to the target task flow, and then the scheduling mapping relationship with the best performance is determined to perform pipeline scheduling on each subtask flow, so as to realize parallel computing on the computing units within the core, and accelerate the execution of the target task flow. The divided subtask flows can break the boundaries of the computing units as much as possible, allowing for more fine-grained scheduling of each operator of the target task flow. In addition, this embodiment is a unified pure software solution that can rely less on the underlying hardware and can be adopted by different neural network accelerators.

[0153] See also Figure 7 , Figure 7 It is a framework diagram of an embodiment of the operator scheduling device provided by the present application. In this embodiment, the operator scheduling device 70 includes: a division module 71, a generation module 72 and a scheduling module 73. Among them, the division module 71 is used to divide the target task flow into multiple sub-task flows; wherein the target task flow includes multiple operators, each of the sub-task flows includes an operator slice corresponding to at least two of the operators, and the execution order of each of the operator slices in the sub-task flow is the same as the data flow order of the target task flow. The generation module 72 is used to generate the target time scheduling information of each of the sub-task flows; wherein the target time scheduling information includes the scheduling time of each of the operator slices in the sub-task flow, and there are multiple different types of operator slices with the same scheduling time, and the scheduling time of the operator slices of the same type is different. The scheduling module 73 is used to schedule the operator slices in each of the sub-task flows according to the target time scheduling information.

[0154] Optionally, the division module 71 is used to perform loop block processing on each operator to obtain a first loop body corresponding to each operator; wherein the first loop body includes multiple first iterative loops, and each first iterative loop includes an operator slice of the operator; each first loop body is merged to obtain at least one second loop body; wherein the second loop body includes multiple second iterative loops, and each second iterative loop includes an operator slice corresponding to at least two operators; each second iterative loop is treated as a subtask flow.

[0155] Optionally, the division module 71 is used to merge the first loop bodies that meet the preset merging conditions to obtain at least one second loop body; wherein the preset merging conditions include: the loop parameters of each first loop body are the same, and there is a data dependency relationship between the operators corresponding to each first loop body; the loop parameters include the index upper boundary, index lower boundary and index iteration step of the outer loop.

[0156] Optionally, the generation module 72 is used to generate several first candidate scheduling mapping relationships; wherein the first candidate scheduling mapping relationships include several candidate scheduling times and operator slice sets corresponding to each candidate scheduling time, the operator slices belonging to the same subtask flow are respectively located in different operator slice sets, the types of the operator slices in the same operator slice set are different, and the maximum parallel number of operator slices of each first candidate scheduling mapping relationship is greater than the first preset number and is less than or equal to the number of types of computing units required by the target task flow; a second candidate scheduling mapping relationship that meets the requirements is selected from the several first candidate scheduling mapping relationships; for each subtask flow, the candidate scheduling time corresponding to each operator slice of the subtask flow in the second candidate scheduling mapping relationship is used as the target time scheduling information of the subtask flow.

[0157] Optionally, the generating module 72 is configured to generate a third candidate scheduling mapping relationship; and adjust the third candidate scheduling mapping relationship to obtain a plurality of first candidate scheduling mapping relationships.

[0158] Optionally, the maximum parallel number of operator slices of the third candidate scheduling mapping relationship is equal to the number of types of computing units required by the target task flow; and / or, the generation module 72 is used to perform at least one of the following adjustment operations on the third candidate scheduling mapping relationship: adjust the candidate scheduling time corresponding to at least one operator slice in the third candidate scheduling mapping relationship; adjust the maximum parallel number of operator slices of the third candidate scheduling mapping relationship.

[0159] Optionally, the generation module 72 is used to determine the estimated performance parameters corresponding to each first candidate scheduling mapping relationship; wherein the estimated performance parameters of the first candidate scheduling mapping relationship include at least one of the maximum parallel number of operator slices of the first candidate scheduling mapping relationship, the total number of operator slice scheduling times, and the total estimated execution time; for each first candidate scheduling mapping relationship, the estimated performance parameters of the first candidate scheduling mapping relationship are comprehensively considered to determine a first evaluation value of the first candidate scheduling mapping relationship; based on the first evaluation value of each first candidate scheduling mapping relationship, a second candidate scheduling mapping relationship is selected from a number of first candidate scheduling mapping relationships.

[0160] Optionally, the generation module 72 selects a first candidate scheduling mapping relationship whose first evaluation value meets the requirements from a number of first candidate scheduling mapping relationships as the second candidate scheduling mapping relationship. Alternatively, the generation module 72 is used to screen out a plurality of fourth candidate scheduling mapping relationships whose first evaluation values ​​meet the preset conditions from a number of first candidate scheduling mapping relationships; perform at least one operation test on each fourth candidate scheduling mapping relationship to obtain at least one total test execution time corresponding to each fourth candidate scheduling mapping relationship; for each fourth candidate scheduling mapping relationship, based on at least one total test execution time of the fourth candidate scheduling mapping relationship, determine the second evaluation value of the fourth candidate scheduling mapping relationship; and select a fourth candidate scheduling mapping relationship whose second evaluation value meets the requirements from a plurality of fourth candidate scheduling mapping relationships as the second candidate scheduling mapping relationship.

[0161] Optionally, the generation module 72 is used to perform at least one run test on each first candidate scheduling mapping relationship, respectively, to obtain at least one total test execution time corresponding to each first candidate scheduling mapping relationship; for each first candidate scheduling mapping relationship, based on at least one total test execution time of the first candidate scheduling mapping relationship, determine a third evaluation value of the first candidate scheduling mapping relationship; and select a first candidate scheduling mapping relationship whose third evaluation value meets the requirements from a number of first candidate scheduling mapping relationships as a second candidate scheduling mapping relationship.

[0162] It should be noted that the device of this embodiment can execute the steps in the above method. For detailed description of the relevant content, please refer to the above method part, which will not be repeated here.

[0163] See also Figure 8 , Figure 8 8 is a schematic diagram of a framework of an electronic device according to an embodiment of the present application. In this embodiment, the electronic device 80 includes a memory 81 and a processor 82 .

[0164] The processor 82 may also be referred to as a CPU (Central Processing Unit). The processor 82 may be an integrated circuit chip having signal processing capabilities. The processor 82 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. A general-purpose processor may be a microprocessor or the processor 82 may also be any conventional processor 82, etc.

[0165] The memory 81 in the electronic device 80 is used to store program instructions required for the processor 82 to run.

[0166] The processor 82 is used to execute program instructions to implement the operator scheduling method in the present application.

[0167] See also Fig. 9 , Fig. 9 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium provided in the present application. The computer-readable storage medium 90 of the embodiment of the present application stores a program instruction 91, and when the program instruction 91 is executed, the scheduling method of the operator provided in the present application is implemented. Among them, the program instruction 91 can form a program file and be stored in the above-mentioned computer-readable storage medium 90 in the form of a software product, so that a computer device (which can be a personal computer, a server, or a network device, etc.) executes all or part of the steps of each implementation method of the present application. The aforementioned computer-readable storage medium 90 includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, or a terminal device such as a computer, a server, a mobile phone, and a tablet.

[0168] This embodiment also provides a computer program product, which includes a computer program. The computer program can be executed by a processor to implement the scheduling method of the above operator.

[0169] It can be understood that the computer program product can be a computer program product contained in a tangible computer-readable medium, and the computer program product contains program code for executing the scheduling method of the above operator. In some embodiments, the computer program product can be downloaded and installed from the network, or copied, transferred, and installed between different computer hardware. Its wireless transmission method can include the Internet, Bluetooth, WIFI, etc., and its wired transmission method can include USB, lighting, type-C, etc.

[0170] The above scheme divides the target task flow into multiple subtask flows, wherein the target task flow includes multiple operators, each subtask flow includes an operator slice corresponding to at least two operators, and the execution order of each operator slice in the subtask flow is the same as the data flow order of the target task flow; generates target time scheduling information for each subtask flow, and schedules the operator slices in each subtask flow according to the target time scheduling information of each subtask flow, wherein the target time scheduling information of the subtask flow includes the scheduling time of each operator slice in the subtask flow, and there are multiple different types of operator slices with the same scheduling time, and the scheduling time of the same type of operator slices is different. For a neural network accelerator using a spatial domain computing architecture, since different types of operator slices require different computing units in the kernel to execute, scheduling in the above manner can execute multiple different types of operator slices in parallel at the same scheduling time, realize parallel computing of computing units in the kernel, and thus improve the utilization rate of computing units in the kernel.

[0171] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0172] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0173] In the several embodiments provided in the present application, it should be understood that the disclosed methods, devices and systems can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0174] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0175] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0176] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0177] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for scheduling operators, characterized in that: The method comprises: Divide the target task flow into a plurality of subtask flows; wherein the target task flow includes a plurality of operators, each of the subtask flows includes an operator slice corresponding to at least two of the operators, and the execution order of each of the operator slices in the subtask flow is the same as the data flow order of the target task flow; Generate target time scheduling information for each of the subtask flows; wherein the target time scheduling information includes the scheduling time of each of the operator slices in the subtask flow, and there are multiple operator slices of different types having the same scheduling time, and the operator slices of the same type have different scheduling times; The operator slices in each of the subtask flows are scheduled according to the target time scheduling information.

2. The method according to claim 1, characterized in that The step of dividing the target task flow into a plurality of subtask flows includes: Performing loop block processing on each of the operators respectively to obtain first loop bodies corresponding to each of the operators; wherein the first loop bodies include multiple first iterative loops, and each of the first iterative loops includes an operator slice of the operator; Merging the first loop bodies to obtain at least one second loop body; wherein the second loop body includes multiple second iteration loops, and each of the second iteration loops includes an operator slice corresponding to at least two of the operators; Each of the second iteration loops is regarded as a subtask flow.

3. The method according to claim 2, characterized in that The merging of the first loop bodies to obtain at least one second loop body includes: Merging the first loop bodies that meet the preset merging condition to obtain the at least one second loop body; Among them, the preset merging conditions include: the loop parameters of each first loop body are the same, and there is a data dependency relationship between the operators corresponding to each first loop body; the loop parameters include the index upper boundary, index lower boundary and index iteration step of the outer loop.

4. The method according to claim 1, characterized in that The generating target time scheduling information of each subtask flow includes: Generate a number of first candidate scheduling mapping relationships; wherein the first candidate scheduling mapping relationships include a number of candidate scheduling times and operator slice sets corresponding to the candidate scheduling times, the operator slices belonging to the same subtask flow are respectively located in different operator slice sets, the types of the operator slices in the same operator slice set are different, and the maximum parallel number of operator slices of each of the first candidate scheduling mapping relationships is greater than a first preset number and less than or equal to the number of types of computing units required by the target task flow; Selecting a second candidate scheduling mapping relationship that meets the requirements from the plurality of first candidate scheduling mapping relationships; For each of the subtask flows, the candidate scheduling time corresponding to each of the operator slices of the subtask flow in the second candidate scheduling mapping relationship is used as the target time scheduling information of the subtask flow.

5. According to the method of claim 4, the generating of a plurality of first candidate scheduling mapping relationships comprises: generating a third candidate scheduling mapping relationship; The third candidate scheduling mapping relationship is adjusted to obtain the plurality of first candidate scheduling mapping relationships.

6. The method according to claim 5, characterized in that The maximum parallel number of operator slices of the third candidate scheduling mapping relationship is equal to the number of types of computing units; And / or, the adjusting the third candidate scheduling mapping relationship includes: Perform at least one of the following adjustment operations on the third candidate scheduling mapping relationship: Adjusting the candidate scheduling time corresponding to at least one of the operator slices in the third candidate scheduling mapping relationship; Adjust the maximum parallel number of the operator slices of the third candidate scheduling mapping relationship.

7. The method according to claim 4, characterized in that The selecting a second candidate scheduling mapping relationship that meets the requirements from the plurality of first candidate scheduling mapping relationships includes: Determine the estimated performance parameters corresponding to each of the first candidate scheduling mapping relationships; wherein the estimated performance parameters of the first candidate scheduling mapping relationship include at least one of the maximum parallel number of operator slices, the total number of operator slice scheduling times, and the total estimated execution time of the first candidate scheduling mapping relationship; For each of the first candidate scheduling mapping relationships, comprehensively considering each of the estimated performance parameters of the first candidate scheduling mapping relationship, and determining a first evaluation value of the first candidate scheduling mapping relationship; The second candidate scheduling mapping relationship is selected from the plurality of first candidate scheduling mapping relationships based on the first evaluation value of each of the first candidate scheduling mapping relationships.

8. The method according to claim 7, characterized in that The selecting the second candidate scheduling mapping relationship from the plurality of first candidate scheduling mapping relationships based on the first evaluation value of each of the first candidate scheduling mapping relationships includes: Selecting, from the plurality of first candidate scheduling mapping relationships, the first candidate scheduling mapping relationship whose first evaluation value meets the requirement as the second candidate scheduling mapping relationship; Alternatively, a plurality of fourth candidate scheduling mapping relationships whose first evaluation values ​​meet preset conditions are selected from the plurality of first candidate scheduling mapping relationships; Perform at least one running test on each of the fourth candidate scheduling mapping relationships to obtain at least one total test execution time corresponding to each of the fourth candidate scheduling mapping relationships; For each of the fourth candidate scheduling mapping relationships, determining a second evaluation value of the fourth candidate scheduling mapping relationship based on the at least one total test execution time of the fourth candidate scheduling mapping relationship; The fourth candidate scheduling mapping relationship whose second evaluation value meets the requirement is selected from the multiple fourth candidate scheduling mapping relationships as the second candidate scheduling mapping relationship.

9. The method according to claim 4, characterized in that The selecting a second candidate scheduling mapping relationship that meets the requirements from the plurality of first candidate scheduling mapping relationships includes: Perform at least one running test on each of the first candidate scheduling mapping relationships to obtain at least one total test execution time corresponding to each of the first candidate scheduling mapping relationships; For each of the first candidate scheduling mapping relationships, determining a third evaluation value of the first candidate scheduling mapping relationship based on the at least one total test execution time of the first candidate scheduling mapping relationship; The first candidate scheduling mapping relationship whose third evaluation value meets the requirement is selected from the plurality of first candidate scheduling mapping relationships as the second candidate scheduling mapping relationship.

10. A scheduling device for operators, characterized in that: The device comprises: A division module, used to divide the target task flow into multiple subtask flows; wherein the target task flow includes multiple operators, each of the subtask flows includes an operator slice corresponding to at least two of the operators, and the execution order of each of the operator slices in the subtask flow is the same as the data flow order of the target task flow; A generation module, used to generate target time scheduling information of each subtask flow; wherein the target time scheduling information includes the scheduling time of each operator slice in the subtask flow, and there are multiple different types of operator slices with the same scheduling time, and the scheduling time of the operator slices of the same type is different; A scheduling module is used to schedule the operator slices in each of the subtask flows according to the target time scheduling information.

11. An electronic device, characterized in that: comprising a memory and a processor coupled to each other, The memory stores program instructions; The processor is used to execute the program instructions stored in the memory to implement the method according to any one of claims 1 to 9.

12. A computer-readable storage medium / computer program product, characterized in that: The computer-readable storage medium is used to store program instructions, and the program instructions can be executed by a processor to implement the method according to any one of claims 1 to 9; The computer program product comprises a computer program, which can be executed by a processor to implement the method of any one of claims 1-9.

Images