Operator execution device, operator dispatching device, method, and chip

Abstract

The present disclosure provides an operator execution device, an operator scheduling device, a method, a chip, a computer device and a storage medium, wherein the operator execution device comprises an operator scheduler and an execution unit; the operator scheduler is configured to, in response to receiving an operator starting instruction of an operator to be started, based on operator type information carried in the operator starting instruction, distribute the operator starting instruction to an execution unit corresponding to the operator type information; the operator starting instruction comprises scheduling information of an operation unit in the execution unit; and the execution unit is configured to, in response to receiving the operator starting instruction distributed by the operator scheduler, based on the scheduling information carried in the operator starting instruction, execute the operator to be started.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer application, in particular, to an operator execution device, an operator scheduling device, a method, a chip, a computer device and a storage medium. BACKGROUND

[0002] In the field of deep learning, when a computer device runs a deep learning model, a deep learning framework deployed on the computer device parses the deep learning model to obtain operators in the deep learning model, and sends the operators to an Artificial Intelligence (AI) acceleration card, a Graphics Processing Unit (GPU) or the like execution device. After receiving the operators sent by the deep learning framework, the execution device is responsible for scheduling computing power, and performs a data processing task corresponding to the operators based on the scheduling result. This kind of data processing mode has the problem of low utilization rate of computing resources. SUMMARY

[0003] The present disclosure provides at least an operator execution device, an operator scheduling device, a method, a chip, a computer device and a storage medium.

[0004] In a first aspect, an embodiment of the present disclosure provides an operator execution device, comprising: an operator scheduler, and an execution unit;

[0005] The operator scheduler is configured to, in response to receiving an operator start instruction of a to-be-started operator, based on operator type information carried in the operator start instruction, issue the operator start instruction to an execution unit corresponding to the operator type information; the operator start instruction comprises scheduling information of an operation unit in the execution unit;

[0006] The execution unit is configured to, in response to receiving the operator start instruction issued by the operator scheduler, based on scheduling information carried in the operator start instruction, execute the to-be-started operator.

[0007] In a possible implementation, the scheduling information comprises at least one of the following:

[0008] an identifier of the to-be-started operator, a number of to-be-executed task blocks in the to-be-started operator, a task block identifier of a starting task block in the to-be-executed task blocks, a number of operation units for executing the to-be-executed task blocks, and an operation unit identifier for executing the to-be-executed task blocks;

[0009] Each task block comprises a plurality of sub-tasks in the to-be-started operator.

[0010] In a possible implementation, the execution unit includes a task block scheduler and a plurality of operation units.

[0011] The task block scheduler is configured to, in response to receiving the operator start instruction issued by the operator scheduler, determine, based on scheduling information carried in the operator start instruction, a target operation unit for executing the to-be-started operator and to-be-executed task blocks from the plurality of operation units, issue the to-be-executed task blocks to the target operation unit, and each of the to-be-executed task blocks includes a plurality of subtasks in the to-be-started operator.

[0012] The operation unit is configured to, in response to receiving the to-be-executed task blocks issued by the task block scheduler, execute data processing tasks corresponding to the to-be-executed task blocks.

[0013] In a possible implementation, the scheduling information includes a number of to-be-executed task blocks in the to-be-started operator and a task block identifier of a starting task block in the to-be-executed task blocks.

[0014] When determining a plurality of to-be-executed task blocks based on the scheduling information carried in the operator start instruction, the task block scheduler is configured to determine, based on the task block identifier of the starting task block and taking the starting task block as a starting point, a plurality of task blocks corresponding to the number of to-be-executed task blocks as the to-be-executed task blocks.

[0015] In a possible implementation, when issuing the to-be-executed task blocks to the target operation unit, the task block scheduler is configured to determine, based on the number of to-be-executed task blocks and the number of target operation units, to-be-executed task blocks for each target operation unit from the to-be-executed task blocks.

[0016] The to-be-executed task blocks determined for each target operation unit are issued to each target operation unit.

[0017] In a possible implementation, the scheduling information includes a number of operation units for executing the to-be-executed task blocks or an operation unit identifier for executing the to-be-executed task blocks.

[0018] When determining a target operation unit for executing the to-be-started operator from the plurality of operation units based on the scheduling information carried in the operator start instruction, the task block scheduler is configured to:

[0019] Based on the number of operation units for executing the to-be-executed task blocks or the operation unit identifier for executing the to-be-executed task blocks, the target operation unit for executing the to-be-started operator is determined from the plurality of operation units.

[0020] In a possible implementation, the operator start instruction comprises at least one of the following at least one field:

[0021] a second field for carrying size information of to-be-processed data corresponding to the to-be-started operator, a third field for carrying size information of one task block, a fourth field for carrying a required memory space size, and a fifth field for carrying a code address corresponding to the to-be-started operator.

[0022] In a second aspect, an operator execution method is provided, comprising:

[0023] The operator scheduler, in response to receiving an operator start instruction of a to-be-started operator, distributes the operator start instruction to an execution unit corresponding to operator type information carried in the operator start instruction based on the operator type information; the operator start instruction comprises scheduling information of an operation unit in the execution unit;

[0024] The execution unit, in response to receiving the operator start instruction distributed by the operator scheduler, executes the to-be-started operator based on scheduling information carried in the operator start instruction.

[0025] In a third aspect, an operator scheduling device is provided, comprising a scheduling policy generator and an instruction generator;

[0026] The scheduling policy generator is configured to generate a scheduling policy for scheduling operation units in the operator execution device when the operator execution device executes multiple operators of a deep learning model, and transmit the scheduling policy to the instruction generator;

[0027] The instruction generator is configured to generate an operator start instruction based on the scheduling policy, and send the operator start instruction to the operator execution device.

[0028] In a possible implementation, the scheduling policy comprises operator start time and scheduling information corresponding to each operator start time.

[0029] The scheduling information comprises at least one of the following: an identifier of the to-be-started operator, a number of to-be-executed task blocks in the to-be-started operator, a task block identifier of a starting task block in the to-be-executed task blocks, a number of operation units for executing the to-be-executed task blocks, and an operation unit identifier for executing the to-be-executed task blocks.

[0030] Each task block comprises multiple sub-tasks in the to-be-started operator.

[0031] In a possible implementation, the scheduling strategy generator is configured to generate, when the plurality of operators are executed by the operator execution device, a scheduling strategy for operation units in the operator execution device, and the scheduling strategy is used for:

[0032] analyzing the deep learning model to obtain a plurality of operators in the deep learning model;

[0033] determining, based on operation information corresponding to each of the plurality of operators and computing power information of the operator execution device, a scheduling strategy for operation units in the operator execution device when the plurality of operators are executed;

[0034] wherein each task block includes a plurality of sub-tasks in a corresponding operator; and the operator execution device executes the plurality of operators based on the scheduling strategy.

[0035] In a possible implementation, the operation information includes execution time of a plurality of task blocks corresponding to the plurality of operators respectively, and parallelism between the plurality of operators.

[0036] In a possible implementation, the scheduling strategy generator is further configured to obtain the execution time of the plurality of task blocks corresponding to the plurality of operators in the following manner:

[0037] estimating execution time required for executing each task block of the operator based on memory required by the operator when executed and computing power information of the operator execution device.

[0038] In a possible implementation, the scheduling strategy generator is configured to, when estimating the execution time required for executing each task block of the operator based on memory required by the operator when executed and computing power information of the operator execution device:

[0039] determining memory access time based on the memory required by the operator when executed and memory access bandwidth of the operation unit to the memory; and

[0040] determining computation time of the task block based on computing power information required by each computing step in each sub-task in the task block and computing power information of each operation unit;

[0041] determining execution time required for executing the task block based on the memory access time and the computation time.

[0042] In a possible implementation, the scheduling strategy generator is configured to obtain the execution time of the plurality of task blocks corresponding to the plurality of operators in the following manner:

[0043] determining, based on the operator, a simulation model corresponding to the operator;

[0044] running the simulation model, and determining execution time lengths of the respective task blocks of the operator based on a running time length of the simulation model.

[0045] In a possible implementation, the scheduling strategy generator, when determining the execution time lengths of the respective task blocks of the operator based on the running time length of the simulation model, is configured to:

[0046] determining the number of the task blocks obtained by dividing the operator in the simulation process according to a size of an operation unit in an operator execution device running the simulation model and an amount of data to be processed by the operator when performing the simulation;

[0047] determining the batch to be processed according to the number of the task blocks obtained by dividing the operator in the simulation process and a number of operation units in an operator execution device running the simulation model;

[0048] determining the execution time lengths of the respective task blocks of the plurality of operators based on the batch and the running time length of the simulation model.

[0049] In a possible implementation, the scheduling strategy generator, when determining the scheduling strategy of the operation units in the operator execution device when performing the plurality of operators based on the operation information corresponding to the plurality of operators respectively and the computing power information of the operator execution device, is configured to:

[0050] constructing an association relationship between the execution time lengths of the respective task blocks of the plurality of operators, the number of the respective task blocks of the plurality of operators, the parallelism degrees among the plurality of operators, the computing power information of the operator execution device, the strategy scheduling parameters, and the total execution time length of the plurality of operators;

[0051] adjusting the strategy scheduling parameters based on the association relationship to obtain a target scheduling strategy, with the objective of reducing the total execution time length.

[0052] In a possible implementation, the scheduling strategy generator, when constructing the association relationship between the execution time lengths of the respective task blocks of the plurality of operators, the number of the respective task blocks of the plurality of operators, the parallelism degrees among the plurality of operators, the computing power information of the operator execution device, the strategy scheduling parameters, and the total execution time length of the plurality of operators, is configured to:

[0053] The execution time length of each of the plurality of task blocks corresponding to the plurality of operators, the number of task blocks corresponding to the plurality of operators, and the computing power information of the operator execution device are taken as parameters, the scheduling strategy parameter is taken as an independent variable, the total execution time length of the plurality of operators is taken as a dependent variable, and the parallelism between operators is taken as a constraint condition to construct a relational equation; and the relational equation is taken as the association relationship.

[0054] In a possible implementation, the instruction generator, when generating the operator start instruction based on the scheduling strategy and sending the operator start instruction to the operator execution device, is configured to:

[0055] generate the operator start instruction based on the scheduling information;

[0056] send any operator start instruction to the operator execution device in response to the instruction sending time point corresponding to the operator start instruction being reached, wherein the instruction sending time point is determined based on the instruction execution time point corresponding to the operator start instruction.

[0057] In a possible implementation, the operator start instruction includes at least one of the following at least one field:

[0058] a second field for carrying size information of to-be-processed data corresponding to the to-be-started operator, a third field for carrying size information of one task block, a fourth field for carrying a required memory space size, and a fifth field for carrying a code address corresponding to the to-be-started operator.

[0059] In a fourth aspect, the embodiments of the present disclosure further provide an operator scheduling method, including:

[0060] A scheduling strategy generator generates a scheduling strategy for scheduling operation units in the operator execution device when the plurality of operators execute the deep learning model by using the operator execution device; and transmits the scheduling strategy to an instruction generator;

[0061] The instruction generator generates an operator start instruction based on the scheduling strategy, and sends the operator start instruction to the operator execution device.

[0062] In a fifth aspect, the embodiments of the present disclosure further provide a chip, including the operator execution device according to any one of the first aspect, and / or the operator scheduling device according to any one of the third aspect.

[0063] In a sixth aspect, the optional implementation of the present disclosure further provides a computer device, including the chip according to the fifth aspect.

[0064] In the seventh aspect, an optional implementation of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is run, it executes the operator execution method as described in the second aspect, or any one of the second aspects; or executes the operator scheduling method as described in the fourth aspect, or any one of the fourth aspects.

[0065] The operator execution device provided by the embodiment of the present disclosure includes an operator scheduler and an execution unit; the operator scheduler can respond to receiving the operator start instruction of the operator to be started and send the operator start instruction to the corresponding execution unit; after receiving the operator start instruction sent by the operator scheduler, the execution unit executes the operator to be started based on the scheduling information carried in the operator start instruction. In this process, the scheduling information is not generated by the operator execution device, but is executed by the device that initiates the operator start instruction, so that the host that sends the operator start can plan the specific process of executing the operator on the execution device in advance, control the issuance of the operator, and improve the utilization rate of the computing resources in the execution device.

[0066] The operator scheduling device provided by the embodiment of the present disclosure includes a scheduling policy generator and an instruction generator; the scheduling policy generator can generate a scheduling policy for scheduling the computing units in the operator execution device when the operator execution device executes the operator of the deep learning model, and transmit the scheduling policy to the instruction generator; the instruction generator can generate an operator start instruction based on the scheduling policy, and send the operator start instruction to the operator execution device, so that the operator execution device executes the corresponding operator based on the operator start instruction sent by the operator scheduling device, thereby being able to plan the specific process of executing the operator on the execution device in advance, control the distribution of the operator, and improve the utilization rate of the computing resources in the execution device.

[0067] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the embodiments. The drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without inventive effort.

[0069] Figure 1 A schematic diagram of an operator execution device provided by an embodiment of the present disclosure is shown;

[0070] Figure 2 Fig. 1 shows a schematic diagram of a specific example of performing a deep learning task by an operator scheduling device and an operator execution device working cooperatively according to an embodiment of the present disclosure;

[0071] Figure 3 Fig. 2 shows a schematic diagram of an operator scheduling device according to an embodiment of the present disclosure;

[0072] Figure 4 Fig. 3 shows a format example of an operator starting instruction according to an embodiment of the present disclosure;

[0073] Figure 5 Fig. 4 shows a flowchart of an operator execution method according to an embodiment of the present disclosure;

[0074] Figure 6 Fig. 5 shows a flowchart of an operator scheduling method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0075] In order to make the objectives, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure and not all the embodiments. The components of the embodiments of the present disclosure described and shown herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure is not intended to limit the scope of the claimed present disclosure, but only represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present disclosure.

[0076] It is found through research that, with the wide use of artificial intelligence, the model and data volume of deep learning are increasing, which naturally leads to a large increase in operators or operations that need to be executed by computers, and the proportion of scheduling overhead of computing resources in computers is increasing. On the one hand, a host (Host) running a deep learning framework such as TensorFlow, Pytorch, Paddlepaddle, etc. obtains operators in a deep model by analyzing various deep learning models, and simply dispatches various operators to an operator execution device; the operator execution device is responsible for scheduling and executing the operators. The host cannot know the state of the computing resources in the execution device, and thus cannot control the delivery of operators according to the state of the computing resources in the execution device, resulting in a decrease in the utilization rate of computing resources of the operator execution device.

[0077] On the other hand, in the process of training a neural network, the training system of the deep learning model needs to perform operators, including simple calculation class operators and communication class operators. The simple calculation class operators include, for example, convolution operators and full connection operators. The communication class operators include, for example, allreduce operators. When the execution device executes these two kinds of operators, the calculation and communication will compete for the computing resources. This not only causes huge fluctuations or jitters in the calculation, but also significantly reduces the efficiency of the training system.

[0078] Based on the above research, the present disclosure provides an operator execution device, which includes an operator scheduler and an execution unit. The operator scheduler can issue an operator start instruction to the corresponding execution unit in response to receiving the operator start instruction of the to-be-started operator. The execution unit executes the to-be-started operator based on the scheduling information carried in the operator start instruction after receiving the operator start instruction issued by the operator scheduler. In this process, the scheduling information is not generated by the operator execution device, but by the device that initiates the operator start instruction. Therefore, the host that issues the operator start instruction can control the issuance of the operator based on the pre-planning of the specific process of the execution device executing the operator, thereby improving the utilization efficiency of the computing resources.

[0079] In addition, the present disclosure also provides an operator scheduling device, which includes a scheduling strategy generator and an instruction generator. The scheduling strategy generator can generate a scheduling strategy for scheduling the operation unit in the operator execution device when the operator execution device executes the deep learning model, and transmit the scheduling strategy to the instruction generator. The instruction generator can generate an operator start instruction based on the scheduling strategy, and send the operator start instruction to the operator execution device, so that the operator execution device executes the corresponding operator based on the operator start instruction sent by the operator scheduling device. Therefore, the operator scheduling device can control the issuance of the operator based on the pre-planning of the specific process of the execution device executing the operator, thereby improving the utilization rate of the computing resources in the execution device.

[0080] The above-mentioned defects are the results of the inventors' practice and careful research, and therefore, the discovery process of the above-mentioned problems and the solutions proposed by the present disclosure to solve the above-mentioned problems should be the contributions of the inventors to the present disclosure.

[0081] It should be noted that similar reference numerals and letters refer to similar items throughout the accompanying drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0082] For the convenience of understanding the embodiments, first, the operator execution device disclosed by the embodiments of the present disclosure is introduced in detail. The operator execution device provided by the embodiments of the present disclosure is generally deployed in a computer device. The computer device includes, for example, a terminal device or a server or other processing device. The terminal device can be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc.

[0083] Referring to Figure 1 As shown in FIG. 1, the structure of an operator execution device provided by the embodiments of the present disclosure is shown. The operator execution device includes an operator scheduler 10 and an execution unit 20.

[0084] The operator scheduler 10 is configured to, in response to receiving an operator start instruction of a to-be-started operator, based on operator type information carried in the operator start instruction, distribute the operator start instruction to an execution unit corresponding to the operator type information; the operator start instruction includes scheduling information of an operation unit in the execution unit.

[0085] The execution unit 20 is configured to, in response to receiving the operator start instruction distributed by the operator scheduler, based on scheduling information carried in the operator start instruction, execute the to-be-started operator.

[0086] In specific implementation, the operator execution device includes, for example, an artificial intelligence (Artificial Intelligence, AI) chip, a graphics processing unit (Graphics Processing Unit, GPU), etc. The execution unit 40 in the operator execution device includes, for example, a data processing core in the AI chip or the GPU.

[0087] The execution unit 20 includes, for example, a task block scheduler 201 and a plurality of operation units 202.

[0088] The task block scheduler 201 is configured to, in response to receiving the operator start instruction distributed by the operator scheduler 10, based on scheduling information carried in the operator start instruction, determine a target operation unit for executing the to-be-started operator and determine a to-be-executed task block from the plurality of operation units 202; distribute the to-be-executed task block to the target operation unit; each to-be-executed task block includes a plurality of sub-tasks in the to-be-started operator.

[0089] The operation unit 202 is configured to execute a data processing task corresponding to the to-be-executed task block in response to receiving the to-be-executed task block issued by the task block scheduler 201.

[0090] The start instruction received by the operator scheduler 10 is generated by an operator scheduling device. The specific manner in which the operator scheduling device generates the operator start instruction can be found in the following Figure 3 The specific embodiments are shown in the corresponding embodiments and will not be described here.

[0091] As shown in Figure 2 The embodiments of the present disclosure provide a specific example of performing a deep learning task by cooperation of an operator scheduling device and an operator execution device.

[0092] The operator scheduling device is configured to parse a deep learning model to obtain an operator, generate a scheduling strategy for scheduling an operation unit, and generate an operator start instruction according to the scheduling strategy and issue the operator start instruction to a flow queue. The operator scheduling device may, for example, be a central processing unit in a computer device.

[0093] The flow queue is located between the operator scheduling device and the operator execution device and is configured to store an operator start instruction that has not been read by the operator execution device.

[0094] The operator execution device includes an operator scheduler and an execution unit, and the execution unit includes a task block scheduler and a plurality of operation units.

[0095] The operator scheduler obtains an operator start instruction from the flow queue (SQ) and distributes the operator start instruction to different execution units based on the operator type carried in the operator start instruction. The execution unit has multiple types, and different types of execution units can execute different operators. For example, the operator type includes a calculation operator for executing a calculation task, an operator for executing a data copy task, and an operator for performing a data synchronization task. Correspondingly, due to the difference in structure, the execution unit of different types can execute different types of operators. The execution unit executes the corresponding operator after receiving the operator start instruction.

[0096] In an execution unit for performing a computing task, a block scheduler and a plurality of operation units are included. The block scheduler is configured to receive an operator start instruction dispatched by an operator scheduler, and decompose the operator start instruction into a plurality of blocks, each of which includes a plurality of sub-tasks. The block scheduler is capable of obtaining the working state of the operation units, and dispatch the blocks to the corresponding operation units according to the scheduling information of the scheduling strategy carried in the operator start instruction after obtaining the blocks. The operation units execute the computing task corresponding to the blocks after receiving the blocks.

[0097] The operation units can send a level signal to the block scheduler to indicate that the operation units are in an idle state after completing the computing task corresponding to the blocks, and the block scheduler obtains the working state of the operation units according to the level signal.

[0098] Each operation unit is generally composed of a two-dimensional processing engine (PE) array and a local register file. Each PE includes a multiplier-adder and other computing elements for executing specific computing tasks of each block. Each operation unit can simultaneously process multiple data. For example, an operation unit includes S PEs in the PE array, and the S PEs can simultaneously process S data. That is, the operation unit can simultaneously process at most S data. The task of processing each data is called a sub-task, and S sub-tasks completed by S PEs in an operation unit form a block.

[0099] The number of blocks corresponding to an operator is related to the configuration of the operation units. Assuming that the number of data to be processed corresponding to an operator is H, and the number of PEs in the PE array in each operation unit is S, then the number of blocks corresponding to the operator is an integer obtained by rounding up H / S. For example, the data to be processed is image data, and the size of the image data is w*h*c, where w represents the image width, h represents the image height, and c represents the number of channels of the image. The number of data to be processed H satisfies H=w*h*c.

[0100] The scheduling information carried in the operator start instruction includes at least one of the following:

[0101] The identifier of the operator to be started, the number of blocks to be executed in the operator to be started, the block identifier of the starting block in the blocks to be executed, the number of operation units for executing the blocks to be executed, and the identifier of the operation units for executing the blocks to be executed.

[0102] The identifier of the to-be-started operator can be represented by a code address of the to-be-started operator. The execution code of the operator can be obtained through the code address, and the data processing task corresponding to the operator can be implemented by executing the execution code of the operator.

[0103] The to-be-started operator refers to an operator that needs to be started at the starting moment of the operator.

[0104] The number of to-be-executed task blocks in the to-be-started operator refers to the number of task blocks that need to be executed corresponding to the starting moment of the operator.

[0105] The starting task block identifier of the to-be-executed task block refers to the identifier of the first task block in the plurality of task blocks that need to be executed corresponding to the starting moment of the operator.

[0106] The number of operation units for executing the to-be-executed task block refers to the number of operation units to be allocated to the to-be-started operator when the to-be-executed task block in the to-be-started operator is executed by using operation units.

[0107] The operation unit identifier for executing the to-be-executed task block refers to the specific identifier of the operation unit to be allocated to the to-be-started operator when the to-be-executed task block in the to-be-started operator is executed by using operation units.

[0108] For example, if the scheduling information includes the number of to-be-executed task blocks in the to-be-started operator and the task block identifier of the starting task block in the to-be-executed task block, when the task block scheduler determines the plurality of to-be-executed task blocks based on the scheduling information carried in the operator starting instruction, it is used to: based on the task block identifier of the starting task block, and taking the starting task block as the starting point, determine the plurality of task blocks corresponding to the number of to-be-executed task blocks as the to-be-executed task blocks.

[0109] After determining the to-be-executed task block, the task block scheduler issues the to-be-executed task block to the operation unit; the operation unit receiving the task block executes the received task block.

[0110] In addition, when the task block scheduler issues the to-be-executed task block to the operation unit, it is used to determine the to-be-executed task block to be issued to each target operation unit from the to-be-executed task block based on the number of to-be-executed task blocks and the number of target operation units; each target operation unit is issued the to-be-executed task block determined for each target operation unit.

[0111] In the task block scheduler, the target operation unit can be determined based on the number of operation units that execute the to-be-executed task block or the identification of the operation units that execute the to-be-executed task block. When the scheduling information does not include the number of operation units that execute the to-be-executed task block or the identification of the operation units that execute the to-be-executed task block, the target operation unit can be determined according to the working state of each operation unit.

[0112] In the operator execution device provided by the embodiments of the present disclosure, the scheduling information is not generated by the operator execution device when the operator execution device executes an operator, but is executed by a device that initiates an operator starting instruction, so that a host that issues an operator starting instruction can control the issuance of an operator according to a specific process of executing an operator by an execution device in advance, and improve the utilization efficiency of computing resources.

[0113] Referring to Figure 3 As shown in the figure, an operator scheduling device provided by the embodiments of the present disclosure includes a scheduling policy generator 30 and an instruction generator 40.

[0114] The scheduling policy generator 30 is configured to generate a scheduling policy for scheduling operation units in the operator execution device when the operator execution device executes an operator of a deep learning model, and transmit the scheduling policy to the instruction generator 40.

[0115] The instruction generator 40 is configured to generate an operator starting instruction based on the scheduling policy, and send the operator starting instruction to the operator execution device.

[0116] In a specific implementation, the operator scheduling device can include, for example, a central processing unit in a computer device. The operator scheduling device can be used to run, for example, a deep learning framework or a deep model training system. When the operator scheduling device runs the deep learning framework, the operator scheduling device executes a deep learning model. When the operator scheduling device runs the deep model training system, the operator scheduling device trains a to-be-trained deep learning model.

[0117] For example, when the operator scheduling device executes a deep learning model or trains a deep learning model, the operator scheduling device can analyze the deep learning model to obtain a plurality of operators included in the deep learning model. The operators can include, for example, a calculation operator and a communication operator. The calculation operator can include, for example, a convolution operator, a full connection operator, an activation operator, and the like. The communication operator can include, for example, a global reduction operator allreduce, a global collection operator allgather, and a syncBN.

[0118] When generating a scheduling policy, the scheduling policy generator 30 in the operator scheduling device can perform, for example:

[0119] parsing the deep learning model to obtain a plurality of operators in the deep learning model;

[0120] determining, based on operation information corresponding to each of the plurality of operators and computing power information of the operator execution device, a scheduling strategy for operation units in the operator execution device when executing the plurality of operators;

[0121] Each task block includes a plurality of sub-tasks in the corresponding operator. The operator execution device is scheduled to execute the plurality of operators based on the scheduling strategy.

[0122] The scheduling strategy includes, for example, an operator start time and scheduling information corresponding to each operator start time.

[0123] The scheduling information includes at least one of the following: an identifier of the operator to be started, a number of task blocks to be executed in the operator to be started, a task block identifier of a starting task block in the task block to be executed, a number of operation units for executing the task block to be executed, and an operation unit identifier for executing the task block to be executed.

[0124] The operation information includes execution duration of the plurality of task blocks corresponding to the plurality of operators and parallelism between the plurality of operators.

[0125] The execution duration of the task block is, for example, the duration required by the operation unit to execute a plurality of sub-tasks corresponding to one task block. In a task block, it usually includes a plurality of sub-tasks that can be executed in parallel by a plurality of PEs in a PE array. Therefore, the task block execution duration corresponding to the task block is the same as or approximately equal to the execution duration of executing one sub-task. The sub-task is, for example, a processing task for any data in image data.

[0126] In another embodiment of the present disclosure, the scheduling strategy generator is further configured to obtain execution duration of a plurality of task blocks corresponding to the plurality of operators before determining the scheduling strategy. The determination of the execution duration of the task block can be performed in at least one of the following (1) or (2):

[0127] (1) Based on the memory required by the task block during execution and the computing power information of the operation unit, the execution duration required for executing the task block is estimated.

[0128] In a specific implementation, the computing power information of the operation unit includes, for example, the number of operations per second (ops) performed by the operation unit per second. The computing power information of each operation unit can be determined in the following manner: the number of PEs included in the operation unit is determined, where one PE can perform n operations per second, representing the computing power information of one PE; the computing power information of the operation unit, i.e., the total number of operations per second (OPS) of one operation unit, is determined according to the number of PEs included in the operation unit and the computing power information of each PE, where the computing power information of the operation unit satisfies R*P*n. Here, R represents the number of PEs in one operation unit, and P represents the number of operation units included in the execution unit.

[0129] When estimating the execution duration required for executing the task block based on the memory required by the task block during execution and the computing power information of the operation unit, the memory access duration can be determined based on the memory required by the operator during execution and the memory access bandwidth of the operation unit; and the computation duration of the task block can be determined based on the computing power information required by each computing step in each subtask in the task block and the computing power information of each operation unit.

[0130] The execution duration required for executing the task block is determined based on the memory access duration and the computation duration.

[0131] The memory required by the task block during execution can represent the amount of data to be processed by the task block during execution. Since the type of the operator is known, the computing steps of each subtask during the execution of the operator can be determined. Therefore, the ops required by the PE during the execution of a computing operation can be determined, for example, the PE is a multiplier-adder that can be used to perform multiplication and addition operations, and 2 ops are required for each multiplication and addition operation. The ops required by each computing step can be determined according to the ops required by the corresponding operation of each computing step. The computation duration of each subtask is determined according to the specific computing steps of the subtask. Since one task block includes multiple subtasks executed in parallel by multiple PEs, the computation duration of each subtask, i.e., the computation duration of the task block, can be determined.

[0132] The memory access duration of a task block is determined according to the amount of data accessed by the memory access task corresponding to the task block and the memory access bandwidth.

[0133] The execution duration required for executing one task block is determined based on the computation duration and the memory access duration.

[0134] For example, the operator is a matrix multiplication operator, and the matrix multiplication task in each task block satisfies CMN=AMK *B KN Wherein, M and K represent the size of operand A respectively; K and N represent the size of operand B respectively; each data needs to occupy b bits, and the data amount (memory access data amount) required by the operator to access the memory during execution is (M*K+K*N+M*N)*b / 8 bytes. Wherein, M*K represents the data amount of operand A required to be read from the memory; K*N represents the data amount of operand B required to be read from the memory; and M*N represents the data amount of the calculation result to be stored in the memory. The number of multipliers (i.e., PEs) in each operation unit in the execution unit for executing the operator is M*K*N.

[0135] The calculation time required for executing one task block of the operator is n*M*K*N ÷ total OPS number of operation units, and the memory access time required by the operation unit for executing one task block of the operator is memory access data amount ÷ memory access bandwidth.

[0136] The execution time of the first task block corresponding to the operator includes the sum of the calculation time and the memory access time, and thus the execution time of one task block in the matrix multiplication operator can be obtained as n*M*K*N ÷ total OPS number of operation units + memory access data amount ÷ memory access bandwidth.

[0137] (2) Based on the operator, a simulation model corresponding to the operator is determined; the simulation model is run, and based on the running time of the simulation model, the execution time of each task block of the operator is determined.

[0138] In a specific implementation, the simulation model corresponding to the operator determined based on the operator only includes the operator and does not include other operators. Wherein, the parameters of the operator in the simulation model are consistent with the parameters of the operator in the deep learning model. Then the simulation model is run to obtain the running time of the simulation model.

[0139] Since the size W*H and the number S of operation units in the operator execution device running the simulation model are known, the number of task blocks obtained by dividing the operator in the simulation process can be determined according to the size W*H of the operation unit in the operator execution device running the simulation model and the data amount U required to be processed by the operator, and then since multiple operation units can synchronously process multiple task blocks, the batch U / (W*H) / S required to be processed can be determined according to the number of task blocks and the number S of operation units. Then the execution time of each task block is obtained according to the running time T of the simulation model and the batch to be processed: T / (U / (W*H) / S)

[0140] Here, when running the simulation model, the configuration of the execution device used may be consistent with or inconsistent with the configuration of the operator execution device when running the deep learning model.

[0141] After parsing the deep learning model and obtaining multiple operators included in the deep learning model, the scheduling strategy generator determines the execution time of the task blocks in each operator according to the above method (1) or (2), and determines the scheduling strategy for the computing unit in the operator execution device according to the computing information corresponding to the multiple operators and the computing power information of the operator execution device, and transmits the scheduling strategy to the instruction generator.

[0142] The operation information corresponding to each operator includes, for example: the execution duration of the task blocks corresponding to the multiple operators, and the parallelism between the multiple operators; wherein each task block includes multiple subtasks in the corresponding operator.

[0143] The embodiment of the present disclosure further provides a specific method for the scheduling strategy generator 30 to determine the scheduling strategy for the operation unit when executing multiple operators, including:

[0144] Establish a correlation between the execution time of the task blocks corresponding to the multiple operators, the number of task blocks corresponding to the multiple operators, the degree of parallelism between the multiple operators, the computing power information of the operator execution devices, the scheduling policy parameters, and the total execution time of the multiple operators. With the goal of reducing the total execution time, adjust the scheduling policy parameters based on the correlation to obtain a target scheduling policy.

[0145] The scheduling strategies include:

[0146] Operator start time and scheduling information corresponding to each operator start time;

[0147] Scheduling policy parameters can be used to characterize the scheduling policy. For example, when the scheduling policy parameters include at least one execution time corresponding to each of the multiple operators, a task block identifier for a starting task block corresponding to each of the at least one execution time, and the number of required computing units, upon the arrival of any operator execution time, the host sends an operator start instruction to the operator execution device, carrying two pieces of information: the task block identifier of the corresponding starting task block and the number of required computing units. Upon receiving the operator start instruction, the operator execution device schedules computing units based on the number of required computing units and controls the scheduled computing units to execute the task block based on the task block identifier.

[0148] The correlation relationship can be embodied in the form of an equation, that is, a relationship equation is constructed with the execution time of the task blocks corresponding to the plurality of operators respectively, the number of the task blocks corresponding to the plurality of operators respectively, the computing power information of the operator execution devices as parameters, the scheduling policy parameter as an independent variable, the total execution time of the plurality of operators as a dependent variable, and the parallelism between the operators as a constraint condition, and the total execution time is optimized by continuously adjusting the independent variable in the relationship equation to reduce the total execution time, and finally the scheduling policy obtained in the case where the total execution time reaches a minimum (for example, in a plurality of optimization iteration periods, the total execution time no longer decreases) is taken as the target scheduling policy.

[0149] In the process of continuously adjusting the independent variable, a plurality of iteration periods can be executed, for example:

[0150] In the 0th iteration period, the scheduling policy parameter is determined, and the total execution time of the 0th iteration period is determined according to the scheduling policy parameter and the relationship equation. The scheduling policy parameter is taken as the reference scheduling policy parameter of the 1st iteration period, and the total execution time of the 0th iteration period is taken as the reference total execution time of the 1st iteration period, and the 1st iteration period is entered.

[0151] In the 1st iteration period, the reference scheduling policy parameter of the 1st iteration period is adjusted, a new total execution time is determined based on the adjusted scheduling policy parameter and the relationship equation, the new total execution time and the reference total execution time are compared, if the new total execution time is less than the reference total execution time, the adjusted scheduling policy parameter determined in the 1st iteration period is taken as the reference scheduling policy parameter of the 2nd iteration period, and the new total execution time is taken as the reference total execution time of the 2nd iteration period, and the 2nd iteration period is entered, if the new total execution time is greater than or equal to the reference total execution time, the reference scheduling policy parameter of the 1st iteration period is taken as the reference scheduling policy parameter of the 2nd iteration period, and the reference total execution time of the 1st iteration period is taken as the reference total execution time of the 2nd iteration period, and the 2nd iteration period is entered.

[0152] …

[0153] In the ith iteration cycle, the reference scheduling strategy parameter of the ith iteration cycle is adjusted; based on the adjusted scheduling strategy parameter and the relationship equation, a new total execution duration is determined; the new total execution duration and the reference total execution duration are compared; if the new total execution duration is less than the reference total execution duration, the adjusted scheduling strategy parameter determined in the ith iteration cycle is taken as the reference scheduling strategy parameter of the (i+1)th iteration cycle, and the new total execution duration is taken as the reference total execution duration of the (i+1)th iteration cycle, and the (i+1)th iteration cycle is entered; if the new total execution duration is greater than or equal to the reference total execution duration, the reference scheduling strategy parameter of the ith iteration cycle is taken as the reference scheduling strategy parameter of the (i+1)th iteration cycle, and the reference total execution duration of the ith iteration cycle is taken as the reference total execution duration of the (i+1)th iteration cycle, and the (i+1)th iteration cycle is entered.

[0154] The above process is iteratively executed until the reference total execution duration no longer changes in the consecutive n iteration cycles.

[0155] In another embodiment, the execution strategy of the above operator can be determined according to the deep learning model and the operator execution device before executing the deep learning model, or can be a scheduling strategy corresponding to each operator execution device determined in advance for a certain deep learning model when the deep learning model is executed on multiple operator execution devices; when the deep learning model is executed, the scheduling strategy corresponding to the used operator execution device can be directly called through the provided interface.

[0156] It should be noted that the present disclosure does not limit the equation form corresponding to the above-mentioned relationship, and a user using the scheme provided by the present disclosure can determine the corresponding equation form according to his own needs and the hardware environment (such as the number of operation units, computing power, tasks to be executed, etc.) applying the scheme provided by the present disclosure, or determine the corresponding equation form in combination with a certain algorithm, and adjust the equation by the above-mentioned iteration method, and obtain the corresponding scheduling strategy parameter under the condition that the total execution duration reaches the minimum, that is, obtain the scheduling strategy.

[0157] For example, after the scheduling strategy generator analyzes the deep learning model, the obtained operators include A, B, C, and D. The parallelism between A, B, C, and D includes that A and B need to be executed sequentially, C and D need to be executed sequentially, and AB and CD have no dependency, that is, AC, AD, BC, and BD can be executed in parallel; the execution priority of operator B is higher than that of operators C and D.

[0158] According to any one of the above (1) or (2), the number of task blocks corresponding to A, B, C, and D respectively and the execution duration required for each task block to run are as shown in Table 1 respectively:

[0159] Table 1

[0160] Operator thread cycle A 68 20 B 8 80 C 60 20 D 48 20

[0161] Wherein, thread represents a task block, and cycle represents the execution duration of the task block, in units of processing cycles.

[0162] In the operator execution device, the number of operation units is 16; each operation unit can synchronously process multiple sub-tasks in a task block.

[0163] Therefore, the theoretical time consumption of serially executing each operator respectively by using the operator execution device is shown in Table 2 as follows:

[0164] Table 2

[0165] Operator thread cycle Serial scheduling Theoretical time consumption A 68 20 100 1360 B 8 80 80 640 C 60 20 80 1200 D 48 20 60 960 total 320 260

[0166] Serial scheduling represents the number of processing cycles required when executing any operator.

[0167] Taking operator A as an example, it includes 68 task blocks, and since the operator execution device includes 16 operation units, the calculation batch required to execute operator A is: 68 / 16 = 4.25; since multiple operators are executed in series, all operation units will be scheduled during the processing of any batch, regardless of whether all operation units will actually be used or not, therefore, 5 batches are required to implement the processing of the operator. Each batch occupies 20 processing cycles, so the processing cycles required by 5 batches are 20*5 = 100, that is, 100 processing cycles are required to process the task corresponding to operator A.

[0168] Theoretical time consumption: represents the theoretical number of processing cycles required to execute operator A. Taking operator A as an example, it includes 68 task blocks, and the execution duration of each task block is 20 processing cycles, so the theoretical number of processing cycles is: 68*20 = 1360 processing cycles.

[0169] It can be seen that the total execution duration of the four operators is: 320 processing cycles.

[0170] However, theoretically, the total execution duration of the four operators = sum(thread*cycle) / 16 = 260.

[0171] If the prior art is used to schedule operation units by using a GPU, the specific scheduling process is shown in Table 3 as follows:

[0172] Table 3

[0173]

[0174]

[0175] Wherein, the time represents: the i th processing period. For the above four operators A-D:

[0176] In the 0 th processing period, 16 operation units are allocated to operator A, and 80 processing periods are executed;

[0177] In the 80 th processing period, 4 operation units are allocated to operator A, 20 processing periods are executed, and operator A is executed completely; and 12 operation units are allocated to operator C, and 20 processing periods are executed.

[0178] In the 100 th processing period, since the execution priority of operator B is higher than that of operator C, 8 operation units are allocated to operator B, 80 processing periods are executed first, and operator B is executed completely; the idle 8 operation units are allocated to operator C, and 80 processing periods are executed;

[0179] In the 180 th processing period, 16 operation units are allocated to operator C, and 20 processing periods are executed.

[0180] In the 200 th processing period, 4 operation units are allocated to operator C, and 20 processing periods are executed, and operator C is executed completely.

[0181] In the 220 th processing period, 16 operation units are allocated to operator D, and 60 processing periods are executed, and operator D is executed completely.

[0182] To the 280 th processing period, operators A-D are all executed completely.

[0183] If the scheduling method provided by the embodiment of the present disclosure is used, the specific process of scheduling is shown in Table 4 as follows:

[0184] Table 4

[0185] time A B C D 0 16x20 20 8x120 8x120 140 4x20 12x20 160 8x80 8x80 240 16x20 260 done

[0186] Wherein, for the above four operators A-D:

[0187] In the 0 th processing period, 16 operation units are allocated to operator A, and 20 processing periods are executed.

[0188] In the 20 th processing period, 8 operation units are allocated to operator A, and 120 processing periods are executed, and 8 operation units are allocated to operator C, and 120 processing periods are executed.

[0189] In the 140th processing cycle, 4 operation units are allocated to the operator A, 20 processing cycles are executed, and the operator A is executed; and 12 operation units are allocated to the operator C, 20 processing cycles are executed, and the operator C is executed.

[0190] In the 160th processing cycle, 8 operation units are allocated to the operator B, 80 processing cycles are executed, and the operator B is executed; and 8 operation units are allocated to the operator D, 80 processing cycles are executed.

[0191] In the 240th processing cycle, 16 operation units are allocated to the operator D, 20 processing cycles are executed, and the operator D is executed.

[0192] In the 260th processing cycle, the operators A to D are all executed.

[0193] Therefore, the GPU scheduling needs 280 processing cycles, and the scheduling method provided in the embodiment of the present disclosure needs only 260 processing cycles, so that the scheduling method provided in the embodiment of the present disclosure can improve the execution efficiency of the operator, can make more sufficient use of the computing power, and improve the utilization rate of the computing power.

[0194] The instruction generator 40 may, for example, but not limited to, generate the operator start instruction in any one of the following A or B, and send the operator start instruction to the operator execution device:

[0195] A: In response to the arrival of each instruction start time, write the scheduling information corresponding to each instruction start time into the preset bit of the operator start instruction, generate the operator start instruction corresponding to each operator start time, and send the operator start instruction corresponding to each operator start time to the operator execution device.

[0196] In this example, the operator start instruction is generated after the arrival of the operator start time, and is sent to the operator execution device.

[0197] B: For each operator start time, write the scheduling information corresponding to each operator start time into the preset bit of the operator start instruction, generate the operator start instruction corresponding to each operator start time, and in response to the arrival of each operator start time, send the operator start instruction corresponding to each operator start time to the operator execution device.

[0198] In this example, after obtaining the scheduling strategy, the operator start instruction can be generated based on the scheduling strategy first; and after the arrival of any operator start time, the corresponding operator start instruction is sent to the operator execution device.

[0199] When the scheduling information corresponding to each instruction start time is written into the preset bit of the operator start instruction, and the operator start instruction corresponding to each operator start time is generated, the following method can be used, for example:

[0200] For each to-be-started operator corresponding to each instruction start time, the scheduling information corresponding to the to-be-started operator is written into the preset bit of the operator start instruction corresponding to the to-be-started operator, to generate the operator start instruction corresponding to the to-be-started operator.

[0201] For example, in the example corresponding to Table 4, the operator start times t0-t4 are respectively:

[0202] t0: the 0th processing period;

[0203] t1: the 20th processing period;

[0204] t2: the 140th processing period;

[0205] t3: the 160th processing period;

[0206] t4: the 240th processing period.

[0207] Then, for the operator start time t0, the operator start instruction L0-A for the operator A is generated;

[0208] For the operator start time t1, the operator start instruction L1-A for the operator A and the operator start instruction L1-C for the operator C are generated;

[0209] For the operator start time t2, the operator start instruction L2-A for the operator A and the operator start instruction L2-C for the operator C are generated;

[0210] For the operator start time t3, the operator start instruction L3-B for the operator B and the operator start instruction L3-D for the operator D are generated;

[0211] For the operator start time t4, the operator start instruction L4-D for the operator D is generated.

[0212] For the instruction sending time corresponding to any operator start instruction, for example, after the instruction start time corresponding to the operator start instruction is determined, the instruction start time is reduced by at least one processing period to obtain the instruction sending time. For example, in the example shown in Table 4, the 16 start units need to be allocated to the operator A in the 0th processing period, and the instruction sending time can be determined as the -1th processing period. Here, the ith processing period is relative to the four operators A-D.

[0213] In the 20th processing cycle, 8 operation units need to be allocated to operator A and 8 operation units need to be allocated to operator C. Then, the instruction sending time can be determined as the 19th processing cycle.

[0214] In another possible implementation, the operator start time may be used as the operator sending time, that is, when the operator start time arrives, the scheduling information is written into the preset bit of the operator start instruction to generate the operator start instruction.

[0215] After generating the operator start instruction, the instruction generator sends the operator start instruction to the operator execution device.

[0216] The operator start instruction includes, for example, a first field for carrying the scheduling information, and at least one of the following fields:

[0217] A second field for carrying size information of the data to be processed corresponding to the operator to be started, a third field for carrying size information corresponding to a task block, a fourth field for carrying the required memory space size, and a fifth field for carrying the code address corresponding to the operator to be started.

[0218] See also Figure 5 As shown, a structural example of an operator start instruction is provided; wherein the operator start instruction includes: an instruction header and an instruction body; wherein the header includes: four fields, which are used to store the following data in sequence: command type (cmd_type), command subtype (cmd_sub_type), command body length (Body length), and cyclic redundancy check code (Cyclic Redundancy Check, CRC).

[0219] The command body includes multiple fields, such as the first field and at least one of the second to fifth fields.

[0220] The second field stores the three-dimensional dimensions of the operator's processed data (GridDim_X, GridDim_Y, and GridDim_Z). The third field stores the three-dimensional dimensions of a task block corresponding to the operator (BlockDim_X, BlockDim_Y, BlockDim_Z), along with a reserved field (RZ). The fourth field stores the required memory (Share_memory_size). The fifth field stores the operator's code addresses (Kernel_Addr_lo and Kernel_Addr_hi). The operator's code address can be used to retrieve the operator's execution code, which is used by the computing device to execute the data processing task corresponding to the operator.

[0221] The first field can include a plurality of sub-fields for storing the following information respectively:

[0222] Sub-field a1: instruction mode Mode, wherein:

[0223] Mode = 0, indicating that no operation unit is specified;

[0224] Mode = 1, indicating that the number of operation units is specified, and the operator execution device allocates a corresponding number of operation units for the operator start instruction according to the state of each operation unit.

[0225] Mode = 2, indicating that a specific operation unit is specified.

[0226] Mode = 3, indicating that it is reserved.

[0227] Sub-field a2: required operation unit PE VALID, occupying 32 bits, wherein,

[0228] Mode = 1, indicating the number of operation units; for example, 8 PEs are required, and PE_valid = 8;

[0229] Mode = 2, indicating that each bit represents a corresponding operation unit. For example, operation units 0-7 are required, and PE_valid = 0x00FF; operation units 8-15 are required, and PE_valid = 0xFF00.

[0230] Sub-field a3: start task block identifier Block_start_id;

[0231] Sub-field a4: task block number block_num;

[0232] Wherein: Block_start_id and block_num indicate: from which task block (Block_start_id) to start and how many task blocks (block_num) to execute each time the operator starts.

[0233] If block_num = 0, it indicates that all task blocks in the operator are executed.

[0234] Taking the example in Table 4 above as an example:

[0235] Operator start time t0: the 0th processing period, 16 operation units are to be allocated to operator A, and 20 processing periods are to be executed, and the execution time of one task block of operator A is 20 processing periods, so the number of task blocks to be executed in the to-be-started operator A is determined to be 16, and the start task block identifier is A-1.

[0236] The values of the four subfields a1-a4 in the L0-A of the operator start instruction corresponding to the operator start time t0 are respectively:

[0237] a1: Mode = 1;

[0238] a2: PE_valid = 16;

[0239] a3: Block_start_id = A-1; wherein A-1 represents the first task block in the operator A;

[0240] a4: block_num = 16.

[0241] The operator start time t1: the 20th processing period, 8 operation units are allocated to the operator A, and 8 operation units are allocated to the operator C, and both execute 120 processing periods, the execution time required for executing one task block of the operator A is 20 processing periods, and the number of the task blocks to be executed in the operator A can be determined as 48, since 16 task blocks in the operator A are instructed to be executed at the operator start time t0, the starting task block identifier of the task blocks to be executed in the operator A is A-17; the execution time required for executing one task block of the operator C is 20 processing periods, and the number of the task blocks to be executed in the operator C can be determined as 48, and the starting task block identifier is C-1, and at this time, the operator start instructions corresponding to the operators A and C can be generated.

[0242] The values of the four subfields a1-a4 in the L1-A of the operator start instruction corresponding to the operator start time t1 of the operator A are respectively:

[0243] a1: Mode = 1;

[0244] a2: PE_valid = 8;

[0245] a3: Block_start_id = A-17; wherein A-17 represents the 17th task block in the operator A;

[0246] a4: block_num = 48.

[0247] The values of the four subfields a1-a4 in the L1-C of the operator start instruction corresponding to the operator start time t1 of the operator C are respectively:

[0248] a1: Mode = 1;

[0249] a2: PE_valid = 8;

[0250] a3: Block_start_id = C-1; wherein C-1 represents the first task block in the operator C;

[0251] a4: block_num = 48.

[0252] Operator start time t2: 140th processing cycle; 4 operation units are to be allocated to operator A, and 12 operation units are to be allocated to operator C, and both perform 20 processing cycles; the execution time required for performing one task block of operator A is 20 processing cycles, so it can be determined that the number of task blocks to be executed in operator A is 4, since 48 task blocks in operator A are indicated to be executed at operator start time t1, thus the actual task block identifier of the task blocks to be executed in operator A is A-65; the execution time required for performing one task block of operator C is 20 processing cycles, and 48 task blocks in operator C have been indicated to be executed at operator start time t2, so the starting task block identifier of operator C at operator start time t3 is C-49, and the number of task blocks to be executed is 12. Then the operator start instructions corresponding to operator A and operator C can be generated.

[0253] Among them, in the operator start instruction L2-A corresponding to the operator start time t2 of operator B, the values of the four subfields a1-a4 are respectively:

[0254] a1: Mode = 1;

[0255] a2: PE_valid = 4;

[0256] a3: Block_start_id = A-65; where A-65 represents the 65th task block in operator A;

[0257] a4: block_num = 4.

[0258] In the operator start instruction L2-C corresponding to the operator start time t2 of operator C, the values of the four subfields a1-a4 are respectively:

[0259] a1: Mode = 1;

[0260] a2: PE_valid = 12;

[0261] a3: Block_start_id = C-49; where C-49 represents the 49th task block in operator C;

[0262] a4: block_num = 12.

[0263] Operator start time t3: the 160th processing period; 8 operation units are to be allocated to operator B, and 8 operation units are to be allocated to operator D, and both perform 80 processing periods; the execution time required to perform one task block of operator B and D is 20 processing periods, the starting task block identifier of the task block to be performed is B-1, because operator B only includes 8 task blocks, therefore the number of task blocks to be performed is all the task blocks; the number of task blocks to be performed in operator D is 32, and the real task block identifier of the task block to be performed is D-1. Operator start instructions corresponding to operator B and operator D can be generated respectively,

[0264] In the operator start instruction L3-B corresponding to operator B at the operator start time t3, the values of the four subfields a1-a4 are respectively:

[0265] a1: Mode = 1;

[0266] a2: PE_valid = 8;

[0267] a3: Block_start_id = B-1;

[0268] a4: block_num = 0.

[0269] In the operator start instruction L3-D corresponding to operator D at the operator start time t3, the values of the four subfields a1-a4 are respectively:

[0270] a1: Mode = 1;

[0271] a2: PE_valid = 8;

[0272] a3: Block_start_id = D-1;

[0273] a4: block_num = 32.

[0274] Operator start time t4: the 240th processing period; 16 operation units are to be allocated to operator D, and perform 20 processing periods; the execution time required to perform one task block of operator D is 20 processing periods, the starting task block identifier of the task block to be performed is D-33, and the number of task blocks to be performed is 16; operator start instructions corresponding to operator D can be generated,

[0275] In the operator start instruction L4-D corresponding to operator D at the operator start time t4, the values of the four subfields a1-a4 are respectively:

[0276] a1: Mode = 1;

[0277] a2: PE_valid = 16;

[0278] a3: Block_start_id = D33-1;

[0279] a4: block_num = 16.

[0280] Through the above process, the host 10 generates an operator start instruction corresponding to each operator start time, and sends the operator start instruction corresponding to the operator start time to the operator execution device 20 after any operator start time arrives.

[0281] In this way, by adding a field corresponding to the scheduling information in the operator start instruction, the customer can flexibly schedule the operator; for example, for the above operator A, the processing of the operator A can be realized by three times of scheduling, and the specific scheduling process is as shown in the following table 5:

[0282] Table 5

[0283] block PE number Mode PE_valid block_start_id block_num 1 16 16 1 16 0 16 2 48 8 1 8 16 48 3 4 4 1 4 64 4

[0284] In addition, when training a deep learning model by using a deep learning framework, there are both calculation tasks of forward and backward propagation and communication tasks, and in the prior art, the calculation and communication will occupy the calculation resources of each other, and the embodiments of the present application can assign different calculation units to different operators, thereby solving the calculation resource occupation conflict. For example, the operator A is a calculation task, and occupies the calculation units 0-7; the operator B is a communication task, and occupies the calculation units 8-15, and the scheduling process can be as shown in the following table 6:

[0285] Table 6

[0286] block PE number Mode PE_valid block_start_id block_num A 16 0～7 2 0x00FF 0 16 or 0 B 48 8～15 2 0xFF00 0 48 or 0

[0287] As can be seen from the above examples, by scheduling by the operator scheduling device provided in the embodiments of the present application, the execution time of the operator can be effectively reduced, and the utilization rate of the computing power in the operator execution device can be improved.

[0288] After the operator scheduling device sends the operator start instruction to the operator execution device, the operator scheduler in the operator execution device issues the operator start instruction to the execution unit capable of executing the operator start instruction. After the task block scheduler in the execution unit receives the operator start instruction issued by the operator scheduler, the task block scheduler decomposes the operator start instruction into a plurality of to-be-executed task blocks (including at least part of the plurality of task blocks corresponding to the operator) according to the number of to-be-executed task blocks carried in the operator start instruction and the task block identifier of the starting task block in the to-be-executed task blocks, and issues the to-be-executed task blocks to the calculation unit. After the calculation unit receives the to-be-executed task blocks, the calculation unit executes the received task blocks.

[0289] For example, taking Figure 5The structure of the operator start instruction shown is an example. The field a1: Mode = 0 indicates that no operation unit is specified; the subfield a3 writes the starting task block identifier Block_start_id = "13" (in the operator start instruction, represented in binary, and represented in decimal here for the sake of writing convenience); and the subfield a4 writes the number of task blocks block_num = 10, so that the operator start instruction is decomposed into 10 task blocks, and the identifiers of the 10 task blocks are "13" to "22", respectively.

[0290] The operator scheduling device provided in the embodiments of the present disclosure includes a scheduling policy generator and an instruction generator. The scheduling policy generator can generate a scheduling policy for scheduling operation units in an operator execution device when the operator execution device executes a deep learning model, and transmit the scheduling policy to the instruction generator. The instruction generator can generate an operator start instruction based on the scheduling policy, and send the operator start instruction to the operator execution device, so that the operator execution device executes a corresponding operator based on the operator start instruction sent by the operator scheduling device. Thus, the specific process of the operator execution device executing the operator can be planned in advance, the operator can be controlled to be issued, and the utilization rate of the computing resources in the operator execution device can be improved.

[0291] Those skilled in the art can understand that, in the above method of the specific implementation, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0292] Based on the same inventive concept, the embodiments of the present disclosure also provide an operator execution method corresponding to the operator execution device. Since the principle of the device in the embodiments of the present disclosure solves the problem similarly to the above-mentioned operator execution method, the implementation of the operator execution method can be referred to the implementation of the method, and the repeated parts will not be described herein.

[0293] Referring to Figure 5 As shown, a flowchart of an operator execution method provided in the embodiments of the present disclosure includes:

[0294] S501: In response to receiving an operator start instruction of a to-be-started operator, the operator scheduler distributes the operator start instruction to an execution unit corresponding to operator type information carried in the operator start instruction based on the operator type information; the operator start instruction includes scheduling information of an operation unit in the execution unit;

[0295] S502: In response to receiving the operator start instruction distributed by the operator scheduler, the execution unit executes the to-be-started operator based on the scheduling information carried in the operator start instruction.

[0296] In a possible implementation, the scheduling information comprises at least one of the following:

[0297] the identifier of the to-be-started operator, the number of to-be-executed task blocks in the to-be-started operator, the task block identifier of a starting task block in the to-be-executed task blocks, the number of operation units for executing the to-be-executed task blocks, and the operation unit identifier for executing the to-be-executed task blocks;

[0298] Each task block comprises a plurality of sub-tasks in the to-be-started operator.

[0299] In a possible implementation, the execution unit comprises a task block scheduler and a plurality of operation units. In response to receiving the operator start instruction issued by the operator scheduler, the execution unit executes the to-be-started operator based on the scheduling information carried in the operator start instruction, comprising:

[0300] In response to receiving the operator start instruction issued by the operator scheduler, the task block scheduler determines, based on the scheduling information carried in the operator start instruction, a target operation unit for executing the to-be-started operator and to-be-executed task blocks from a plurality of operation units, issues the to-be-executed task blocks to the target operation unit, and each to-be-executed task block comprises a plurality of sub-tasks in the to-be-started operator.

[0301] In response to receiving the to-be-executed task blocks issued by the task block scheduler, the operation unit executes data processing tasks corresponding to the to-be-executed task blocks.

[0302] In a possible implementation, the scheduling information comprises the number of to-be-executed task blocks in the to-be-started operator and the task block identifier of a starting task block in the to-be-executed task blocks.

[0303] The task block scheduler determines, based on the scheduling information carried in the operator start instruction, a plurality of to-be-executed task blocks, comprising: determining, based on the task block identifier of the starting task block and taking the starting task block as the start, a plurality of task blocks corresponding to the number of to-be-executed task blocks as the to-be-executed task blocks.

[0304] In a possible implementation, the task block scheduler issues the to-be-executed task blocks to the target operation unit, comprising: determining, based on the number of to-be-executed task blocks and the number of target operation units, to-be-executed task blocks to be issued to each target operation unit from the to-be-executed task blocks.

[0305] Each target operation unit is issued the to-be-executed task blocks determined for the target operation unit.

[0306] In a possible implementation, the scheduling information includes: a number of operation units for executing the to-be-executed task block, or an identifier of an operation unit for executing the to-be-executed task block.

[0307] The task block scheduler determines, based on the scheduling information carried in the operator start instruction, a target operation unit for executing the to-be-started operator from a plurality of operation units.

[0308] The target operation unit for executing the to-be-started operator is determined from a plurality of operation units based on a number of operation units for executing the to-be-executed task block, or an identifier of an operation unit for executing the to-be-executed task block.

[0309] In a possible implementation, the operator start instruction includes: a first field for carrying the scheduling information, and at least one of the following fields:

[0310] a second field for carrying size information of to-be-processed data corresponding to the to-be-started operator, a third field for carrying size information of one task block, a fourth field for carrying a required memory space size, and a fifth field for carrying a code address corresponding to the to-be-started operator.

[0311] Based on the same inventive concept, the present disclosure also provides an operator scheduling method corresponding to the operator scheduling device; since the principle of the device in the present disclosure solves the problem, which is similar to the above-mentioned operator scheduling method of the present disclosure, the implementation of the operator scheduling method can be referred to the implementation of the method, and the repeated parts will not be described here.

[0312] Referring to Figure 6 FIG. 1 shows a flowchart of an operator scheduling method provided by an embodiment of the present disclosure, which includes:

[0313] S601: A scheduling policy generator generates a scheduling policy for scheduling operation units in an operator execution device when the operator execution device executes a plurality of operators of a deep learning model, and transmits the scheduling policy to an instruction generator;

[0314] S602: The instruction generator generates an operator start instruction based on the scheduling policy, and sends the operator start instruction to the operator execution device.

[0315] In a possible implementation, the scheduling policy includes: operator start moments, and scheduling information corresponding to each operator start moment.

[0316] The scheduling information comprises at least one of the following: an identifier of the to-be-started operator, a number of to-be-executed task blocks in the to-be-started operator, a task block identifier of a starting task block in the to-be-executed task blocks, a number of operation units for executing the to-be-executed task blocks, and an operation unit identifier for executing the to-be-executed task blocks.

[0317] Each task block comprises a plurality of sub-tasks in the to-be-started operator.

[0318] In a possible implementation, the scheduling strategy generator generates a scheduling strategy for operation units in the operator execution device when the operator execution device executes a plurality of operators of a deep learning model, comprising:

[0319] The deep learning model is parsed to obtain a plurality of operators in the deep learning model;

[0320] Based on operation information corresponding to each of the plurality of operators and computing power information of the operator execution device, a scheduling strategy for operation units in the operator execution device when executing the plurality of operators is determined.

[0321] Each task block comprises a plurality of sub-tasks in the corresponding operator; and the operator execution device executes the plurality of operators based on the scheduling strategy.

[0322] In a possible implementation, the operation information comprises execution time of task blocks corresponding to each of the plurality of operators and parallelism between the plurality of operators.

[0323] In a possible implementation, the scheduling strategy generator obtains the execution time of the task blocks corresponding to each of the plurality of operators in the following manner:

[0324] Based on memory required by the operator when executing and computing power information of the operator execution device, execution time required for executing each task block of the operator is estimated.

[0325] In a possible implementation, the scheduling strategy generator estimates the execution time required for executing each task block of the operator based on memory required by the operator when executing and computing power information of the operator execution device, comprising:

[0326] Based on memory required by the operator when executing and memory access bandwidth of the operation unit to the memory, memory access time is determined; and

[0327] Based on computing power information required by each computing step in each sub-task in the task block and computing power information of each operation unit, computing time of the task block is determined.

[0328] determine an execution duration required for executing the task block based on the memory access duration and the calculation duration.

[0329] In a possible implementation, the scheduling strategy generator obtains the execution duration of the task block corresponding to each of the plurality of operators in the following manner:

[0330] determine a simulation model corresponding to the operator based on the operator;

[0331] run the simulation model, and determine the execution duration required for each task block of the operator based on a running duration of the simulation model.

[0332] In a possible implementation, the scheduling strategy generator determines the execution duration required for each task block of the operator based on the running duration of the simulation model, including:

[0333] determine the number of task blocks obtained by dividing the operator in the simulation process according to the size of the operation unit in the operator execution device running the simulation model and the amount of data required for processing by the operator during execution;

[0334] determine the batch required for processing according to the number of task blocks obtained by dividing the operator in the simulation process and the number of operation units in the operator execution device running the simulation model;

[0335] determine the execution duration of the task block corresponding to each of the plurality of operators based on the batch and the running duration of the simulation model.

[0336] In a possible implementation, the scheduling strategy generator determines the scheduling strategy of the operation unit in the operator execution device when executing the plurality of operators based on the operation information corresponding to each of the plurality of operators and the computing power information of the operator execution device, including:

[0337] construct an association relationship between the execution duration of the task block corresponding to each of the plurality of operators, the number of task blocks corresponding to each of the plurality of operators, the parallelism between the plurality of operators, the computing power information of the operator execution device, the strategy scheduling parameter, and the total execution duration of the plurality of operators;

[0338] adjust the strategy scheduling parameter based on the association relationship to obtain a target scheduling strategy, with the objective of reducing the total execution duration.

[0339] In a possible implementation, the scheduling strategy generator constructs an association relationship between the total execution time of the plurality of operators and the execution time length of the task block corresponding to each of the plurality of operators, the number of task blocks corresponding to each of the plurality of operators, the parallelism between the plurality of operators, the computing power information of the operator execution device, and the policy scheduling parameter, including:

[0340] The relationship equation is constructed with the execution time length of the task block corresponding to each of the plurality of operators, the number of task blocks corresponding to each of the plurality of operators, and the computing power information of the operator execution device as parameters, with the policy scheduling parameter as the independent variable, with the total execution time of the plurality of operators as the dependent variable, and with the parallelism between the operators as the constraint condition. The relationship equation is taken as the association relationship.

[0341] In a possible implementation, the instruction generator generates the operator start instruction based on the scheduling strategy, and sends the operator start instruction to the operator execution device, including:

[0342] The operator start instruction is generated based on the scheduling information.

[0343] In response to the instruction sending time corresponding to any operator start instruction being reached, the any operator start instruction is sent to the operator execution device. The instruction sending time is determined based on the instruction execution time corresponding to the any operator start instruction.

[0344] In a possible implementation, the operator start instruction includes: a first field for carrying the scheduling information, and at least one of the following at least one field:

[0345] A second field for carrying size information of to-be-processed data corresponding to the to-be-started operator, a third field for carrying size information of one task block, a fourth field for carrying a required memory space size, and a fifth field for carrying a code address corresponding to the to-be-started operator.

[0346] The embodiments of the present disclosure further provide a chip, including: the operator execution device according to any one of the embodiments of the present disclosure, and / or the operator scheduling device according to any one of the embodiments of the present disclosure.

[0347] The embodiments of the present disclosure further provide a computer device, and a structural schematic diagram of the computer device provided by the embodiments of the present disclosure, including: the chip according to the embodiments of the present disclosure.

[0348] The embodiment of the present disclosure further provides a computer readable storage medium, and the computer readable storage medium stores a computer program. When the computer program is run by a processor, steps of the scheduling method described in the above method embodiment are executed. The storage medium can be a volatile or non-volatile computer readable storage medium.

[0349] The embodiment of the present disclosure further provides a computer program product, and the computer program product carries a program code. The program code includes instructions for executing steps of the scheduling method described in the above method embodiment. For details, refer to the above method embodiment, which will not be described here.

[0350] The computer program product can be implemented by hardware, software or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium. In another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK) and the like.

[0351] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system and device can refer to the corresponding process in the above method embodiment, which will not be described here. In several embodiments provided by the present disclosure, it should be understood that the disclosed system, device and method can be implemented by other ways. The above-described device embodiments are only schematic, for example, the division of the units is only a logical function division, and there can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some communication interfaces, devices or units, and can be electrical, mechanical or other forms.

[0352] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. According to actual needs, some or all of the units can be selected to achieve the purpose of the embodiment.

[0353] In addition, each functional unit in each embodiment of the present disclosure can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit.

[0354] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a nonvolatile computer readable storage medium executable by a processor. Based on this understanding, the technical solutions of the present disclosure essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present disclosure. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0355] Finally, it should be noted that: the above-described embodiments are only specific embodiments of the present disclosure, used to illustrate the technical solutions of the present disclosure, and not to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art who is familiar with the technology in the art can still make modifications or easily think of changes to the technical solutions described in the foregoing embodiments, or make equivalent replacements to some of the technical features; and these modifications, changes or replacements do not cause the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be covered within the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. An operator execution device, characterized by, Comprise: An operator scheduler and an execution unit; The operator scheduler is configured to, in response to receiving an operator starting instruction of an operator to be started, distribute the operator starting instruction to an execution unit corresponding to operator type information carried in the operator starting instruction based on the operator type information; the operator starting instruction comprises scheduling information of an operation unit in the execution unit; The execution unit is configured to, in response to receiving the operator starting instruction distributed by the operator scheduler, execute the operator to be started based on scheduling information carried in the operator starting instruction; The scheduling information comprises at least one of the following: The identifier of the operator to be started, the number of task blocks to be executed in the operator to be started, the task block identifier of the starting task block in the task block to be executed, the number of operation units for executing the task block to be executed, and the operation unit identifier for executing the task block to be executed; Each task block comprises a plurality of sub-tasks in the operator to be started; The execution unit comprises a task block scheduler and a plurality of operation units; The task block scheduler is configured to, in response to receiving the operator starting instruction distributed by the operator scheduler, determine a target operation unit for executing the operator to be started and determine a task block to be executed from a plurality of operation units based on the scheduling information carried in the operator starting instruction; distribute the task block to be executed to the target operation unit; each of the task blocks to be executed comprises a plurality of sub-tasks in the operator to be started; The operation unit is configured to, in response to receiving the task block to be executed distributed by the task block scheduler, execute a data processing task corresponding to the task block to be executed.

2. The operator execution device according to claim 1, characterized in that, The scheduling information comprises the number of task blocks to be executed in the operator to be started and the task block identifier of the starting task block in the task block to be executed; When determining a plurality of task blocks to be executed based on the scheduling information carried in the operator starting instruction, the task block scheduler is configured to determine a plurality of task blocks corresponding to the number of task blocks to be executed as the task blocks to be executed based on the task block identifier of the starting task block and taking the starting task block as the starting point.

3. The operator execution device according to claim 2, characterized in that, When distributing the task blocks to be executed to the target operation units, the task block scheduler is configured to determine the task blocks to be executed distributed to each of the target operation units from the task blocks to be executed based on the number of task blocks to be executed and the number of target operation units; Each of the target operation units is distributed with the task blocks to be executed determined for each target operation unit.

4. The operator execution device according to claim 2 or 3, characterized in that, The scheduling information comprises the number of operation units for executing the task block to be executed or the operation unit identifier for executing the task block to be executed; When determining a target operation unit for executing the operator to be started from a plurality of operation units based on the scheduling information carried in the operator starting instruction, the task block scheduler is configured to: The target operation unit is determined from the plurality of operation units based on an operation unit quantity of the task block to be executed or an operation unit identifier of the task block to be executed.

5. The operator execution device according to any one of claims 1 to 3, characterized in that, The operator starting instruction comprises: a first field for carrying the scheduling information, and at least one of the following fields: A second field for carrying size information of the to-be-processed data corresponding to the to-be-started operator, a third field for carrying size information corresponding to a task block, a fourth field for carrying a required memory space size, and a fifth field for carrying a code address corresponding to the to-be-started operator.

6. An operator dispatching device characterized by Comprise: A scheduling policy generator and an instruction generator; The scheduling policy generator is configured to generate a scheduling policy for scheduling operation units in the operator execution device when the operator execution device executes operators of a deep learning model, and transmit the scheduling policy to the instruction generator; The instruction generator is configured to generate an operator starting instruction based on the scheduling policy, and send the operator starting instruction to the operator execution device; The scheduling policy comprises: operator starting time and scheduling information corresponding to each operator starting time; The scheduling information comprises at least one of the following: an identifier of a to-be-started operator, a quantity of task blocks to be executed in the to-be-started operator, a task block identifier of a starting task block in the to-be-executed task block, an operation unit quantity of the to-be-executed task block, and an operation unit identifier of the to-be-executed task block; Each task block comprises a plurality of sub-tasks in the to-be-started operator. The scheduling policy generator is configured to generate a scheduling policy for scheduling operation units in the operator execution device when the operator execution device executes multiple operators of a deep learning model, and perform the following operations: Parse the deep learning model to obtain a plurality of operators in the deep learning model; Determine a scheduling policy for scheduling operation units in the operator execution device when the plurality of operators are executed based on operation information corresponding to each of the plurality of operators and computing power information of the operator execution device. Each task block comprises a plurality of sub-tasks in the corresponding operator; and the operator execution device executes the plurality of operators based on the scheduling policy.

7. The operator dispatching device of claim 6, wherein, The operation information comprises: operation information corresponding to each of the plurality of operators, which comprises execution durations of task blocks corresponding to the plurality of operators and parallelism degrees between the plurality of operators.

8. The operator dispatching device according to claim 6 or 7, characterized in that The scheduling policy generator is further configured to obtain the execution durations of the task blocks corresponding to the plurality of operators in the following manner: Estimate execution durations required for executing each task block of the operator based on memory required by the operator when executing and computing power information of the operator execution device.

9. The operator dispatching device of claim 8, wherein, When estimating the execution durations required for executing each task block of the operator based on memory required by the operator when executing and computing power information of the operator execution device, the scheduling policy generator is configured to: Determine a memory access duration based on the memory required by the operator when executing and a memory access bandwidth of the operation unit to the memory; and ​ determine a computation duration of the task block based on the computing power information required by each computing step in each subtask in the task block and the computing power information of each operation unit; determine an execution duration required for executing the task block based on the memory access duration and the computation duration.

10. The operator dispatching device according to claim 6 or 7, characterized in that, The scheduling strategy generator is configured to obtain the execution duration of the task block corresponding to each of the plurality of operators in the following manner: determine a simulation model corresponding to the operator based on the operator; determine the execution duration required for each task block of the operator based on the running duration of the simulation model.

11. The operator dispatching device of claim 10, wherein, When determining the execution duration required for each task block of the operator based on the running duration of the simulation model, the scheduling strategy generator is configured to: determine the number of task blocks obtained by dividing the operator in the simulation process according to the size of the operation unit in the operator execution device running the simulation model and the amount of data required for processing by the operator during execution; determine the batch to be processed according to the number of task blocks obtained by dividing the operator in the simulation process and the number of operation units in the operator execution device running the simulation model; determine the execution duration of the task block corresponding to each of the plurality of operators based on the batch and the running duration of the simulation model.

12. The operator dispatching device according to claim 6 or 7, characterized in that, When determining the scheduling strategy for the operation units in the operator execution device during execution of the plurality of operators based on the operation information corresponding to each of the plurality of operators and the computing power information of the operator execution device, the scheduling strategy generator is configured to: build an association relationship between the execution duration of the task block corresponding to each of the plurality of operators, the number of task blocks corresponding to each of the plurality of operators, the parallelism between the plurality of operators, the computing power information of the operator execution device, the policy scheduling parameter, and the total execution duration of the plurality of operators; adjust the policy scheduling parameter based on the association relationship to obtain a target scheduling strategy, with the goal of reducing the total execution duration.

13. The operator dispatching device of claim 12, wherein, When building the association relationship between the execution duration of the task block corresponding to each of the plurality of operators, the number of task blocks corresponding to each of the plurality of operators, the parallelism between the plurality of operators, the computing power information of the operator execution device, the policy scheduling parameter, and the total execution duration of the plurality of operators, the scheduling strategy generator is configured to: build a relationship equation with the execution duration of the task block corresponding to each of the plurality of operators, the number of task blocks corresponding to each of the plurality of operators, the computing power information of the operator execution device as parameters, the policy scheduling parameter as an independent variable, the total execution duration of the plurality of operators as a dependent variable, and the parallelism between the operators as a constraint condition; use the relationship equation as the association relationship.

14. The operator dispatching device according to claim 6 or 7, characterized in that, When generating the operator start instruction based on the scheduling strategy and sending the operator start instruction to the operator execution device, the instruction generator is configured to: generate the operator start instruction based on the scheduling information; In response to arrival of an instruction sending time corresponding to any operator starting instruction, the operator starting instruction is sent to the operator execution device; wherein the instruction sending time is determined based on an instruction execution time corresponding to the any operator starting instruction.

15. The operator dispatching device of claim 14, wherein, The operator starting instruction comprises: a first field for carrying the scheduling information, and at least one of the following fields: A second field for carrying size information of the to-be-started operator corresponding to to-be-processed data, a third field for carrying size information corresponding to one task block, a fourth field for carrying a required memory space size, and a fifth field for carrying a code address corresponding to the to-be-started operator.

16. A chip, characterized by Comprise: The operator execution device according to any one of claims 1-5, and / or the operator scheduling device according to any one of claims 6-15.

17. An operator execution method characterized by, Comprise: In response to receiving an operator starting instruction of a to-be-started operator, the operator scheduler distributes the operator starting instruction to an execution unit corresponding to operator type information carried in the operator starting instruction based on the operator type information; the operator starting instruction comprises scheduling information of an operation unit in the execution unit; In response to receiving the operator starting instruction distributed by the operator scheduler, the execution unit executes the to-be-started operator based on scheduling information carried in the operator starting instruction; The scheduling information comprises at least one of: An identifier of the to-be-started operator, a number of task blocks to be executed in the to-be-started operator, a task block identifier of a starting task block in the to-be-executed task block, a number of operation units for executing the to-be-executed task block, and an operation unit identifier for executing the to-be-executed task block; Wherein each task block comprises a plurality of sub-tasks in the to-be-started operator; The execution unit comprises: a task block scheduler, and a plurality of operation units; In response to receiving the operator starting instruction distributed by the operator scheduler, the task block scheduler determines a target operation unit for executing the to-be-started operator and determines to-be-executed task blocks from the plurality of operation units based on the scheduling information carried in the operator starting instruction; the to-be-executed task blocks are distributed to the target operation unit; each of the to-be-executed task blocks comprises a plurality of sub-tasks in the to-be-started operator; In response to receiving the to-be-executed task blocks distributed by the task block scheduler, the operation unit executes a data processing task corresponding to the to-be-executed task blocks.

18. An operator dispatching method characterized by, Comprise: The scheduling strategy generator generates a scheduling strategy for scheduling operation units in the operator execution device when the operator execution device executes multiple operators of a deep learning model; And the scheduling strategy is transmitted to the instruction generator; The instruction generator generates an operator starting instruction based on the scheduling strategy, and sends the operator starting instruction to the operator execution device; The scheduling strategy comprises: operator starting times, and scheduling information corresponding to each operator starting time; The scheduling information comprises at least one of the following: an identifier of the to-be-started operator, a number of to-be-executed task blocks in the to-be-started operator, a task block identifier of a starting task block in the to-be-executed task blocks, a number of operation units for executing the to-be-executed task blocks, and operation unit identifiers for executing the to-be-executed task blocks. Each task block comprises a plurality of sub-tasks in the to-be-started operator. The scheduling strategy generator generates a scheduling strategy for scheduling operation units in the operator execution device when the operator execution device executes a plurality of operators of a deep learning model, comprising: parsing the deep learning model to obtain a plurality of operators in the deep learning model; determining, based on operation information corresponding to each of the plurality of operators and computing power information of the operator execution device, a scheduling strategy for scheduling operation units in the operator execution device when the plurality of operators are executed; Each task block comprises a plurality of sub-tasks in the corresponding operator; and the operator execution device executes the plurality of operators based on the scheduling strategy.

19. A computer device, comprising: Comprise: The chip of claim 16.

20. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and when the computer program is run by a computer device, the computer device executes the steps of the operator execution method of claim 17 or the steps of the operator scheduling method of claim 18.

Images