Scheduling system, method, device, chip, computer device and storage medium

Abstract

The disclosure provides a scheduling system, a scheduling method, a device, computer equipment and a storage medium, wherein the scheduling system comprises: the host, configured to parse a deep learning model to obtain a plurality of operators in the deep learning model; based on operation information corresponding to each of the plurality of operators and computing power information of an operator execution device, determine a scheduling strategy for operation units in the operator execution device when executing the plurality of operators; and the operator execution device, configured to execute the plurality of operators based on the scheduling of the host.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, in particular to a scheduling system, a scheduling method, an apparatus, a chip, a computer device and a storage medium. BACKGROUND

[0002] In the field of deep learning, when a computer device runs a deep learning model, a deep learning framework deployed on the computer device parses the deep learning model to obtain operators in the deep learning model, and sends the operators to an Artificial Intelligence (AI) accelerator card, a Graphics Processing Unit (GPU) or the like execution device. After receiving the operators sent by the deep learning framework, the execution device is responsible for scheduling of computing power and execution of the operators. This kind of scheduling of computing power has the problem of low utilization rate of computing power. SUMMARY

[0003] The present disclosure provides at least a scheduling system, a scheduling method, an apparatus, a chip, a computer device and a storage medium.

[0004] In a first aspect, the present disclosure provides a scheduling system, comprising: a host and an operator execution device; the host is configured to parse a deep learning model to obtain a plurality of operators in the deep learning model; based on operation information corresponding to each of the plurality of operators and computing power information of the operator execution device, determine a scheduling strategy for operation units in the operator execution device when executing the plurality of operators; and the operator execution device is configured to execute the plurality of operators based on the scheduling strategy of the host.

[0005] In this way, the host determines a scheduling strategy for operation units when executing operators based on operation information corresponding to each of the plurality of operators in the deep learning model and computing power information of the execution device, and schedules operation units to execute operators based on the scheduling strategy, so as to be able to control the delivery of operators according to the planning of the specific process of executing operators in the execution device, and improve the utilization rate of computing resources in the execution device.

[0006] In a possible implementation, the operation information corresponding to each of the plurality of operators includes execution duration of task blocks corresponding to each of the plurality of operators, and parallelism between the plurality of operators; and each task block includes a plurality of sub-tasks in the corresponding operator.

[0007] In a possible implementation, when obtaining the execution duration of the task blocks corresponding to each of the plurality of operators, the host is configured to estimate the execution duration required for executing the task blocks based on memory required by the task blocks when executing and the computing power information of the operation units.

[0008] In this way, the execution time required by each task block can be estimated with high efficiency.

[0009] In a possible implementation, when estimating the execution time required for executing the task block based on the memory required by the task block when executing and the computing power information of the operation unit, the host is configured to: determine the memory access time based on the memory required by the operator when executing and the memory access bandwidth of the operation unit to the memory; determine the computing time of the task block based on the computing power information required by each computing step in each subtask in the task block and the computing power information of each operation unit; and determine the execution time required for executing the task block based on the memory access time and the computing time.

[0010] In a possible implementation, when obtaining the execution time of the task block corresponding to each of the plurality of operators, the host is configured to: determine the simulation model corresponding to the operator based on the operator; run the simulation model, and determine the execution time required by each task block of the operator based on the running time of the simulation model.

[0011] In a possible implementation, when determining the execution time required by each task block of the operator based on the running time of the simulation model, the host is configured to: determine the number of task blocks obtained by dividing the operator in the simulation process according to the size of the operation unit in the operator execution device running the simulation model and the amount of data required by the operator when executing; determine the batch to be processed according to the number of task blocks obtained by dividing the operator in the simulation process and the number of operation units in the operator execution device running the simulation model; and determine the execution time of the task block corresponding to each of the plurality of operators based on the batch and the running time of the simulation model.

[0012] In this way, the execution time required by each task block can be obtained through simulation, and the accuracy is high.

[0013] In a possible implementation, when determining the scheduling strategy of the operation unit in the operator execution device when executing the plurality of operators based on the operation information corresponding to each of the plurality of operators and the computing power information of the operator execution device, the host is configured to: construct the association relationship between the execution time of the task block corresponding to each of the plurality of operators, the number of task blocks corresponding to each of the plurality of operators, the parallelism between the plurality of operators, the computing power information of the operator execution device, the scheduling strategy parameter, and the total execution time of the plurality of operators; and adjust the scheduling strategy parameter based on the association relationship to obtain a target scheduling strategy, with the objective of reducing the total execution time.

[0014] In a possible implementation, the host, when constructing the association between the execution total time of the plurality of operators and the execution duration of the plurality of task blocks corresponding to the plurality of operators respectively, the number of the plurality of task blocks corresponding to the plurality of operators respectively, the parallelism between the plurality of operators, the computing power information of the operator execution device, the scheduling strategy parameter, is configured to: construct a relational equation with the execution duration of the plurality of task blocks corresponding to the plurality of operators respectively, the number of the plurality of task blocks corresponding to the plurality of operators respectively, and the computing power information of the operator execution device as parameters, the scheduling strategy parameter as an independent variable, the execution total time of the plurality of operators as a dependent variable, and the parallelism between the operators as a constraint condition; and use the relational equation as the association.

[0015] In this way, the execution total time can be continuously optimized, and the rationality of the scheduling strategy can be improved.

[0016] In a possible implementation, the scheduling strategy parameter includes at least one of the following: at least one execution time corresponding to the plurality of operators respectively, a task block identifier of a starting task block corresponding to each execution time in the at least one execution time, a required number of operation units, and an identifier of an operation unit that executes the operator.

[0017] In this way, the scheduling of the operation units can be implemented through each item in the scheduling strategy, so that the scheduling system can have a clear perception of the working conditions of the operation units.

[0018] In a possible implementation, the host, when scheduling the operation units to execute the plurality of operators based on the scheduling strategy, is configured to: generate an operator start instruction based on the scheduling strategy; and send the operator start instruction to the operator execution device.

[0019] The operator execution device, when executing the plurality of operators based on the scheduling of the host, is configured to: in response to receiving the operator start instruction sent by the host, execute an operator corresponding to the operator start instruction in a manner indicated by the operator start instruction.

[0020] In a possible implementation, the scheduling strategy parameter includes an operator execution time corresponding to the plurality of operators respectively; the operator start instruction includes an operator start instruction corresponding to the plurality of operators respectively; and the host, when sending the operator start instruction to the operator execution device, is configured to: in response to an instruction sending time corresponding to any operator start instruction being reached, send the any operator start instruction to the operator execution device; wherein the instruction sending time is determined based on an instruction execution time corresponding to the any operator start instruction.

[0021] In a second aspect, the embodiments of the present disclosure provide a scheduling method, including: parsing a deep learning model to obtain a plurality of operators in the deep learning model; determining a scheduling strategy for operation units in an operator execution device when executing the plurality of operators based on operation information corresponding to each of the plurality of operators and computing power information of the operator execution device; wherein each task block includes a plurality of sub-tasks in a corresponding operator; and scheduling the operation units to execute the plurality of operators based on the scheduling strategy.

[0022] In an optional implementation, the operation information corresponding to each of the plurality of operators includes execution time of a task block corresponding to each of the plurality of operators and parallelism between the plurality of operators; and each task block includes a plurality of sub-tasks in a corresponding operator.

[0023] In an optional implementation, the execution time of the task block corresponding to each of the plurality of operators is obtained in the following manner: based on memory required by the task block when executing and computing power information of the operation unit, the execution time required for executing the task block is estimated.

[0024] In this way, the execution time required for each task block can be estimated, and the estimation efficiency is high.

[0025] In an optional implementation, the execution time required for executing the task block is estimated based on the memory required by the operator when executing and the memory access bandwidth of the operation unit to the memory, including: determining a memory access time based on the memory required by the operator when executing and the memory access bandwidth of the operation unit to the memory; determining a calculation time of the task block based on computing power information required by each calculation step in each sub-task in the task block and computing power information of each operation unit; and determining the execution time required for executing the task block based on the memory access time and the calculation time.

[0026] In an optional implementation, the execution time of the task block corresponding to each of the plurality of operators is obtained in the following manner: based on the operator, a simulation model corresponding to the operator is determined; the simulation model is run, and the execution time required for each task block of the operator is determined based on a running time of the simulation model.

[0027] In this way, the execution time required for each task block can be obtained through simulation, and the accuracy is high.

[0028] In an optional implementation, the determining of the execution time length of each task block of the operator based on the running time length of the simulation model comprises: determining the number of task blocks obtained by dividing the operator in the simulation process according to the size of the operation unit in the operator execution device running the simulation model and the amount of data required by the operator to process when executing; determining the batch to be processed according to the number of task blocks obtained by dividing the operator in the simulation process and the number of operation units in the operator execution device running the simulation model; and determining the execution time length of the task block corresponding to each of the plurality of operators based on the batch and the running time length of the simulation model.

[0029] In an optional implementation, the determining of the scheduling strategy of the operation unit in the operator execution device when executing the plurality of operators based on the operation information corresponding to each of the plurality of operators and the computing power information of the operator execution device comprises: constructing an association relationship between the execution time length of the task block corresponding to each of the plurality of operators, the number of task blocks corresponding to each of the plurality of operators, the parallelism between the plurality of operators, the computing power information of the operator execution device, the scheduling strategy parameter, and the total execution time length of the plurality of operators; and adjusting the scheduling strategy parameter based on the association relationship to obtain a target scheduling strategy, with the objective of reducing the total execution time length.

[0030] In a possible implementation, the constructing of the association relationship between the execution time length of the task block corresponding to each of the plurality of operators, the number of task blocks corresponding to each of the plurality of operators, the parallelism between the plurality of operators, the computing power information of the operator execution device, the scheduling strategy parameter, and the total execution time length of the plurality of operators comprises: constructing a relationship equation with the execution time length of the task block corresponding to each of the plurality of operators, the number of task blocks corresponding to each of the plurality of operators, and the computing power information of the operator execution device as parameters, the scheduling strategy parameter as an independent variable, the total execution time length of the plurality of operators as a dependent variable, and the parallelism between the operators as a constraint condition; and taking the relationship equation as the association relationship.

[0031] In an optional implementation, the scheduling strategy parameter comprises at least one of the following: at least one execution time corresponding to each of the plurality of operators, a task block identifier of a starting task block corresponding to each of the at least one execution time, a required number of operation units, and an identifier of an operation unit executing the operator.

[0032] In an alternative implementation, the scheduling the operation units to execute the plurality of operators based on the scheduling strategy comprises: generating an operator starting instruction based on the scheduling strategy; sending the operator starting instruction to the operator execution device; and the operator starting instruction is used by the operator execution device to execute an operator corresponding to the operator starting instruction in a manner indicated by the operator starting instruction.

[0033] In an alternative implementation, the scheduling strategy parameters comprise operator execution time points corresponding to the plurality of operators respectively; the operator starting instruction comprises operator starting instructions corresponding to the plurality of operators respectively; and the sending of the operator starting instruction to the operator execution device comprises: sending any operator starting instruction to the operator execution device in response to an instruction sending time point corresponding to the operator starting instruction being reached; and the instruction sending time point is determined based on an instruction execution time point corresponding to the operator starting instruction.

[0034] In a third aspect, the embodiments of the present disclosure further provide a scheduling device, and the device comprises:

[0035] The parsing module is configured to parse the deep learning model to obtain a plurality of operators in the deep learning model.

[0036] The generating module is configured to determine a scheduling strategy for operation units in the operator execution device when executing the plurality of operators based on operation information corresponding to the plurality of operators respectively and computing power information of the operator execution device; and each task block comprises a plurality of sub-tasks in a corresponding operator.

[0037] In a possible implementation, the operation information corresponding to the plurality of operators respectively comprises execution time of task blocks corresponding to the plurality of operators respectively and parallelism between the plurality of operators; and each task block comprises a plurality of sub-tasks in a corresponding operator.

[0038] In a possible implementation, when the execution time of the task blocks corresponding to the plurality of operators respectively is obtained, the obtaining module is configured to estimate the execution time required for executing the task blocks based on memory required by the task blocks when being executed and computing power information of the operation units.

[0039] In a possible implementation, the obtaining module, when estimating the execution duration required for executing the task block based on the memory required by the task block during execution and the computing power information of the operation unit, is configured to: determine a memory access duration based on the memory required by the operator during execution and the memory access bandwidth of the operation unit to the memory; determine a computing duration of the task block based on the computing power information required by each computing step in each subtask in the task block and the computing power information of each operation unit; and determine the execution duration required for executing the task block based on the memory access duration and the computing duration.

[0040] In a possible implementation, the obtaining module, when obtaining the execution duration of the task block corresponding to each of the plurality of operators, is configured to: determine a simulation model corresponding to the operator based on the operator; run the simulation model, and determine the execution duration required by each task block of the operator based on the running duration of the simulation model.

[0041] In a possible implementation, the obtaining module, when determining the execution duration required by each task block of the operator based on the running duration of the simulation model, is configured to: determine the number of task blocks obtained by dividing the operator in the simulation process according to the size of the operation unit in the operator execution device running the simulation model and the amount of data required by the operator during execution; determine the batch required for processing according to the number of task blocks obtained by dividing the operator in the simulation process and the number of operation units in the operator execution device running the simulation model; and determine the execution duration of the task block corresponding to each of the plurality of operators based on the batch and the running duration of the simulation model.

[0042] In a possible implementation, the generating module, when determining the scheduling strategy for the operation unit in the operator execution device during execution of the plurality of operators based on the operation information corresponding to each of the plurality of operators and the computing power information of the operator execution device, is configured to: construct an association relationship between the execution duration of the task block corresponding to each of the plurality of operators, the number of task blocks corresponding to each of the plurality of operators, the parallelism between the plurality of operators, the computing power information of the operator execution device, the scheduling strategy parameter, and the total execution duration of the plurality of operators; and adjust the scheduling strategy parameter based on the association relationship to obtain a target scheduling strategy, so as to reduce the total execution duration.

[0043] In a possible implementation, the generation module, when constructing the association between the execution duration of each of the plurality of operators, the number of task blocks corresponding to each of the plurality of operators, the parallelism between the plurality of operators, the computing power information of the operator execution device, the scheduling strategy parameter, and the total execution duration of the plurality of operators, is configured to: construct a relational equation by taking the execution duration of each of the plurality of operators, the number of task blocks corresponding to each of the plurality of operators, and the computing power information of the operator execution device as parameters, taking the scheduling strategy parameter as an independent variable, taking the total execution duration of the plurality of operators as a dependent variable, and taking the parallelism between the operators as a constraint condition; and use the relational equation as the association.

[0044] In a possible implementation, the scheduling strategy parameter includes at least one of the following: at least one execution time corresponding to each of the plurality of operators, a task block identifier of a starting task block corresponding to each of the at least one execution time, a required number of operation units, and an identifier of an operation unit that executes the operator.

[0045] In a possible implementation, the scheduling module, when scheduling the operation units to execute the plurality of operators based on the scheduling strategy, is configured to: generate an operator start instruction based on the scheduling strategy; and send the operator start instruction to the operator execution device; the operator start instruction is used to instruct the operator execution device to execute an operator corresponding to the operator start instruction in a manner indicated by the operator start instruction.

[0046] In a possible implementation, the scheduling strategy parameter includes: an operator execution time corresponding to each of the plurality of operators; and the scheduling module, when sending the operator start instruction to the operator execution device, is configured to: in response to an instruction sending time corresponding to any operator start instruction being reached, send the any operator start instruction to the operator execution device; the instruction sending time is determined based on an instruction execution time corresponding to the any operator start instruction.

[0047] In a fourth aspect, the embodiments of the present disclosure further provide a chip, including the scheduling system according to the first aspect or any possible implementation of the first aspect.

[0048] In a fifth aspect, the optional implementation of the present disclosure further provides a computer device, including the chip according to the fourth aspect.

[0049] In a sixth aspect, the optional implementation of the present disclosure further provides a computer readable storage medium, which stores a computer program. When the computer program is run, the steps in the first aspect or any possible implementation of the first aspect are executed.

[0050] For the effect description of the above scheduling method, device, chip, computer equipment and computer readable storage medium, please refer to the description of the scheduling method above, which will not be repeated here.

[0051] In order to make the above objectives, characteristics and advantages of the present disclosure more obvious and easy to understand, the following preferred embodiments are specifically described below, and the accompanying drawings are described in detail as follows. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following will briefly introduce the drawings needed to be used in the embodiments. The drawings herein are incorporated into the specification and form a part of the specification, which show the embodiments consistent with the present disclosure, and are used to explain the technical solutions of the present disclosure together with the specification. It should be understood that the following drawings only show some embodiments of the present disclosure, and therefore should not be regarded as a limitation on the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0053] Figure 1 A flow chart of a scheduling method provided by an embodiment of the present disclosure is shown;

[0054] Figure 2 A specific architecture example of the scheduling method provided by an embodiment of the present disclosure is shown;

[0055] Figure 3 A flow chart of a specific method for determining a scheduling strategy in the scheduling method provided by an embodiment of the present disclosure is shown;

[0056] Figure 4 A flow chart of a specific method for executing multiple operators by an operation unit in the scheduling operator execution device in the scheduling method provided by an embodiment of the present disclosure is shown;

[0057] Figure 5 A flow chart of a scheduling device provided by an embodiment of the present disclosure is shown;

[0058] Figure 6 A flow chart of an operator execution method provided by an embodiment of the present disclosure is shown;

[0059] Figure 7 A schematic diagram of a scheduling system provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0060] To make the purposes, technical solutions, and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure and are not all the embodiments. The components of the embodiments of the present disclosure described and shown herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure is not intended to limit the scope of the claimed present disclosure, but only represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present disclosure.

[0061] It is found through research that, with the wide use of artificial intelligence, the model and data volume of deep learning are increasing, which naturally leads to a large increase in operators or operations that need to be executed by computers, and the proportion of scheduling overhead of computing resources in computers is increasing. On the one hand, a host (Host) running a deep learning framework such as TensorFlow, Pytorch, Paddlepaddle, etc. obtains operators in a deep model by analyzing various deep learning models, and simply dispatches various operators to an operator execution device; the operator execution device is responsible for scheduling and executing the operators. The host cannot know the state of the computing resources in the execution device, and thus cannot control the dispatching of the operators according to the state of the computing resources in the execution device, resulting in a decrease in computing efficiency.

[0062] On the other hand, in the process of neural network training, the training system of the deep learning model needs to execute operators including pure computing class operators and communication class operators. The pure computing class operators include, for example, convolution operators, fully connected operators, etc.; the communication class operators include, for example, allreduce operators. When the execution device executes these two kinds of operators, the computing and communication will preempt the computing resources. This not only leads to huge fluctuations or jitters in computing, but also significantly reduces the efficiency of the training system.

[0063] Based on the above research, the present disclosure provides a scheduling method. The host determines a scheduling strategy for an operation unit when executing an operator based on operation information corresponding to a plurality of operators in a deep learning model and computing power information of an execution device, and schedules the operation unit to execute the operator based on the scheduling strategy, so as to be able to control the dispatching of the operator according to the planning of the specific process of the execution device executing the operator, and improve the utilization rate of the computing resources in the execution device.

[0064] The above-mentioned defects are the results of the inventors' practice and careful research, and thus the discovery process of the above-mentioned problems and the solutions proposed by the present disclosure to solve the above-mentioned problems should be the contributions of the inventors to the present disclosure.

[0065] It should be noted that similar reference numerals and letters represent similar items in the following drawings, and thus, once an item is defined in one drawing, it need not be further defined and explained in subsequent drawings.

[0066] To facilitate the understanding of the present embodiment, first, a scheduling method disclosed by the present embodiment is introduced in detail, and the execution subject of the scheduling method provided by the present embodiment is, for example, a host in a computer device, and the host is, for example, a central processing unit (CPU) in a computer. In some possible implementation manners, the scheduling method can be implemented by the host calling computer-readable instructions stored in a memory.

[0067] Next, the scheduling method provided by the present embodiment is described by taking the host as an example.

[0068] Referring to FIG. 1, Figure 1 The method includes steps S101-S103, and specifically includes the following steps.

[0069] S101: Analyzing a deep learning model to obtain a plurality of operators in the deep learning model;

[0070] S102: Based on operation information corresponding to each of the plurality of operators and computing power information of an operator execution device, determining a scheduling strategy for operation units in the operator execution device when executing the plurality of operators;

[0071] S103: Based on the scheduling strategy, scheduling the operation units to execute the plurality of operators.

[0072] The present embodiment determines a scheduling strategy for operation units when executing operators based on operation information corresponding to each of a plurality of operators in a deep learning model and computing power information of an execution device, and schedules operation units to execute operators based on the scheduling strategy, so as to be able to plan the specific process of executing operators in the execution device in advance, control the delivery of operators, and improve the utilization rate of computing resources in the execution device.

[0073] Next, S101-S103 are described in detail by taking an application of the scheduling method provided by the present embodiment to the field of deep learning as an example.

[0074] For the above S101: a deep learning framework or a deep model training system can be deployed in the computer device; the processor in the computer device can execute the deep learning model in the environment of the deep learning framework; or the processor trains the deep learning model to be trained in the environment of the deep model training system.

[0075] When the processor executes the deep learning model or trains the deep learning model, the deep learning model can be parsed to obtain a plurality of operators included in the deep learning model, for example; the operators include computing operators and communication operators, for example; the computing operators include convolution operators, fully connected operators, activation operators, etc.; the communication operators include global reduction operators allreduce, global collection operators allgather, and syncBN, etc.

[0076] For the above S102:

[0077] In specific implementations, the operator execution device includes a GPU, an AI accelerator card, etc.; the commonly used AI accelerator hardware architecture mainly includes an execution unit and a scheduler, etc.; the execution unit in the operator execution device includes processing cores in the GPU, for example.

[0078] As shown in the example of FIG. 1, the deep learning model is parsed to obtain a plurality of operators, and the operators are sent to a stream queue (SQ) in the computer device. Figure 2 As shown in the example of FIG. 1, the deep learning model is parsed to obtain a plurality of operators, and the operators are sent to a stream queue (SQ) in the computer device.

[0079] The operator scheduler obtains the operator execution instructions from the stream queue (SQ), and dispatches the operator execution instructions to different execution units based on the types of the operators; there are various execution units; different types of execution units can execute different operators; the types of the operators include computing operators for executing computing tasks, operators for executing data copying tasks, and operators for performing data synchronization tasks, for example.

[0080] In an execution unit for performing a computing task, a block scheduler and a plurality of operation units are included. The block scheduler is configured to receive an execution instruction dispatched by an operator scheduler, and decompose the operator into a plurality of task blocks, each of which includes a plurality of sub-tasks. The block scheduler is capable of sensing whether the operation units are in a working state, and dispatches the task blocks to the corresponding operation units according to the related information of the scheduling strategy carried in the operator execution instruction after obtaining the task blocks. The operation units execute the computing task corresponding to the task blocks after receiving the task blocks.

[0081] Each operation unit is generally composed of a two-dimensional processing engine (PE) array and a local register file. Each PE includes a multiplier-accumulator and other computing elements for performing specific computing tasks of each block. Each operation unit is capable of synchronously processing a plurality of data, for example, the PE array in an operation unit includes S PEs, and the S PEs are capable of synchronously processing S data; that is, the operation unit can synchronously process at most S data. The task of processing each data is called a sub-task, and S sub-tasks completed synchronously by S PEs in an operation unit constitute a task block.

[0082] The number of task blocks corresponding to an operator is related to the configuration of the operation unit; assuming that the number of data to be processed corresponding to the operator includes H, and the number of PEs in the PE array in each operation unit is S, then the number of task blocks corresponding to the operator is the integer obtained by rounding up H / S. For example, the data to be processed is image data, and the size of the image data is w*h*c, where w represents the image width, h represents the image height, and c represents the number of channels of the image. The number of data to be processed H satisfies: H=w*h*c.

[0083] For each task block, since one operation unit is used to synchronously process a plurality of sub-tasks corresponding to the task block, the task block execution duration corresponding to the task block is the same as or approximately equal to the execution duration of one sub-task. The sub-task is, for example, a processing task of any data in the image data.

[0084] The execution duration of the task blocks corresponding to the plurality of operators is the duration consumed by the operator execution device to execute one task block of the corresponding operator; therefore, in another embodiment of the present disclosure, before determining the scheduling strategy, the execution duration of the task blocks corresponding to the plurality of operators is obtained. The determination of the execution duration of the task blocks can adopt at least one of the following (1) or (2), but is not limited thereto:

[0085] (1) estimating an execution duration required for executing the task block based on memory required by the task block when executing and computing power information of the operation unit.

[0086] In a specific implementation, the computing power information of the operation unit, for example, includes operations per second (ops) of the operation unit; for example, the computing power information of each operation unit can be determined in the following manner: determining the number of PEs included in the operation unit, wherein one PE can perform n operations per second, which represents the computing power information of one PE; determining the computing power information of the operation unit, that is, the total number of OPS of one operation unit, according to the number of PEs included in the operation unit and the computing power information of each PE, wherein the computing power information of the operation unit satisfies: R*P*n. Wherein R represents the number of PEs in one operation unit, and P represents the number of operation units included in the execution unit.

[0087] When estimating the execution duration required for executing the task block based on the memory required by the task block when executing and the computing power information of the operation unit, for example, the memory access duration can be determined based on the memory required by the operator when executing and the memory access bandwidth of the operation unit to the memory; and the computing duration of the task block can be determined based on the computing power information required by each computing step in each subtask in the task block and the computing power information of each operation unit.

[0088] The execution duration required for executing the task block is determined based on the memory access duration and the computing duration.

[0089] The memory required by the task block when executing can represent the amount of data to be processed by the task block when executing. Since the type of the operator is known, the computing steps of each subtask when executing the operator can be determined, and thus the ops required by the PE when executing one computing operation can be determined, for example, the PE is a multiplier-adder, which can be used to execute multiplication and addition operations, and 2 ops are required for each multiplication and addition operation; the computing steps of each subtask are known, that is, the required computing operations are known, and thus the ops required by each computing step can be determined according to the ops required by each computing step; then the computing duration of one subtask is determined according to the specific computing steps of the subtask; one task block includes multiple subtasks executed in parallel by multiple PEs, and thus the computing duration of each subtask, that is, the computing duration of the task block.

[0090] The memory access duration of the task block is determined according to the amount of data required by the memory access task corresponding to one task block to access the memory and the memory bandwidth.

[0091] determine an execution duration required for executing one task block based on the computation duration and the memory access duration.

[0092] For example, the operator is a matrix multiplication operator, and the matrix multiplication task in each task block satisfies: CMN=A MK *B KN wherein M and K represent the dimensions of the operator A respectively; K and N represent the dimensions of the operator B respectively; each data needs to occupy b bits, and the data amount required for the operator to access the memory during execution (memory access data amount) is (M*K+K*N+M*N)*b / 8 bytes. Wherein M*K represents the data amount of the operator A required to be read from the memory; K*N represents the data amount of the operator B required to be read from the memory; and M*N represents the data amount of the calculation result to be stored in the memory. The number of multipliers (i.e., PEs) in each operation unit in the execution unit for executing the operator is M*K*N.

[0093] The computation duration required for executing one task block of the operator is n*M*K*N ÷ total OPS number of operation units, and the memory access duration required for the operation unit to execute one task block of the operator is memory access data amount ÷ memory access bandwidth.

[0094] The execution duration of the first task block corresponding to the operator includes the sum of the computation duration and the memory access duration, and thus the execution duration of one task block of the matrix multiplication operator is n*M*K*N ÷ total OPS number of operation units + memory access data amount ÷ memory access bandwidth.

[0095] (2) determining a simulation model corresponding to the operator based on the operator, running the simulation model, and determining the execution duration of each task block of the operator based on the runtime of the simulation model.

[0096] In a specific implementation, the simulation model corresponding to the operator determined based on the operator only includes the operator and does not include other operators. The parameters of the operator in the simulation model are consistent with the parameters of the operator in the deep learning model. Then the simulation model is run to obtain the runtime of the simulation model.

[0097] Since the size W*H and the number S of the operation units in the operator execution device running the simulation model are known, the number of task blocks U / (W*H) obtained by dividing the operators during simulation can be determined according to the size W*H of the operation units in the operator execution device running the simulation model and the amount of data U that needs to be processed by the operator, and then since multiple operation units can synchronously process multiple task blocks, the batch U / (W*H) / S that needs to be processed can be determined according to the number of task blocks and the number S of operation units. Then, the execution time of each task block is obtained according to the running time T of the simulation model and the batch that needs to be processed: T / (U / (W*H) / S)

[0098] Here, the configuration of the execution device used when running the simulation model can be consistent with the configuration of the operator execution device when running the deep learning model, or can be inconsistent.

[0099] After the processor of the computer device parses the deep learning model to obtain the multiple operators included in the deep learning model, the execution time of the task blocks in each operator is determined according to the above-mentioned manner (1) or (2), and the scheduling strategy for the operation units in the operator execution device is determined according to the operation information corresponding to the multiple operators respectively and the computing power information of the operator execution device, and then the corresponding instructions are generated according to the scheduling strategy, and the instructions are sent to the control unit in the operator execution device; the control unit schedules multiple operation units to execute the corresponding operators based on the instructions.

[0100] The operation information corresponding to each operator, for example, includes: the execution time of the task blocks corresponding to the multiple operators respectively, and the parallelism between the multiple operators; wherein each task block includes multiple sub-tasks in the corresponding operator.

[0101] Referring to Figure 3 The embodiments of the present disclosure also provide a specific method for determining a scheduling strategy for the operation units when executing multiple operators, which comprises:

[0102] S301: constructing the association relationship between the execution time of the task blocks corresponding to the multiple operators respectively, the number of the task blocks corresponding to the multiple operators respectively, the parallelism between the multiple operators, the computing power information of the operator execution device, the scheduling strategy parameters, and the total execution time of the multiple operators.

[0103] S302: adjusting the scheduling strategy parameters based on the association relationship to reduce the total execution time, and obtaining a target scheduling strategy.

[0104] The scheduling policy parameters include, for example, at least one execution time corresponding to each of the plurality of operators, a task block identifier of a starting task block corresponding to each of the at least one execution time, a required number of operation units, and an identifier of an operation unit performing the operator. The scheduling policy parameters can be used to represent a scheduling policy. For example, when the scheduling policy parameters include at least one execution time corresponding to each of the plurality of operators, a task block identifier of a starting task block corresponding to each of the at least one execution time, and a required number of operation units, in response to the arrival of any operator execution time, the host sends an operator start instruction carrying the task block identifier of the corresponding starting task block and the required number of operation units to the operator execution device, that is, the scheduling policy represented by the scheduling policy parameters. After receiving the operator start instruction, the operator execution device schedules operation units according to the required number of operation units, and controls the scheduled operation units to perform the task block according to the task block identifier.

[0105] The association relationship can be represented in the form of an equation, that is, the execution time of the task block corresponding to each of the plurality of operators, the number of task blocks corresponding to each of the plurality of operators, the computing power information of the operator execution device, the scheduling policy parameters as the independent variable, the total execution time of the plurality of operators as the dependent variable, and the parallelism between the operators as the constraint condition are used to construct a relationship equation, and the independent variable in the relationship equation is adjusted to optimize the total execution time, and finally the scheduling policy obtained under the condition that the total execution time reaches the minimum (for example, in multiple optimization iteration periods, the total execution time no longer decreases) is used as the target scheduling policy.

[0106] During the adjustment of the independent variable, for example, the following multiple iteration periods can be performed:

[0107] In the 0th iteration period, the scheduling policy parameters are determined, and the total execution time of the 0th iteration period is determined according to the scheduling policy parameters and the relationship equation. The scheduling policy parameters are used as the reference scheduling policy parameters of the 1st iteration period, and the total execution time of the 0th iteration period is used as the reference total execution time of the 1st iteration period, and the 1st iteration period is entered.

[0108] In the first iteration cycle, the reference scheduling policy parameter of the first iteration cycle is adjusted; based on the adjusted scheduling policy parameter and the relationship equation, a new total execution time is determined; the new total execution time and the reference total execution time are compared; if the new total execution time is less than the reference total execution time, the adjusted scheduling policy parameter determined in the first iteration cycle is taken as the reference scheduling policy parameter of the second iteration cycle, and the new total execution time is taken as the reference total execution time of the second iteration cycle, and the second iteration cycle is entered; if the new total execution time is greater than or equal to the reference total execution time, the reference scheduling policy parameter of the first iteration cycle is taken as the reference scheduling policy parameter of the second iteration cycle, and the reference total execution time of the first iteration cycle is taken as the reference total execution time of the second iteration cycle, and the second iteration cycle is entered.

[0109] …

[0110] In the i th iteration cycle, the reference scheduling policy parameter of the i th iteration cycle is adjusted; based on the adjusted scheduling policy parameter and the relationship equation, a new total execution time is determined; the new total execution time and the reference total execution time are compared; if the new total execution time is less than the reference total execution time, the adjusted scheduling policy parameter determined in the i th iteration cycle is taken as the reference scheduling policy parameter of the (i+1) th iteration cycle, and the new total execution time is taken as the reference total execution time of the (i+1) th iteration cycle, and the (i+1) th iteration cycle is entered; if the new total execution time is greater than or equal to the reference total execution time, the reference scheduling policy parameter of the i th iteration cycle is taken as the reference scheduling policy parameter of the (i+1) th iteration cycle, and the reference total execution time of the i th iteration cycle is taken as the reference total execution time of the (i+1) th iteration cycle, and the (i+1) th iteration cycle is entered.

[0111] The above process is iteratively executed until the reference total execution time no longer changes in the consecutive n iteration cycles.

[0112] In another embodiment, the execution strategy of the above operator can be determined according to the deep learning model and the operator execution device before executing the deep learning model, or can be a scheduling policy corresponding to each operator execution device determined in advance for a certain deep learning model when the deep learning model is executed on multiple operator execution devices; when the deep learning model is executed, the scheduling policy corresponding to the used operator execution device can be directly called through the provided interface.

[0113] It should be noted that the disclosure does not limit the equation form corresponding to the above-mentioned association relationship, and the user using the scheme provided by the disclosure can determine the corresponding equation form according to the needs of the user and the hardware environment (such as the number of operation units, computing power, tasks to be executed, etc.) of the application of the scheme provided by the disclosure, or in combination with a certain algorithm to determine the corresponding equation form, and adjust the equation using the above-mentioned iteration method, under the condition that the total execution time reaches the minimum, the corresponding scheduling strategy parameters are obtained, that is, the scheduling strategy is obtained.

[0114] For example, after the deep learning framework parses the deep learning model, the obtained operators include: A, B, C, and D. The parallelism between A, B, C, and D includes: A and B need to be executed in sequence, C and D need to be executed in sequence; and AB and CD have no dependency relationship, that is, AC, AD, BC, and BD can be executed in parallel; the execution priority of operator B is higher than that of operators C and D.

[0115] According to any one of the above (1) or (2), the number of task blocks corresponding to A, B, C, and D respectively and the execution time required by each task block to run are as shown in Table 1 respectively:

[0116] Table 1

[0117] Operator thread cycle A 68 20 B 8 80 C 60 20 D 48 20

[0118] Wherein, thread represents a task block, and cycle represents the execution time of the task block, and the unit is: processing cycle.

[0119] In the operator execution device, the number of operation units is 16; each operation unit can synchronously process a plurality of sub-tasks in a task block.

[0120] Then, the theoretical time consumption of each operator corresponding to the serial execution of the operator execution device is as shown in Table 2:

[0121] Table 2

[0122] Operator thread cycle Serial schedule Theoretical time consumption A 68 20 100 1360 B 8 80 80 640 C 60 20 80 1200 D 48 20 60 960 total 320 260

[0123] Serial scheduling represents the number of processing cycles required when executing any operator.

[0124] Taking the operator A as an example, it includes 68 task blocks, and since 16 operation units are included in the operator execution device, if the operator A is to be executed, the calculation batches required are: 68 / 16 = 4.25; since multiple operators are executed in series, when the operator A is executed, all operation units will be scheduled in the processing of any batch regardless of whether all operation units will actually be used or not, therefore, 5 batches are required to implement the processing of the operator. Each batch occupies 20 processing cycles, and therefore the processing cycles required by 5 batches are 20*5 = 100, that is, 100 processing cycles are required to process the task corresponding to the operator A.

[0125] Theoretical time consumption: indicates the theoretical processing cycle number required to execute the operator A. Taking the operator A as an example, it includes 68 task blocks, and the execution time of each task block is 20 processing cycles, therefore the theoretical processing cycle number is: 68*20 = 1360 processing cycles.

[0126] It can be known that the total execution time of the four operators is: 320 processing cycles.

[0127] And theoretically, the total execution time of the four operators = sum(thread*cycle) / 16 = 260.

[0128] If the prior art is adopted to schedule the operation units by the GPU, the specific scheduling process is shown in Table 3:

[0129] Table 3

[0130] time A B C D 0 16x80 80 4x20 12x20 100 8x80 8x80 180 16x20 200 4x20 220 16x60 280 done

[0131] Wherein, time indicates: the i-th processing cycle. For the above four operators A-D:

[0132] In the 0th processing cycle, 16 operation units are all allocated to the operator A, and 80 processing cycles are executed;

[0133] In the 80th processing cycle, 4 operation units are allocated to the operator A, 20 processing cycles are executed, and the operator A is executed completely; and 12 operation units are allocated to the operator C, 20 processing cycles are executed.

[0134] In the 100th processing cycle, since the execution priority of the operator B is higher than that of the operator C, 8 operation units are allocated to the operator B, 80 processing cycles are executed, and the operator B is executed completely; the idle 8 operation units are allocated to the operator C, 80 processing cycles are executed;

[0135] In the 180th processing cycle, 16 operation units are all allocated to the operator C, and 20 processing cycles are executed.

[0136] In the 200th processing cycle, 4 operation units are allocated to operator C, 20 processing cycles are executed, and operator C is executed completely.

[0137] In the 220th processing cycle, 16 operation units are allocated to operator D, 60 processing cycles are executed, and operator D is executed completely.

[0138] By the 280th processing cycle, operators A to D are all executed completely.

[0139] If the scheduling method provided in the embodiments of the present disclosure is used, the specific process of scheduling is shown in Table 4 as follows:

[0140] Table 4

[0141]

[0142]

[0143] Among the above four operators A to D,

[0144] In the 0th processing cycle, 16 operation units are all allocated to operator A, and 20 processing cycles are executed.

[0145] In the 20th processing cycle, 8 operation units are allocated to operator A, and 120 processing cycles are executed, and 8 operation units are allocated to operator C, and 120 processing cycles are executed.

[0146] In the 140th processing cycle, 4 operation units are allocated to operator A, 20 processing cycles are executed, and operator A is executed completely, and 12 operation units are allocated to operator C, and 20 processing cycles are executed, and operator C is executed completely.

[0147] In the 160th processing cycle, 8 operation units are allocated to operator B, 80 processing cycles are executed, and operator B is executed completely, and 8 operation units are allocated to operator D, and 80 processing cycles are executed.

[0148] In the 240th processing cycle, 16 operation units are allocated to operator D, 20 processing cycles are executed, and operator D is executed completely.

[0149] In the 260th processing cycle, operators A to D are all executed completely.

[0150] Therefore, it is seen that 280 processing cycles are needed for scheduling by using a GPU, and only 260 processing cycles are needed for scheduling by using the scheduling method provided in the embodiments of the present disclosure. Therefore, the scheduling method provided in the embodiments of the present disclosure can improve the execution efficiency of operators, can make more sufficient use of computing power, and can improve the utilization rate of computing power.

[0151] In the foregoing S103, the scheduling strategy comprises at least one operator execution time point corresponding to each of the plurality of operators.

[0152] Referring to Figure 4 The embodiment of the present disclosure provides a specific manner for an operation unit in an operator execution device to execute a plurality of operators after obtaining a scheduling strategy, comprising:

[0153] S401: generating an operator starting instruction based on the scheduling strategy;

[0154] S402: sending the operator starting instruction to the operator execution device;

[0155] The operator starting instruction is used for the operator execution device to execute the operator corresponding to the operator starting instruction in the manner indicated by the operator starting instruction.

[0156] In a possible implementation, the scheduling strategy comprises at least one operator execution time point corresponding to each of the plurality of operators; when sending the operator starting instruction to the operator execution device, the following manner can be used, for example:

[0157] In response to the arrival of the instruction sending time point corresponding to any operator starting instruction, the operator execution device is sent the any operator starting instruction; wherein the instruction sending time point is determined based on the instruction execution time point corresponding to the any operator starting instruction.

[0158] When generating the operator starting instruction, for example, an operator starting instruction corresponding to each operator execution time point is generated. For example, in the example corresponding to Table 4, the operator execution time points are the 0th processing period, the 20th processing period, the 140th processing period, the 160th processing period, and the 240th processing period, respectively, then for the 0th processing period, the corresponding operator starting instruction a1 is generated; for the 20th processing period, the corresponding operator starting instruction a2 is generated; for the 140th processing period, the corresponding operator starting instruction a3 is generated; for the 160th processing period, the corresponding operator starting instruction a4 is generated; and for the 240th processing period, the corresponding operator starting instruction a5 is generated.

[0159] For the instruction sending time point corresponding to any operator starting instruction, for example, when the instruction execution time point corresponding to the operator starting instruction is determined, the instruction execution time point is reduced by at least one processing period to obtain the instruction sending time point. For example, in the example shown in Table 4, the 16 operation units need to be allocated to operator A in the 0th processing period, then the instruction sending time point can be determined as the -1th processing period. Here, the ith processing period is relative to the four operators A-D.

[0160] In the 20th processing cycle, 8 operation units are needed to be allocated to the operator A, and 8 operation units are needed to be allocated to the operator C, and thus the sending time of the instruction can be determined as the 19th processing cycle.

[0161] In the generated operator starting instruction, at least one of the task block identifier of the starting task block, the number of required operation units, and the identifier of the operation unit executing the operator can also be carried. In this way, the specific manner of scheduling can be informed to the operator execution device.

[0162] When the host issues the operator starting instruction to the operator execution device, the operator starting is stored to the stream queue SQ; the operator scheduler reads the operator starting instruction from the stream queue, and according to the operator type carried in the operator starting instruction, issues the operator starting instruction to the execution unit capable of executing the calculation task corresponding to the operator, that is, the thread block scheduler in the corresponding execution unit. The thread block scheduler decomposes the operator starting instruction into multiple task blocks (which can include part of the task blocks in the operator) according to the task block identifier of the starting task block carried in the operator starting instruction, and issues the task blocks to the corresponding operation units for processing according to the number of required operation units or the identifier of the operation unit executing the operator.

[0163] Those skilled in the art can understand that in the above method of the specific embodiment, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible inherent logic.

[0164] Based on the same inventive concept, the disclosure embodiments also provide a scheduling device corresponding to the scheduling method. Since the principle of solving problems by the device in the disclosure embodiments is similar to the above-mentioned scheduling method of the disclosure embodiments, the implementation of the device can be referred to the implementation of the method, and the repeated parts will not be described.

[0165] Reference Figure 5 As shown in the figure, a scheduling device provided by the disclosure embodiments includes:

[0166] The analysis module 51 is configured to analyze the deep learning model to obtain a plurality of operators in the deep learning model.

[0167] The generation module 52 is configured to determine a scheduling strategy for operation units in the operator execution device when executing a plurality of operators based on operation information corresponding to each of the plurality of operators and computing power information of the operator execution device. Each task block includes a plurality of sub-tasks in the corresponding operator.

[0168] The scheduling module 53 is configured to schedule the operation units to execute the plurality of operators based on the scheduling strategy.

[0169] In a possible implementation, the operation information corresponding to each of the plurality of operators includes execution time lengths of task blocks corresponding to the plurality of operators respectively, and parallelism degrees between the plurality of operators; and each task block includes a plurality of sub-tasks in a corresponding operator.

[0170] In a possible implementation, the apparatus further includes an obtaining module 54, configured to obtain execution time lengths of task blocks corresponding to the plurality of operators respectively.

[0171] In a possible implementation, the obtaining module 54, when obtaining the execution time lengths of the task blocks corresponding to the plurality of operators respectively, is configured to:

[0172] estimate the execution time length required for executing the task block based on memory required by the task block when being executed and computing power information of the operation unit.

[0173] In a possible implementation, the obtaining module 54, when estimating the execution time length required for executing the task block based on the memory required by the task block when being executed and the computing power information of the operation unit, is configured to:

[0174] determine a memory access time length based on memory required by the operator when being executed and memory access bandwidth of the operation unit to the memory; and

[0175] determine a computing time length of the task block based on computing power information required by each computing step in each sub-task in the task block and the computing power information of each operation unit;

[0176] determine the execution time length required for executing the task block based on the memory access time length and the computing time length.

[0177] In a possible implementation, the obtaining module 54, when obtaining the execution time lengths of the task blocks corresponding to the plurality of operators respectively, is configured to:

[0178] determine a simulation model corresponding to the operator based on the operator;

[0179] run the simulation model, and determine the execution time length required by each task block of the operator based on a running time length of the simulation model.

[0180] In a possible implementation, the obtaining module 54, when determining the execution time length required by each task block of the operator based on the running time length of the simulation model, is configured to:

[0181] determine the number of task blocks obtained by dividing the operator in the simulation process according to the size of the operation unit in the operator execution device running the simulation model and the amount of data required to be processed by the operator when executing;

[0182] determine the batch to be processed according to the number of task blocks obtained by dividing the operator in the simulation process and the number of operation units in the operator execution device running the simulation model;

[0183] determine the execution duration of the task blocks corresponding to the plurality of operators respectively based on the batch and the running duration of the simulation model.

[0184] In a possible implementation, the generation module 52, when determining the scheduling strategy for the operation units in the operator execution device when executing the plurality of operators based on the execution duration of the task blocks corresponding to the plurality of operators respectively, the parallelism between the plurality of operators, and the computing power information of the operator execution device, is configured to:

[0185] construct the association relationship between the execution duration of the task blocks corresponding to the plurality of operators respectively, the number of task blocks corresponding to the plurality of operators respectively, the parallelism between the plurality of operators, the computing power information of the operator execution device, the scheduling strategy parameter, and the total execution duration of the plurality of operators;

[0186] adjust the scheduling strategy parameter based on the association relationship to obtain a target scheduling strategy, with the goal of reducing the total execution duration.

[0187] In a possible implementation, the generation module 52, when constructing the association relationship between the execution duration of the task blocks corresponding to the plurality of operators respectively, the number of task blocks corresponding to the plurality of operators respectively, the parallelism between the plurality of operators, the computing power information of the operator execution device, the scheduling strategy parameter, and the total execution duration of the plurality of operators, is configured to:

[0188] construct a relationship equation with the execution duration of the task blocks corresponding to the plurality of operators respectively, the number of task blocks corresponding to the plurality of operators respectively, and the computing power information of the operator execution device as parameters, with the scheduling strategy parameter as the independent variable, with the total execution duration of the plurality of operators as the dependent variable, and with the parallelism between the operators as the constraint condition; and use the relationship equation as the association relationship.

[0189] In a possible implementation, the scheduling strategy parameter includes at least one of the following:

[0190] at least one execution time point corresponding to the plurality of operators respectively, a task block identifier of a start task block corresponding to each of the at least one execution time point, a number of required operation units, and an identifier of an operation unit performing the operator.

[0191] In a possible implementation, the scheduling module 53, when scheduling the operation units to execute the plurality of operators based on the scheduling strategy, is configured to:

[0192] generate an operator start instruction based on the scheduling strategy;

[0193] send the operator start instruction to the operator execution device;

[0194] The operator start instruction is used for the operator execution device to execute an operator corresponding to the operator start instruction in a manner indicated by the operator start instruction.

[0195] In a possible implementation, the scheduling strategy parameter includes an operator execution time point corresponding to the plurality of operators respectively.

[0196] The scheduling module 53, when sending the operator start instruction to the operator execution device, is configured to: in response to an instruction sending time point of any operator start instruction being reached, send the any operator start instruction to the operator execution device; wherein the instruction sending time point is determined based on an instruction execution time point corresponding to the any operator start instruction.

[0197] The description of the processing procedure of each module in the apparatus and the interaction procedure between the modules can refer to the related description in the foregoing method embodiments, and will not be described in detail here.

[0198] Referring to Figure 6 The embodiments of the present disclosure further provide an operator execution method, which includes:

[0199] S601: A host parses a deep learning model to obtain a plurality of operators in the deep learning model; determines a scheduling strategy for operation units in an operator execution device when executing the plurality of operators based on operation information corresponding to the plurality of operators respectively and computing power information of the operator execution device; generates an operator start instruction based on the scheduling strategy; and sends the operator start instruction to the operator execution device.

[0200] S602: The operator execution device receives the operator start instruction sent by the host, and executes an operator corresponding to the operator start instruction in a manner indicated by the operator start instruction.

[0201] The embodiment of the present disclosure determines the scheduling strategy of the operation unit when the operators are executed based on the operation information corresponding to each of the plurality of operators in the deep learning model and the computing power information of the execution device, and schedules the operation unit to execute the operators based on the scheduling strategy, so that the specific process of executing the operators by the execution device can be planned in advance, the operators are controlled to be issued, and the utilization rate of the computing resources in the execution device is improved.

[0202] Referring to Figure 7 The embodiment of the present disclosure also provides a scheduling system, which is a hardware system and comprises:

[0203] a host 71 and an operator execution device 72;

[0204] The host 71 is configured to analyze a deep learning model to obtain a plurality of operators in the deep learning model, and determine a scheduling strategy of operation units in the operator execution device when the plurality of operators are executed based on operation information corresponding to each of the plurality of operators and computing power information of the operator execution device.

[0205] The operator execution device 72 is configured to execute the plurality of operators based on the scheduling strategy of the host.

[0206] In a possible implementation, the operation information corresponding to each of the plurality of operators comprises execution time of a task block corresponding to each of the plurality of operators and parallelism between the plurality of operators.

[0207] Each task block comprises a plurality of subtasks in the corresponding operator.

[0208] In a possible implementation, when the host 71 obtains the execution time of the task block corresponding to each of the plurality of operators, the host 71 is configured to:

[0209] estimate the execution time required for executing the task block based on memory required by the task block when the task block is executed and the computing power information of the operation unit.

[0210] In a possible implementation, when the host 71 estimates the execution time required for executing the task block based on the memory required by the task block when the task block is executed and the computing power information of the operation unit, the host 71 is configured to:

[0211] determine the memory access time based on the memory required by the operator when the operator is executed and the memory access bandwidth of the operation unit to the memory; and

[0212] determine the calculation time of the task block based on the computing power information required by each calculation step in each subtask in the task block and the computing power information of each operation unit.

[0213] determine an execution duration required for executing the task block based on the access duration and the calculation duration.

[0214] In a possible implementation, the host 71, when determining the execution duration of each task block of the operator based on the running duration of the simulation model, is configured to:

[0215] determine a simulation model corresponding to the operator based on the operator;

[0216] run the simulation model, and determine the execution duration required for each task block of the operator based on a running duration of the simulation model.

[0217] In a possible implementation, the host 71, when determining the execution duration of each task block of the operator based on the running duration of the simulation model, is configured to:

[0218] determine the number of task blocks obtained by dividing the operator in the simulation process according to a size of an operation unit in the operator execution device 72 running the simulation model and an amount of data required for processing by the operator during execution;

[0219] determine the batch required for processing according to the number of task blocks obtained by dividing the operator in the simulation process and the number of operation units in the operator execution device 72 running the simulation model;

[0220] determine the execution duration of each task block of the plurality of operators based on the batch and the running duration of the simulation model.

[0221] In a possible implementation, the host 71, when determining the scheduling strategy of the operation unit in the operator execution device 72 during execution of the plurality of operators based on the operation information of the plurality of operators and the computing power information of the operator execution device 72, is configured to:

[0222] construct an association relationship between the execution duration of each task block of the plurality of operators, the number of task blocks of each task block of the plurality of operators, the parallelism between the plurality of operators, the computing power information of the operator execution device 72, the scheduling strategy parameter, and the total execution duration of the plurality of operators;

[0223] adjust the scheduling strategy parameter based on the association relationship to obtain a target scheduling strategy, so as to reduce the total execution duration.

[0224] In a possible implementation, the host 71, when constructing the association between the execution duration of each of the plurality of operators corresponding to a task block, the number of task blocks corresponding to each of the plurality of operators, the parallelism between the plurality of operators, the computing power information of the operator execution device 72, the scheduling strategy parameter, and the total execution duration of the plurality of operators, is configured to:

[0225] construct a relational equation with the execution duration of each of the plurality of operators corresponding to a task block, the number of task blocks corresponding to each of the plurality of operators, the computing power information of the operator execution device 72 as parameters, the scheduling strategy parameter as an independent variable, the total execution duration of the plurality of operators as a dependent variable, and the parallelism between the operators as a constraint condition; and use the relational equation as the association.

[0226] In a possible implementation, the scheduling strategy parameter includes at least one of the following:

[0227] at least one execution time corresponding to each of the plurality of operators, the task block identifier of a starting task block corresponding to each of the at least one execution time, the number of required operation units, and the identifier of an operation unit executing the operator.

[0228] In a possible implementation, the host 71, when scheduling the operation units to execute the plurality of operators based on the scheduling strategy, is configured to:

[0229] generate an operator start instruction based on the scheduling strategy;

[0230] send the operator start instruction to the operator execution device 72;

[0231] The operator execution device 72, when executing the plurality of operators based on the scheduling of the host 71, is configured to: in response to receiving the operator start instruction sent by the host 71, execute an operator corresponding to the operator start instruction in a manner indicated by the operator start instruction.

[0232] In a possible implementation, the scheduling strategy parameter includes: an operator execution time corresponding to each of the plurality of operators; and the operator start instruction includes: an operator start instruction corresponding to each of the plurality of operators.

[0233] The host 71, when sending the operator start instruction to the operator execution device 72, is configured to: in response to the arrival of an instruction sending time corresponding to any operator start instruction, send the any operator start instruction to the operator execution device 72; wherein the instruction sending time is determined based on an instruction execution time corresponding to the any operator start instruction.

[0234] An embodiment of the present disclosure also provides a chip, including the scheduling system provided by the embodiment of the present disclosure.

[0235] The embodiments of the present disclosure also provide a computer device, comprising the chip as described in the embodiments of the present disclosure.

[0236] The present disclosure also provides a computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program executes the steps of the scheduling method or operator execution method described in the above method embodiments. The storage medium may be a volatile or non-volatile computer-readable storage medium.

[0237] The embodiments of the present disclosure also provide a computer program product, which carries program code. The instructions included in the program code can be used to execute the steps of the scheduling method or operator execution method described in the above method embodiments. For details, please refer to the above method embodiments and will not be repeated here.

[0238] The computer program product may be implemented in hardware, software, or a combination thereof. In one embodiment, the computer program product is implemented as a computer storage medium. In another embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0239] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems and devices described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0240] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0241] In addition, each function unit in various embodiments of the present disclosure can be integrated in one operation unit, or each unit can be physically present alone, or two or more units can be integrated in one unit.

[0242] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a nonvolatile computer readable storage medium executable by a processor. Based on this understanding, the technical solutions of the present disclosure essentially or the part of the prior art that contributes to the technical solutions or the part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present disclosure. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0243] Finally, it should be noted that: the above-described embodiments are merely specific embodiments of the present disclosure, used to illustrate the technical solutions of the present disclosure, rather than limit them. The protection scope of the present disclosure is not limited thereto, although the present disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand: any person skilled in the art within the technical range disclosed by the present disclosure can still modify or easily think of changes to the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some technical features; and these modifications, changes or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be covered within the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. A dispatch system characterized by, The method comprises the following steps: a host and an operator execution device; the host is used for parsing a deep learning model to obtain a plurality of operators in the deep learning model; based on the operation information corresponding to the plurality of operators respectively and the computing power information of the operator execution device, a scheduling strategy for the operation units in the operator execution device when executing the plurality of operators is determined; the operator execution device is used for executing the plurality of operators based on the scheduling strategy of the host; the operation information corresponding to the plurality of operators respectively comprises the execution time of the task blocks corresponding to the plurality of operators respectively, and the parallelism between the plurality of operators; wherein each task block comprises a plurality of sub-tasks in the corresponding operator; when obtaining the execution time of the task blocks corresponding to the plurality of operators respectively, the host is used for: estimating the execution time required for executing the task blocks based on the memory required by the task blocks when executing and the computing power information of the operation units; when obtaining the execution time of the task blocks corresponding to the plurality of operators respectively, the host is used for: determining a simulation model corresponding to the operator based on the operator; running the simulation model, and determining the execution time required for each task block of the operator based on the running time of the simulation model; when determining the scheduling strategy for the operation units in the operator execution device when executing the plurality of operators based on the operation information corresponding to the plurality of operators respectively and the computing power information of the operator execution device, the host is used for: constructing the association relationship between the execution time of the task blocks corresponding to the plurality of operators respectively, the number of task blocks corresponding to the plurality of operators respectively, the parallelism between the plurality of operators, the computing power information of the operator execution device, the scheduling strategy parameters, and the total execution time of the plurality of operators; based on the association relationship, the scheduling strategy parameters are adjusted to obtain a target scheduling strategy, so as to reduce the total execution time.

2. The dispatch system of claim 1, wherein, when estimating the execution time required for executing the task blocks based on the memory required by the task blocks when executing and the computing power information of the operation units, the host is used for: determining the memory access time based on the memory required by the operator when executing and the memory access bandwidth of the operation units to the memory; and determining the calculation time of the task block based on the computing power information required by each calculation step in each sub-task in the task block and the computing power information of each operation unit; determining the execution time required for executing the task block based on the memory access time and the calculation time. when determining the execution time required for each task block of the operator based on the running time of the simulation model, the host is used for:

3. The dispatch system of claim 1, wherein, determining the number of task blocks obtained by dividing the operator in the simulation process according to the size of the operation unit in the operator execution device running the simulation model and the amount of data required to be processed by the operator when executing; determining the batch to be processed according to the number of task blocks obtained by dividing the operator in the simulation process and the number of operation units in the operator execution device running the simulation model. ​ Determine execution durations of the task blocks corresponding to the plurality of operators respectively based on the batch and a running duration of the simulation model.

4. The dispatch system of claim 1, wherein, The host is configured to: build a relationship equation based on the execution durations of the task blocks corresponding to the plurality of operators respectively, the number of task blocks corresponding to the plurality of operators respectively, the parallelism between the plurality of operators, the computing power information of the operator execution device, the scheduling strategy parameter, and the association between the total execution duration of the plurality of operators.

5. The dispatch system of claim 1, wherein, The relationship equation takes the execution durations of the task blocks corresponding to the plurality of operators respectively, the number of task blocks corresponding to the plurality of operators respectively, and the computing power information of the operator execution device as parameters, takes the scheduling strategy parameter as an independent variable, takes the total execution duration of the plurality of operators as a dependent variable, and takes the parallelism between the operators as a constraint condition. The scheduling strategy parameter includes at least one of the following:

6. The dispatch system according to claim 1 or 5, characterized in that, at least one execution time corresponding to the plurality of operators respectively, a task block identifier of a starting task block corresponding to each execution time in the at least one execution time, a required number of operation units, and an identifier of an operation unit executing the operator. The host is configured to: generate an operator start instruction based on the scheduling strategy; send the operator start instruction to the operator execution device; 7. The dispatch system of claim 6, wherein, The operator execution device is configured to, when executing the plurality of operators based on the scheduling of the host: respond to receiving the operator start instruction sent by the host, and execute the operator corresponding to the operator start instruction in the manner indicated by the operator start instruction.

8. A scheduling method, characterized by, The scheduling strategy parameter includes: an operator execution time corresponding to the plurality of operators respectively; and The host is configured to: respond to the arrival of an instruction sending time corresponding to any operator start instruction, and send the any operator start instruction to the operator execution device; wherein the instruction sending time is determined based on an instruction execution time corresponding to the any operator start instruction. The method includes: parsing a deep learning model to obtain a plurality of operators in the deep learning model; determining a scheduling strategy for operation units in an operator execution device when executing the plurality of operators based on operation information corresponding to the plurality of operators respectively and computing power information of the operator execution device; wherein each task block includes a plurality of sub-tasks in a corresponding operator; scheduling the operation units to execute the plurality of operators based on the scheduling strategy; The operation information corresponding to the plurality of operators respectively includes execution durations of the task blocks corresponding to the plurality of operators respectively and parallelism between the plurality of operators; wherein each task block includes a plurality of sub-tasks in a corresponding operator; Obtaining the execution durations of the task blocks corresponding to the plurality of operators respectively includes: estimating the execution durations required for executing the task blocks based on the memory required by the task blocks when executing and the computing power information of the operation units; and The execution time lengths of the task blocks corresponding to the plurality of operators are obtained, including: determining a simulation model corresponding to the operator based on the operator; running the simulation model, and determining the execution time length required by each task block of the operator based on the running time length of the simulation model; The scheduling strategy of the operation unit in the operator execution device when executing the plurality of operators is determined based on the operation information corresponding to the plurality of operators and the computing power information of the operator execution device, including: constructing an association relationship between the execution time lengths of the task blocks corresponding to the plurality of operators, the number of task blocks corresponding to the plurality of operators, the parallelism between the plurality of operators, the computing power information of the operator execution device, the scheduling strategy parameters, and the total execution time length of the plurality of operators; target scheduling strategy is obtained by adjusting the scheduling strategy parameters based on the association relationship, with the goal of reducing the total execution time length.

9. A scheduling apparatus characterized by comprising: including: The parsing module is configured to parse the deep learning model to obtain a plurality of operators in the deep learning model; The generation module is configured to determine a scheduling strategy of an operation unit in an operator execution device when executing a plurality of operators based on operation information corresponding to the plurality of operators and computing power information of the operator execution device; wherein each task block includes a plurality of sub-tasks in the corresponding operator; The scheduling module is configured to schedule the operation unit to execute the plurality of operators based on the scheduling strategy; The operation information corresponding to the plurality of operators includes execution time lengths of task blocks corresponding to the plurality of operators and parallelism between the plurality of operators; wherein each task block includes a plurality of sub-tasks in the corresponding operator; Further comprising an acquisition module configured to obtain execution time lengths of task blocks corresponding to the plurality of operators; When obtaining the execution time lengths of the task blocks corresponding to the plurality of operators, the acquisition module is configured to: estimate the execution time length required for executing the task blocks based on the memory required by the task blocks when executing and the computing power information of the operation unit; The acquisition module, when obtaining the execution time lengths of the task blocks corresponding to the plurality of operators, is configured to: determine a simulation model corresponding to the operator based on the operator; run the simulation model, and determine the execution time length required by each task block of the operator based on the running time length of the simulation model; When determining the scheduling strategy of the operation unit in the operator execution device when executing the plurality of operators based on the execution time lengths of the task blocks corresponding to the plurality of operators, the parallelism between the plurality of operators, and the computing power information of the operator execution device, the generation module is configured to: construct an association relationship between the execution time lengths of the task blocks corresponding to the plurality of operators, the number of task blocks corresponding to the plurality of operators, the parallelism between the plurality of operators, the computing power information of the operator execution device, the scheduling strategy, and the total execution time length of the plurality of operators; target scheduling strategy is obtained by adjusting the scheduling strategy parameters based on the association relationship, with the goal of reducing the total execution time length.

10. A chip, characterized by including: The scheduling system of any one of claims 1-7.

11. A computer device, comprising: Comprising: The chip of claim 10.

12. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is run by a computer device, and the computer device executes the steps of the scheduling method of claim 8.

Images