Scheduler, method of operating a scheduler, and accelerator device comprising a scheduler

Abstract

Disclosed are a scheduler, a method of operating a scheduler, and an accelerator device including the scheduler. A method of operating a scheduler to perform scheduling for models to be executed in an accelerator, the method including receiving at least one execution request for first and second models to be executed independently of each other in the accelerator, and performing scheduling in units of layers for the first and second models based on workload characteristics of the first and second models.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2020-0065626, filed on June 1, 2020, with the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference for all purposes. Technical Field

[0002] The following description relates to a scheduler, methods of operating the scheduler, and accelerator devices that include the scheduler. Background Technology

[0003] With the development of artificial intelligence (AI) technology, the demand for dedicated hardware for AI execution inference and learning is increasing. Therefore, various devices are being developed as hardware specifically designed for implementing AI.

[0004] Such dedicated hardware for AI can be implemented using, for example, repurposeable central processing units (CPUs), graphics processing units (GPUs), field-programmable gate arrays (FPGAs), and application-specific integrated circuits (ASICs). Summary of the Invention

[0005] This summary is provided to introduce, in a simplified form, the selection of concepts that will be further described in the detailed embodiments below. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to help determine the scope of the claimed subject matter.

[0006] In one general aspect, a method for scheduling an accelerator is provided, the method comprising: receiving at least one execution request for a first model and a second model that are executed independently of each other in the accelerator; and performing layer-by-layer scheduling on the first model and the second model based on the workload characteristics of the first model and the second model.

[0007] The step of performing layer-by-layer scheduling may include: performing layer-by-layer scheduling on the first model and the second model independently of the order in which the at least one execution request is received.

[0008] One of the first model and the second model may have a workload characteristic where the computational cost is greater than the memory access cost, and the other of the first model and the second model may have a workload characteristic where the memory access cost is greater than the computational cost.

[0009] The step of performing layer-by-layer scheduling may include: in response to the first model and the second model having the same workload characteristics, performing layer-by-layer scheduling such that layers included in the first model and layers included in the second model are processed at time intervals between the layers included in the first model and the layers included in the second model.

[0010] The steps of performing layer-based scheduling may include: allocating two layers with different workload characteristics from each of the first and second models to the accelerator's compute resources and the accelerator's memory access resources, respectively.

[0011] The steps of performing layer-by-layer scheduling may include: in response to the first layer of the first model being allocated computing resources to the accelerator, allocating the second layer following the first layer in the first model or the third layer to be processed subsequently in the second model to the memory access resources of the accelerator.

[0012] The first layer allocated to computing resources may have different workload characteristics than the second or third layer allocated to memory access resources.

[0013] The steps of performing layer-by-layer scheduling may include: performing layer-by-layer scheduling on the first and second models based on the workload characteristics of each layer of the first and second models and the hardware resources of the accelerator.

[0014] The step of performing layer-by-layer scheduling may include: performing layer-by-layer scheduling on the first model and the second model to utilize the intermediate data of each of the first model and the second model that is reused in the accelerator’s internal memory.

[0015] The step of performing layer-by-layer scheduling may include: performing layer-by-layer scheduling on the first model and the second model to determine the best scheduling result in the current step based on the best scheduling result up to the previous step when searching for paths indicating the execution order from the input layer included in each of the first model and the second model to the output layer included in each of the first model and the second model.

[0016] The first and second models can be executed in the accelerator without having data dependencies on each other.

[0017] The hardware resources of the accelerator may include: computing resources based on at least one processing element included in the accelerator and memory access resources based on the accelerator's internal memory and / or external memory.

[0018] Workload characteristics may include: memory-intensive characteristics where the memory access cost for processing the corresponding workload is greater than the computation cost, and computation-intensive characteristics where the computation cost for processing the corresponding workload is greater than the memory access cost.

[0019] The method may include: selecting at least two models to be scheduled from the models that have received execution requests, based on the workload characteristics of each model and the hardware resources of the accelerator, wherein the step of performing layer-by-layer scheduling may include: performing layer-by-layer scheduling on the selected at least two models based on the workload characteristics.

[0020] In another general aspect, a scheduler is provided, configured to schedule the execution of models in an accelerator, the scheduler comprising: a processor configured to: receive at least one execution request for a first model and a second model that are executed independently of each other in the accelerator; and to perform layer-by-layer scheduling of the first model and the second model based on the workload characteristics of the first model and the second model.

[0021] The processor can be configured to perform layer-by-layer scheduling on the first and second models, independent of the order in which the at least one execution request is received.

[0022] One of the first model and the second model may have a workload characteristic where the computational cost is greater than the memory access cost, and the other of the first model and the second model may have a workload characteristic where the memory access cost is greater than the computational cost.

[0023] The processor can be configured to perform layer-by-layer scheduling in response to the first model and the second model having the same workload characteristics, such that layers included in the first model and layers included in the second model are processed at time intervals between the layers included in the first model and the layers included in the second model.

[0024] In another general aspect, an accelerator device is provided, the accelerator device comprising: a scheduler configured to: receive at least one execution request for a first model and a second model executed independently of each other, and to perform layer-by-layer scheduling of the first model and the second model based on the workload characteristics of the first model and the second model; and an accelerator configured to: execute the first model and the second model based on the scheduling determined by the scheduler.

[0025] In another general aspect, a processing apparatus is provided, the processing apparatus comprising: a scheduler configured to: determine, based on the respective workload characteristics of one or more models and the hardware resources of an accelerator, a layer-by-layer schedule for executing the one or more models, and send the schedule to the accelerator; and an accelerator configured to: execute the one or more models based on the schedule determined by the scheduler, wherein the one or more models are not data dependent on each other when executed in the accelerator.

[0026] The scheduler can be configured to determine scheduling based on the respective workload characteristics of each layer of the one or more models and the hardware resources of the accelerator.

[0027] The scheduling can be configured to determine the current step's scheduling based on the scheduling obtained up to the previous step.

[0028] The accelerator may include: a plurality of processing elements and a multi-level memory accessible by at least one of the plurality of processing elements.

[0029] Other features and aspects will become clear from the following detailed description, drawings, and claims. Attached Figure Description

[0030] Figure 1A and Figure 1B This is a diagram showing an example of an accelerator device.

[0031] Figure 2 This is a diagram illustrating an example of the hardware resources of an accelerator.

[0032] Figure 3 This is a diagram showing an example of a scheduler and an accelerator.

[0033] Figure 4 This is a diagram illustrating an example of how the scheduler operates.

[0034] Figure 5 This is a diagram illustrating an example of scheduling search at the layer level.

[0035] Figure 6 This is a diagram illustrating an example of a method for operating the scheduler.

[0036] Figure 7 and Figure 8 This is a diagram showing an example of an accelerator device.

[0037] Throughout the accompanying drawings and detailed embodiments, unless otherwise described or provided, the same reference numerals will be understood to denote the same elements, features, and structures. The drawings may not be to scale, and for clarity, illustration, and convenience, the relative sizes, proportions, and depictions of elements in the drawings may be exaggerated. Detailed Implementation

[0038] The following detailed embodiments are provided to aid the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to the order set forth herein, but may be changed as will become clear upon understanding this disclosure, except for operations that must occur in a specific order. Furthermore, for clarity and conciseness, descriptions of features known upon understanding this disclosure may be omitted.

[0039] The features described herein may be implemented in different forms and should not be construed as limited to the examples described herein. Rather, the examples provided herein are merely to illustrate some of the many feasible ways of implementing the methods, apparatus, and / or systems described herein, which will be clear upon understanding the disclosure of this application.

[0040] The terminology used herein is for the purpose of describing various examples only and is not intended to limit disclosure. Unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. The terms “comprising,” “including,” and “having” indicate the presence of the stated features, quantities, operations, components, elements, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, quantities, operations, components, elements, and / or combinations thereof.

[0041] Although terms such as “first,” “second,” and “third” may be used herein to describe various components, assemblies, regions, layers, or parts, these components, assemblies, regions, layers, or parts are not limited by these terms. Rather, these terms are used only to distinguish one component, assembly, region, layer, or part from another. Thus, without departing from the teaching of the examples described herein, the first component, first assembly, first region, first layer, or first part referred to as the first component, first assembly, first region, first layer, or first part may also be referred to as the second component, second assembly, second region, second layer, or second part.

[0042] Throughout this specification, when a component is described as "connected to" or "bonded to" another component, that component may be directly "connected to" or directly "bonded to" another component, or there may be one or more other components in between. In contrast, when an element is described as "directly connected to" or "directly bonded to" another element, there may not be any other element in between. Similarly, similar expressions, such as "between," "immediately between," "adjacent to," and "closely adjacent to," should be interpreted in the same manner. As used herein, the term "and / or" includes any one and any combination of any two or more of the associated listed items.

[0043] Furthermore, in the description of the exemplary embodiments, descriptions will be omitted where a detailed description of a structure or function known therefrom after understanding the disclosure of this application would lead to a vague interpretation of the exemplary embodiments. Hereinafter, the examples will be described in detail with reference to the accompanying drawings, in which the same reference numerals always denote the same elements.

[0044] Figure 1A and Figure 1B This is a diagram showing an example of an accelerator device.

[0045] Reference Figure 1A The accelerator device 100 includes a host processor 110, off-chip memory 120, memory controller 130, and accelerator 140. The host processor 110, off-chip memory 120, memory controller 130, and accelerator 140 can communicate with each other via bus 150.

[0046] For example, host processor 110 may be a means configured to control the operation of various components included in accelerator device 100, and may include a central processing unit (CPU). Host processor 110 may receive at least one request for processing a neural network in accelerator 140, and generate instructions executable in accelerator 140 in response to the received request. Requests may be made for neural network-based data inference, and requests may be made for obtaining results of data inference by allowing accelerator 140 to execute neural networks for tasks such as speech recognition, speech translation, machine translation, machine interpretation, object recognition, pattern recognition, object verification, authentication, and computer vision. Target inference data and parameters of the neural network may be transmitted from host processor 110 to accelerator 140.

[0047] Off-chip memory 120 may be memory located outside of accelerator 140. In one example, off-chip memory 120 may be dynamic random access memory (DRAM) used as the main memory of accelerator device 100. Off-chip memory 120 can be accessed via memory controller 130. Off-chip memory 120 may store parameters of a neural network to be executed in accelerator 140, and off-chip memory 120 may be used when on-chip memory in accelerator 140 is insufficient to execute the neural network in accelerator 140.

[0048] Off-chip memory 120 may have a larger memory capacity than the on-chip memory in accelerator 140. However, when executing neural networks, the cost of accessing off-chip memory 120 by accelerator 140 may be greater than the cost of accessing on-chip memory. Such memory access cost can indicate the amount of power and / or time required to access memory and then read data from or write data to memory.

[0049] Accelerator 140 may be an artificial intelligence (AI) accelerator configured to execute neural networks and infer input data according to instructions from host processor 110. In one example, accelerator 140 may be a separate processor distinct from host processor 110. Accelerator 140 may be a device such as a neural processing unit (NPU), a graphics processing unit (GPU), and a tensor processing unit (TPU).

[0050] Accelerator 140 can process workloads based on the operational characteristics of neural networks, and processing these workloads by a separate dedicated processor (e.g., accelerator 140) is more efficient than processing them by a host processor 110 for general purposes. Here, one or more processing elements (PEs) and on-chip memory included in accelerator 140 may be used. On-chip memory may be a means including global shared buffers and local buffers included in accelerator 140, and is distinct from off-chip memory 120 located outside accelerator 140. On-chip memory may include, for example, memory such as address space-accessible temporary storage and static random access memory (SRAM).

[0051] Neural networks consist of multiple layers. In one example, a neural network may include an input layer, multiple hidden layers, and an output layer. Each layer may include multiple nodes, each node being called an artificial neuron. Each node may represent a computational unit with at least one input and output, and nodes may be connected to each other. Weights can be set for the connections between nodes, and these weights can be adjusted or changed. Weights can increase, decrease, or maintain relevant data values ​​that determine the influence of data values ​​on the final result. Weighted inputs from nodes included in previous layers can be fed into each node included in the output layer. The processing of weighted data from one layer's input to subsequent layers of that layer is called propagation.

[0052] In one example, when host processor 110 receives multiple requests, accelerator 140 can execute multiple neural networks according to instructions sent from host processor 110. In this example, the neural networks to be executed in accelerator 140 may have different structures, or the same neural network may be executed multiple times. For example, when simply executing neural networks in accelerator 140 based on the order in which requests are received by host processor 110, it may not be possible to reduce the idle time during which the hardware resources of accelerator 140 are not used due to the workload characteristics of the neural networks. Furthermore, large tail latency may occur, in which later-received requests are significantly delayed while previous requests are being processed. To prevent this reduction in the utilization of accelerator 140, it may be necessary to schedule the neural networks to be executed in accelerator 140. By performing scheduling on a per-layer unit, the idle time occurring during execution can be minimized. For ease of description, the neural network described herein may also be referred to as a model.

[0053] Figure 1B An example structure of an accelerator 140 configured to execute a scheduled model is shown. The accelerator 140 may include a plurality of physical exciters (PEs) and a multi-level memory accessible by at least one of the plurality of PEs. The multi-level memory may be a common representation of level 0 (LV) memory (e.g., LV0 SRAM) 141-1, LV1 memory 142-1, and LV2 memory 143-1 corresponding to the on-chip memory of the accelerator 140.

[0054] PE 141 in PE may include LV0 memory 141-1, LV0 direct memory access (DMA) 141-3, multiplier-accumulator (MAC) 141-5 and LV0 controller 141-7.

[0055] LV0 memory 141-1 can be memory that can be accessed by the corresponding PE 141. That is, LV0 memory 141-1 can be accessed only by PE 141, which is one of a plurality of PEs included in accelerator 140.

[0056] LV0 DMA 141-3 can control the input and / or output data of LV0 memory 141-1 based on instructions from LV0 controller 141-7. LV0 DMA 141-3 can read data from or write data to LV0 memory 141-1 based on information associated with the source, destination, and data size included in the instructions from LV0 controller 141-7.

[0057] In one example, data input to or output from LV0 memory 141-1 can be monitored and / or profiled (e.g., analyzed). Such monitoring and / or profiling can be performed in LV0 DMA 141-3 or in separate components. Monitoring and / or profiling can verify the access costs of LV0 memory 141-1, usage information of LV0 memory 141-1, and the types of data stored in LV0 memory 141-1. For example, LV0 DMA 141-3 can verify what percentage of LV0 memory 141-1 usage information is indicated, and which workloads involve data stored in LV0 memory 141-1.

[0058] The MAC 141-5 can perform operations or calculations related to the workload assigned to the PE 141. For example, the MAC 141-5 can perform a multiplication-accumulation operation on given data. Furthermore, the MAC 141-5 can apply activation functions to given data. For example, the activation function can be a sigmoid function, a hyperbolic tangent function, or a linear rectified function (also known as a modified linear unit, ReLU).

[0059] The LV0 controller 141-7 can be a device configured to control components included in PE 141. For example, the LV0 controller 141-7 can control the LV0 memory 141-1, the LV0 DMA 141-3, and the MAC 141-5.

[0060] Accelerator 140 may include multiple PEs, each of which performs operations or computations independently, and the foregoing description of PE 141 may be applied to each PE included in accelerator 140.

[0061] In one example, every n PEs in a plurality of PEs can be clustered together. In this example, n is a natural number greater than 1 and less than the number of PEs included in accelerator 140. That is, a subset of the PEs included in accelerator 140 can be clustered together to form a cluster (e.g., PE cluster 142).

[0062] The PEs included in cluster 142 can share a single LV1 memory 142-1. That is, LV1 memory 142-1 can be accessed by multiple PEs included in cluster 142. For example, even if the operations performed by the first PE and the second PE among the multiple PEs in cluster 142 are different from each other, a portion of the data required for the operation can be common. Because the common data is stored in LV1 memory 142-1, rather than in LV0 memory 141-1 included in each of the first PE and the second PE, the first PE and the second PE can share common data, and the overall system operating efficiency can be improved. Figure 1B In the example, each PE can access the LV1 memory 142-1 adjacent to each PE.

[0063] Despite Figure 1B Although not shown, an LV1 DMA is present, configured to monitor and / or analyze data input to or output from the LV1 memory 142-1. Furthermore, an LV1 controller is present to control the LV1 memory 142-1 and the LV1 DMA.

[0064] Furthermore, all PEs can share LV2 memory 143-1. That is, LV2 memory 143-1 can be accessed by all PEs included in accelerator 140. For example, among the PEs included in accelerator 140, there may be PEs that, although not clustered together to form the same cluster, share a portion of the data required to perform operations. In this example, such PEs can efficiently share common data through LV2 memory 143-1 instead of sharing data through LV1 memory 142-1, thereby improving overall operational efficiency. Although in Figure 1B Although not shown, an LV2 DMA is present, configured to monitor and / or analyze data input to or output from the LV2 memory 143-1. Furthermore, an LV2 controller is present for controlling the LV2 memory 143-1 and the LV2 DMA.

[0065] As described above, each PE can access its own LV0 memory 141-1, the LV1 memory 142-1 adjacent to each PE, and the LV2 memory 143-1 of the accelerator 140, and use these memories to perform assigned workloads. The accelerator 140 may include a multi-level memory containing hierarchical memory. Furthermore, the DMA and controller included in the accelerator 140 may be of a hierarchical multi-level type.

[0066] exist Figure 1BIn the example, the PEs included in accelerator 140 can perform four types of workloads simultaneously. For example, workloads with relatively large operation volumes can be assigned to a larger number of PEs and processed therein, while workloads with relatively small operation volumes can be assigned to a smaller number of PEs and processed therein.

[0067] For ease of description, Figure 1B The diagram shows eight PEs grouped together out of 64 to form eight clusters, and three levels of memory are used to perform four workloads. However, there are no limitations on the number of PEs, workloads, and levels that can be applied.

[0068] The scheduling of the model will be described in detail below with reference to the accompanying drawings.

[0069] Figure 2 This is a diagram illustrating an example of the hardware resources of an accelerator.

[0070] exist Figure 2 In the example, accelerator 210 and off-chip memory 220 are shown.

[0071] In this example, accelerator 210 includes a global shared buffer and multiple PE arrays sharing the global shared buffer. Each PE array includes a local buffer and multiple PEs sharing the local buffer. The global shared buffer and the local buffers may be referred to as on-chip memory disposed within accelerator 210.

[0072] In order to execute the model in accelerator 210, processes such as reading data required for executing the model via memory access, performing operations or calculations in at least one PE, and storing the results of the calculations in memory can be performed. In addition to on-chip memory, the memory described herein may also include off-chip memory 220.

[0073] On-chip memory can refer to memory located within accelerator 210 and has lower access costs than off-chip memory 220. However, on-chip memory may have a smaller memory capacity than off-chip memory 220, and therefore, on-chip memory may be insufficient to store all the data used for processing operations in PE. Therefore, off-chip memory 220 can be used in such cases.

[0074] To execute the model in accelerator 210, various hardware resources can be used. For example, computing resources based on at least one PE and memory access resources based on on-chip memory and / or off-chip memory 220 can be used.

[0075] For example, computing resources can indicate the amount of computation or operations that can be processed in the PE, and can be expressed in units such as floating-point operations per second (FLOPS) and terabytes per second (TOPS). Memory access resources can indicate the on-chip network (NoC) bandwidth between PE arrays and the memory bandwidth between accelerator 210 and off-chip memory 220, and can be expressed in units such as gigabytes per second (GB / s). Furthermore, memory access resources can indicate the memory capacity of global shared buffers and local buffers, and can be expressed in units such as megabytes per second (MB).

[0076] Models and / or layers included in each model may have different workload characteristics; therefore, the computational and memory access resources required for each model or layer may differ for each model or layer. Therefore, by performing scheduling to maximize the overlap between the time used for resources in accelerator 210 based on the workload characteristics of the models and / or the workload characteristics of the layers included in each model, overall system performance can be improved.

[0077] In one example, for model scheduling, the data dependencies and availability of on-chip memory can be considered.

[0078] Data dependencies indicate the order in which a programmer or compiler intends to compute the data to achieve the desired result, and multiple layers included in a model can be processed sequentially in a preset order. However, there are no data dependencies between the multiple models to be processed in accelerator 210, so changing the processing order of the models will not have a significant impact. For example, after a layer included in the first model is processed, subsequent layers of the first model can be processed, or layers of the second model that will be processed subsequently can be processed. As previously stated, the processing order between the first and second models can be changed per layer unit.

[0079] The availability of on-chip memory can limit the processing power of accelerator 210. On-chip memory may be internal memory of accelerator 210 that is fast-accessible but may not have sufficient memory capacity to perform computations in PE. In this case, when using off-chip memory 220 corresponding to the external memory of accelerator 210, the memory access time of off-chip memory is greater than that of on-chip memory, so the memory access time of off-chip memory can be considered when performing scheduling. That is, the method of reusing intermediate data for each model in the on-chip memory of accelerator 210 can also affect memory access costs, and therefore can also be considered for scheduling.

[0080] Figure 3 This is a diagram showing an example of a scheduler and an accelerator.

[0081] Reference Figure 3Scheduler 310 is configured to schedule the models to be executed in accelerator 320. For example, scheduler 310 may be implemented as follows: Figure 1A The host processor 110, or it can be implemented on a separate device.

[0082] Requests from multiple users or a single user can be allocated by a request allocator based on workload characteristics. For example, the request allocator can assign each request to either a memory-intensive characteristic or a compute-intensive characteristic, where the memory access cost of the memory-intensive characteristic is greater than the compute cost, and the compute-intensive characteristic has a compute cost greater than the memory access cost. Such allocated requests can be stored in request queues. In other words, user requests sent to the accelerator device can be analyzed based on their characteristics and then stored in different queues.

[0083] Scheduler 310 can perform layer-unit scheduling on multiple requested models based on workload characteristics. Here, because there are no data dependencies between models, scheduler 310 can perform layer-unit scheduling on models independently of the order in which requests are received.

[0084] In one example, scheduler 310 can perform scheduling on models with different workload characteristics. Scheduler 310 can perform scheduling on models with computationally intensive characteristics and models with memory-intensive characteristics. For example, models with computationally intensive characteristics may include convolutional neural networks (CNNs), etc. Models with memory-intensive characteristics may include, for example, multilayer perceptrons, bidirectional encoder representations from transformers (BERT), recommendation models, etc. By performing scheduling on models with higher computational costs and models with higher memory access costs, idle time in the hardware resources of accelerator 320 can be effectively prevented and overall system throughput can be improved.

[0085] Furthermore, scheduler 310 can also perform scheduling for models with similar characteristics. For example, when multiple models with similar characteristics (also referred to herein as identical models) are executed in accelerator 320, scheduling can be performed such that layers included in each model are processed at intervals or predetermined time differences between layers. For example, among the multiple layers included in a model, there may be layers with computationally intensive characteristics and layers with memory-intensive characteristics. Therefore, even when multiple models with similar characteristics are executed together, the utilization of accelerator 320 can be improved by using the difference between computational cost and memory access cost based on the workload characteristics of each layer of the model.

[0086] In one example, scheduler 310 may select at least two models from among multiple models that have received execution requests for them, based on the workload characteristics of each model and the hardware resources of accelerator 320, to perform scheduling. For example, scheduler 310 may perform scheduling by selecting models with different workload characteristics from multiple models or by selecting models with the same characteristics from multiple models.

[0087] The schedule determined by scheduler 310 can be recorded in the form of a table, and the allocator can control the hardware resources of accelerator 320 through accelerator drivers to process memory accesses and computations corresponding to each layer at a certain time. In another example, the schedule determined by scheduler 310 can be passed to accelerator 320 and executed directly in accelerator 320.

[0088] Accelerator 320 can execute the model according to the schedule determined by scheduler 310. Here, model parameters stored in the model repository and input data stored in the input storage device can be transmitted to accelerator 320.

[0089] Figure 4 This is a diagram illustrating an example of how the scheduler operates.

[0090] Figure 4 Examples of scheduling models with different workload characteristics are shown. Figure 4 In the example, it is assumed that the first model has workload characteristics where the computational cost is greater than the memory access cost, and the second model has workload characteristics where the memory access cost is greater than the computational cost. The first model may include multiple layers (e.g., L11, L12, L13, etc.). Similarly, the second model may also include multiple layers (e.g., L21, L22, L23, etc.).

[0091] To perform computations on computing resources, it is necessary to first read the data intended for the computation from memory access resources. Furthermore, memory access resources and computing resources can operate in parallel; therefore, while computation is being performed on computing resources, data for the next computation can be pre-fetched from memory access resources. When there is no idle time or a minimum idle time between memory access resources and computing resources, the utilization of accelerator devices can be improved, and higher levels of performance can be expected.

[0092] For example, such as Figure 4As shown, layer L11, as the first layer of the first model with lower memory access costs, can be allocated to memory access resources, and then memory access operations based on layer L11 can be executed. When a memory access operation is terminated, layer L11 can be allocated to computation resources, and then computation operations based on layer L11 can be executed. Furthermore, layer L21, as the first layer of the second model, can be allocated to memory access resources, and memory access operations based on layer L21 can be executed. When a memory access operation of layer L21 terminates while a computation operation of layer L11 is being executed, layer L12, as the second layer of the first model, can subsequently be allocated to memory access resources. Therefore, idle time in memory access resources can be prevented. Layer-based scheduling can be performed in such a manner until the last layer of each of the first and second models.

[0093] As described above, the utilization of each resource differs for each layer. Therefore, two layers with different workload characteristics, included in multiple layers in each of the first and second models, can be allocated to the accelerator's compute resources and memory access resources, respectively. For example, while a layer included in the first model (e.g., layer n of the first model) is being allocated to the accelerator's compute resources, the scheduler can allocate subsequent layers of the first model (e.g., layer n+1 of the first model) or layers of the second model to be processed subsequently (e.g., layer m of the second model) to the accelerator's memory access resources, where n and m are positive integers. In this example, the layer of the first model to be allocated compute resources may have different workload characteristics than the subsequent layers of the first model and the layers of the second model to be allocated memory access resources.

[0094] As described above, by performing layer-by-layer scheduling on the first and second models based on the workload characteristics of each layer of the first and second models and the hardware resources of the accelerator, idle time in each resource can be prevented and utilization improved. The scheduler can perform scheduling to change the execution order based on the layer level between independent models or based on the corresponding computing units (e.g., residual blocks, starter modules, etc.).

[0095] Figure 5 This is a diagram illustrating an example of scheduling search at the layer level.

[0096] Figure 5 An example of layer-by-layer scheduling search between compute-intensive model 510 and memory-intensive model 520 is shown. Figure 5 In the examples, for ease of description, it is assumed that the computationally intensive model 510 includes p layers and the memory-intensive model 520 includes q layers, where p and q are natural numbers greater than 0.

[0097] In one example, the scheduler can perform layer-by-layer scheduling based on dynamic programming for optimal schedule search. In the process of searching for paths indicating the execution order from the input layer (or first layer) included in each model to the output layer (or last layer) included in each model, the scheduler can progressively determine the optimal schedule in the current step based on the optimal schedule obtained up to the previous step.

[0098] exist Figure 5 In the example, from <0,0> at the top left to <0,0> at the bottom right<p,q> The path can indicate the execution order of the layers included in each model. As mentioned above, it can be based on the previous step.<i-1,j> and<i,j-1> The optimal scheduling result is used to gradually determine the current step.<j,j> The optimal scheduling result in C. i M can indicate the computational cost of the i-th layer of the computationally intensive model 510. j This can indicate the memory access cost of the j-th layer of the memory-intensive model 520. Here, each cost can be represented by the number of loops. As the number of loops decreases, the computation speed can be improved.

[0099] In order to schedule and calculate or measure the current step<i,j> The cost of this process may need to take into account the accelerator's on-chip memory usage and data dependencies between layers. Based on the layer structure, the parameters or output data of a neural network stored in on-chip memory can be reused in subsequent layers or moved to off-chip memory. Here, each operation may come with a cost (e.g., number of loops, energy, etc.), thus the operation speed can be determined differently based on the total cost required for each path. In this way, on-chip memory can be managed by the compiler, runtime hardware, or a separate hardware architecture, or a scheduler can take this into account and be implemented to search for more desirable schedules.

[0100] For example, using until the previous step<i-1, j> The optimal scheduling result obtained and the result obtained from the previous step<i-1, j> Up to the current step<i,j> The simulation results, obtained by adding a layer in the direction of the current path, can update the path and determine the number of loops at this point. Furthermore, the simulation can be performed up to the previous step.<i, j-1> The optimal scheduling result obtained and the result obtained from the previous step<i, j-1> Up to the current step<i,j> The simulation results, obtained by adding a layer in the direction of the current step, can update the path and determine the number of loops at this point.<i,j> In this process, the two schedules and the number of loops obtained from the previous step can all be recorded, and these can be used in subsequent search steps. In the final step...<p,q> In this process, the path with the fewest number of loops can be selected from the path updated in the manner described above.

[0101] As described above, by reusing the steps up to the end<i-1, j> and<i, j-1> The optimal scheduling result of the search determines the current step.<i, j> The optimal scheduling result minimizes the search cost. This is achieved by connecting <0,0> to the nearest node with the minimum number of iterations.<p,q> The path determination is the optimal schedule, which can determine the high-quality layer execution order within a finite time.

[0102] Figure 6 This is a diagram illustrating an example of a method for operating the scheduler. Figure 6 The operations can be performed in the order and manner shown, although the order of some operations may be changed or some operations may be omitted without departing from the spirit and scope of the illustrative examples described. Figure 6 Many of the operations shown can be performed in parallel or simultaneously. Figure 6 One or more boxes, and combinations thereof, may be implemented by a computer (such as a processor) based on dedicated hardware that performs the specified function, or by a combination of dedicated hardware and computer instructions. In addition to the following... Figure 6 In addition to the description, Figures 1A to 5 The description also applies to Figure 6 And it is included here by reference. Therefore, the above description need not be repeated here.

[0103] Figure 6 This demonstrates the method for operating the scheduler.

[0104] Reference Figure 6 In operation 610, the scheduler receives at least one execution request for a first model and a second model that are executed independently of each other in the accelerator. In operation 620, the scheduler performs layer-by-layer scheduling on the first model and the second model based on the workload characteristics of the first model and the second model.

[0105] The scheduler can be comprehensively applied to product lines that utilize multiple System-on-a-Chip (SoC) architectures to accelerate AI processing, with each SoC comprising multiple Processing Objects (PEs) and shared memory. Furthermore, when accelerators handle a wide range of workloads, the scheduler can be applied to appropriately allocate workloads to hardware resources.

[0106] Furthermore, the scheduler can categorize deep learning model workloads into compute-intensive and memory-intensive characteristics for each model or layer, and dynamically allocate workloads to resources based on accelerator hardware resources. This maximizes accelerator utilization. Additionally, by decoupling access and computation, the scheduler can hide memory access time and improve the efficiency of computational resources.

[0107] Figure 7 and Figure 8This is a diagram showing an example of an accelerator device.

[0108] Reference Figure 7 The accelerator device can be implemented as a server 700.

[0109] Server 700 may represent a separate device distinct from user terminals controlled by users, and may communicate with one or more user terminals via wired and / or wireless networks. Server 700 may receive requests simultaneously sent from multiple users through their user terminals. Through scheduler 710, server 700 may schedule multiple models to be executed in accelerator 720. Accelerator 720 may execute models based on the schedule and determine inference results. Server 700 may then return the inference results to the respective corresponding user terminals. User terminals described herein may include, for example, computing devices (such as smartphones, personal computers (PCs), tablet PCs, desktop computers, personal digital assistants (PDAs), set-top boxes, and laptop computers), wearable devices (such as smartwatches and smart glasses), home appliances (such as smart speakers, smart TVs, and smart refrigerators), and other devices (such as smart vehicles, intelligent vehicles, gaming devices, smart self-service terminals, smart home systems, and Internet of Things (IoT) devices).

[0110] Reference Figure 8 The accelerator device can be implemented as a user terminal 800 including a scheduler 810 and an accelerator 820. Although for ease of description, the user terminal 800 is... Figure 8 The device is shown as a smartphone, but any user-controlled device may be applicable without limitation. User terminal 800 may receive requests directly from the user and schedule the model to be executed in accelerator 820 via scheduler 810. Accelerator 820 may execute the model based on the schedule and determine the results (such as inference, translation, and recognition).

[0111] In this regard Figures 1A to 1B , Figures 2 to 4 as well as Figures 7 to 8The schedulers, accelerators, accelerator devices 100, 140, 310, 320, 710, 720, 810, and 820, as well as other devices, units, modules, apparatuses, and components described herein, are implemented by hardware components. Examples of hardware components that can be used to perform the operations described herein include, where appropriate, controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described herein. In other examples, one or more of the hardware components performing the operations described herein are implemented by computing hardware (e.g., by one or more processors or computers). Processors or computers can be implemented by one or more processing elements, such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field-programmable gate arrays, programmable logic arrays, microprocessors, or any other means or combination of means configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, the processor or computer includes or is connected to one or more memories storing instructions or software executed by the processor or computer. Hardware components implemented by the processor or computer can execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) for performing the operations described herein. The hardware components can also access, manipulate, process, create, and store data in response to the execution of the instructions or software. For simplicity, the singular terms "processor" or "computer" are used in the description of the examples described herein; however, in other examples, multiple processors or computers may be used, or a processor or computer may include multiple processing elements or multiple types of processing elements or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or additional processors and additional controllers. One or more processors, or a processor and a controller, may implement a single hardware component or two or more hardware components.The hardware components may be any one or more with different processing configurations. Examples of different processing configurations include: a single processor, a standalone processor, a parallel processor, a single instruction single data (SISD) multiprocessor, a single instruction multiple data (SIMD) multiprocessor, multiple instruction single data (MISD) multiprocessor, multiple instruction multiple data (MIMD) multiprocessor, a controller and arithmetic logic unit (ALU), a DSP, a microcomputer, an FPGA, a programmable logic unit (PLU), a central processing unit (CPU), a graphics processing unit (GPU), a neural processor (NPU), or any other device capable of responding to and executing instructions in a defined manner.

[0112] Figures 1A to 6 The methods for performing the operations described in this application, as shown, are executed by computing hardware (e.g., by one or more processors or a computer), which is implemented to execute instructions or software as described above to perform the operations performed by the methods described in this application. For example, a single operation or two or more operations may be executed by a single processor or two or more processors, or a processor and a controller. One or more operations may be executed by one or more processors, or a processor and a controller, and one or more other operations may be executed by one or more other processors, or additional processors and additional controllers. One or more processors, or a processor and a controller, may execute a single operation or two or more operations.

[0113] Instructions or software for controlling a processor or computer to implement hardware components and perform the methods described above are written as computer programs, code segments, instructions, or any combination thereof to individually or collectively instruct or configure the processor or computer, such as a machine or special-purpose computer, to perform the operations performed by the hardware components and methods described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) that is directly executed by the processor or computer. In one example, the instructions or software include at least one of an applet, a dynamic link library (DLL), middleware, firmware, a device driver, or an application of a method for scheduling a model to be executed in an accelerator. In another example, the instructions or software include high-level code that is executed by the processor or computer using an interpreter. Instructions or software can be readily written by a programmer of ordinary skill in the art based on the block diagrams and flowcharts shown in the accompanying drawings and the corresponding descriptions in the specification, which disclose algorithms for performing the operations performed by the hardware components and methods described above.

[0114] Instructions or software used to control a processor or computer to implement hardware components and perform the methods described above, as well as any associated data, data files, and data structures, may be recorded, stored, or fixed in, or on, one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+. R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-RLTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card-type storage (such as multimedia cards or microcards (e.g., Secure Digital (SD) or Extreme Digital (XD))), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state disk, and any other device configured to store instructions or software and any associated data, data files, and data structures in a non-transitory manner and to provide said instructions or software and any associated data, data files, and data structures to a processor or computer, such that the processor or computer can execute the instructions.

[0115] While this disclosure includes specific examples, it will be clear upon understanding this disclosure that various changes in form and detail may be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are to be considered descriptive only and not for limiting purposes. The description of features or aspects in each example will be considered applicable to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order, and / or if components in the described system, architecture, apparatus, or circuit are combined in a different manner, and / or replaced or supplemented by other components or their equivalents.

[0116] Therefore, the scope of the disclosure is not limited by the specific embodiments, but by the claims and their equivalents, and all variations within the scope of the claims and their equivalents should be interpreted as included in the disclosure.

Claims

1. A method for scheduling an accelerator, the method comprising: Receive at least one execution request for a first model and a second model that are executed independently of each other in the accelerator; as well as Based on the workload characteristics of the first and second models, scheduling is performed on a layer-by-layer basis for both models. The step of performing layer-by-layer scheduling includes: performing layer-by-layer scheduling on the first and second models by determining the optimal scheduling result in the current step based on the optimal scheduling result up to the previous step during the search path processing, wherein the path indicates the execution order from the input layer included in each of the first and second models to the output layer included in each of the first and second models, and The step of determining the optimal scheduling result in the current step is based on the results of simulations performed using the optimal scheduling results up to the previous step and by adding a layer in the direction from the previous step to the current step.

2. The method according to claim 1, wherein, The steps for performing scheduling at the layer level include: Independent of the order in which the at least one execution request is received, scheduling is performed on a layer-by-layer basis for the first and second models.

3. The method according to claim 1, wherein, One of the first and second models has a workload characteristic where the computational cost is greater than the memory access cost, and the other of the first and second models has a workload characteristic where the memory access cost is greater than the computational cost.

4. The method according to claim 1, wherein, The steps for performing scheduling at the layer level include: In response to the first model and the second model having the same workload characteristics, scheduling is performed on a layer-by-layer basis, such that layers included in the first model and layers included in the second model are processed with a predetermined time difference or with a time interval between layers included in the first model and layers included in the second model.

5. The method according to any one of claims 1 to 4, wherein, The steps for performing scheduling at the layer level include: In each of the first and second models, two layers with different workload characteristics are assigned to the accelerator's computing resources and memory access resources, respectively.

6. The method according to any one of claims 1 to 4, wherein, The steps for performing scheduling at the layer level include: In response to the nth layer of the first model being allocated computing resources to the accelerator, the (n+1)th layer after the nth layer in the first model or the mth layer to be processed in the second model is allocated memory access resources to the accelerator, where n and m are positive integers.

7. The method according to claim 6, wherein, The nth layer of computing resources allocated to the accelerator has different workload characteristics than the (n+1)th or mth layer of memory access resources allocated to the accelerator.

8. The method according to any one of claims 1 to 4, wherein, The steps for performing scheduling at the layer level include: Based on the workload characteristics of each layer of the first and second models and the hardware resources of the accelerator, scheduling is performed on a layer-by-layer basis for the first and second models.

9. The method according to any one of claims 1 to 4, wherein, The steps for performing scheduling at the layer level include: By using the intermediate data of each of the first and second models reused in the accelerator's internal memory, layer-by-layer scheduling is performed on the first and second models.

10. The method according to any one of claims 1 to 4, wherein, The first and second models do not have data dependencies on each other when they are executed in the accelerator.

11. The method according to any one of claims 1 to 4, wherein, The hardware resources of the accelerator include: computing resources based on at least one processing element included in the accelerator and memory access resources based on the accelerator's internal and / or external memory.

12. The method according to any one of claims 1 to 4, wherein, Workload characteristics include: memory-intensive characteristics where the memory access cost for processing the corresponding workload is greater than the computation cost, and computation-intensive characteristics where the computation cost for processing the corresponding workload is greater than the memory access cost.

13. The method according to any one of claims 1 to 4, further comprising: Based on the workload characteristics of each model and the hardware resources of the accelerator, at least two models are selected from those receiving execution requests to be scheduled for execution. The steps for performing scheduling at the layer level include: Scheduling is performed on a layer-by-layer basis for the selected at least two models based on workload characteristics.

14. A non-transitory computer-readable storage medium for storing commands, which, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 13.

15. A scheduler configured to schedule the execution of a model in an accelerator, the scheduler comprising: The processor is configured as follows: Receive execution requests for a first model and a second model that are executed independently of each other in the accelerator; as well as Based on the workload characteristics of the first and second models, scheduling is performed on a layer-by-layer basis for both models. The layer-by-layer scheduling process includes: performing layer-by-layer scheduling on the first and second models by determining the optimal scheduling result in the current step based on the optimal scheduling result up to the previous step during the path search process, wherein the path indicates the execution order from the input layer included in each of the first and second models to the output layer included in each of the first and second models. The process of determining the optimal scheduling result in the current step is based on the results of simulations performed using the optimal scheduling results up to the previous step and by adding a layer in the direction from the previous step to the current step.

16. The scheduler according to claim 15, wherein, The processor is also configured as follows: Scheduling is performed on a layer-by-layer basis for the first and second models, independent of the order in which the execution requests are received.

17. The scheduler according to claim 15 or claim 16, wherein, One of the first and second models has a workload characteristic where the computational cost is greater than the memory access cost, and the other of the first and second models has a workload characteristic where the memory access cost is greater than the computational cost.

18. The scheduler according to claim 15 or claim 16, wherein, The processor is also configured as follows: In response to the first model and the second model having the same workload characteristics, scheduling is performed on a layer-by-layer basis, such that layers included in the first model and layers included in the second model are processed with a predetermined time difference or with a time interval between layers included in the first model and layers included in the second model.

19. An accelerator device, comprising: The scheduler is configured to: receive at least one execution request for a first model and a second model that are executed independently of each other, and perform layer-by-layer scheduling on the first model and the second model based on the workload characteristics of the first model and the second model; as well as The accelerator is configured to execute the first and second models based on a schedule determined by the scheduler. The layer-by-layer scheduling process includes: performing layer-by-layer scheduling on the first and second models by determining the optimal scheduling result in the current step based on the optimal scheduling result up to the previous step during the path search process, wherein the path indicates the execution order from the input layer included in each of the first and second models to the output layer included in each of the first and second models. The process of determining the optimal scheduling result in the current step is based on the results of simulations performed using the optimal scheduling results up to the previous step and by adding a layer in the direction from the previous step to the current step.

20. A processing apparatus, comprising: The scheduler is configured to: determine layer-by-layer scheduling for executing the one or more models based on their respective workload characteristics and the hardware resources of the accelerator, and send the scheduling to the accelerator; and The accelerator is configured to execute one or more models based on a schedule determined by the scheduler. Wherein, the one or more models are executed in the accelerator without any data dependencies on each other. The process of determining the layer-by-layer scheduling for executing the one or more models includes: determining the layer-by-layer scheduling for executing the one or more models by determining the optimal scheduling result in the current step based on the optimal scheduling result up to the previous step in the path search process, wherein the path indicates the execution order from the input layer included in each of the one or more models to the output layer included in each of the one or more models, and The process of determining the optimal scheduling result in the current step is based on the results of simulations performed using the optimal scheduling results up to the previous step and by adding a layer in the direction from the previous step to the current step.

21. The processing apparatus according to claim 20, wherein, The scheduler is also configured to determine scheduling based on the respective workload characteristics of each layer of the one or more models and the hardware resources of the accelerator.

22. The processing apparatus according to claim 20 or claim 21, wherein, The scheduler is also configured to determine the schedule for the current step based on the schedule obtained up to the previous step.

23. The processing apparatus according to claim 20 or claim 21, wherein, The accelerator includes: Multiple processing elements and a multi-level memory accessible by at least one of the multiple processing elements.

Images