Data processing method, data processing device, and electronic device including data processing device

Abstract

To provide a data processing method, a data processing device and an electronic device including the data processing device, and an accelerator system.SOLUTION: A data processing method includes steps of: receiving a request to execute a model of a neural network by an accelerator; generating a plurality of candidate kernels with respect to each of a plurality of layers contained in the model; and allocating either one candidate kernel, selected based on corresponding kernel information and state information of the accelerator out of the plurality of candidate kernels with respect to the layer executed by the accelerator, to the accelerator.SELECTED DRAWING: Figure 1

Description

Technical Field

[0004] , , , , ,

[0005] ,

[0001] The following embodiments relate to a data processing method, a data processing apparatus, and an electronic apparatus including the data processing apparatus.

Background Art

[0002] As artificial intelligence (AI) technology develops, the need for dedicated hardware for AI alone is increasing. Artificial intelligence can perform inference and learning through specific operations, for example. Thus, various devices have been developed as dedicated hardware for realizing and executing artificial intelligence.

[0003] The dedicated hardware for artificial intelligence may be realized by, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc., or may be realized by an FPGA (Field Programmable Gate Array) and an ASIC (Application Specific Integrated Circuit) whose use can be changed.

Summary of the Invention

Problems to be Solved by the Invention

[0004] An object of the embodiments is to provide a data processing method, a data processing apparatus, and an electronic apparatus including the data processing apparatus that can efficiently execute a plurality of neural networks on a multi-core accelerator.

Means for Solving the Problems

[0005] <00A data processing method performed by one or more processors according to one embodiment includes the steps of: receiving a request to run a neural network model in an accelerator; generating a plurality of candidate kernels for each of a plurality of layers included in the model; and assigning one of the plurality of candidate kernels for the layer to be run in the accelerator, selected based on the corresponding kernel information and accelerator state information.

[0006] In a data processing method according to one embodiment, the generation step can generate multiple candidate kernels that execute each layer with different accelerator resource usage.

[0007] In a data processing method according to one embodiment, the generation step can generate multiple candidate kernels in which the corresponding layer differs in one or more of the following: the number of cores used by the accelerator during execution, the data reuse pattern, and the tailing factor.

[0008] In a data processing method according to one embodiment, the number of cores among the multiple cores included in the accelerator that execute the model can be varied at runtime according to the resource status of the accelerator.

[0009] In a data processing method according to one embodiment, the allocation step allows selecting one of several candidate kernels that can maximize the accelerator utilization rate based on the accelerator's resource status and allocating it to the accelerator's remaining resources.

[0010] In a data processing method according to one embodiment, the accelerator state information may include one or more of the following: accelerator resource usage information, kernel information of the layer currently running in the accelerator, and information about the subsequent layers of the layer currently running in the accelerator.

[0011] In a data processing method according to one embodiment, the assignment step can select a candidate kernel and assign it to the accelerator based on whether or not a collision occurs with the data access pattern of the layer currently being executed in the accelerator.

[0012] In a data processing method according to one embodiment, the assignment step, in the event of a collision, selects from among multiple candidate kernels a candidate kernel that satisfies the QoS (Quality of Service) of the currently executing layer and the layer scheduled to be executed, or a candidate kernel that has the least impact on QoS, and assigns the selected candidate kernel to the accelerator.

[0013] In a data processing method according to one embodiment, the allocation step may involve allocating a candidate kernel in response to the completion of any one of the one or more layers currently running in the accelerator, or allocating a candidate kernel in response to whether the remaining resources of the accelerator are available to run the model according to the new request when a new request is received.

[0014] In a data processing method according to one embodiment, the assignment step can be performed on a host processor that generates multiple candidate kernels or on an accelerator that runs a model.

[0015] In a data processing method according to one embodiment, the assignment step can assign one of several candidate kernels to the accelerator based on one of the following: a greedy method, a resource equal distribution method, a total system performance priority method, or a power consumption minimization method.

[0016] In a data processing method according to one embodiment, the accelerator includes multiple cores and can execute one or more models.

[0017] A data processing method according to one embodiment may further include the step of generating at least a portion of the output feature map by using an accelerator to run layers of the model based on assigned candidate kernels.

[0018] A data processing device according to one embodiment includes one or more processors, which receive a request to run a neural network model in an accelerator, generate multiple candidate kernels for each of the multiple layers included in the model, and assign one of the multiple candidate kernels selected from the multiple candidate kernels for the layer to be run in the accelerator, based on the corresponding kernel information and the accelerator state information, to the accelerator.

[0019] An electronic device according to one embodiment includes a host processor that receives a request to execute a neural network model in an accelerator and generates multiple candidate kernels for each of the multiple layers included in the model, and an accelerator that executes a layer based on one of the multiple candidate kernels selected from the multiple candidate kernels for the layer to be executed, based on the corresponding kernel information and the state information of the accelerator.

[0020] An accelerator system according to one embodiment includes a host processor that generates a plurality of candidate kernels for each layer of a model that is requested to be executed, and an accelerator that executes a layer based on one of the plurality of candidate kernels for the layer to be executed, selected based on the corresponding kernel information and accelerator state information.

[0021] In an accelerator system according to one embodiment, the number of cores among the multiple cores included in the accelerator that execute layers can be varied at runtime according to the resource status of the accelerator.

[0022] A data processing method performed by a processor according to one embodiment includes the steps of: generating a plurality of kernels for a first neural network model and a second neural network model, respectively; executing the kernels of the first neural network model based on the number of cores of the accelerator's available resources; executing the kernels of the second neural network model in response to the execution, based on the remaining cores of the accelerator's available resources; and generating one or more output feature maps for the execution of the kernels.

[0023] In a data processing method according to one embodiment, the step of executing the kernel of the second neural network model can be performed in response to the determination that no collisions occur between memory access patterns between the execution of the kernel of the first neural network model and the execution of the kernel of the second neural network model.

[0024] In a data processing method according to one embodiment, for each kernel of a first neural network model and a kernel of a second neural network model, the kernel can be selected to be executed from among a plurality of kernels based on kernel information that includes the number of cores of the accelerator used to execute the kernel, a data reuse pattern that includes one or more of the following when the kernel is executed: reuse of input feature maps, reuse of weighted values, and reuse of output feature maps, and a tailing factor that includes the size of the multidimensional data of the kernel. [Effects of the Invention]

[0025] According to this embodiment, when running multiple models on an NPU (neural processing unit), the accelerator performance can be improved by minimizing contention between models while maximizing the utilization rate of NPU resources. [Brief explanation of the drawing]

[0026] [Figure 1] It is a diagram for explaining an electronic device according to an embodiment. [Figure 2] It is a diagram for explaining a resource allocation process for executing a multi-model on a multi-core accelerator according to an embodiment. [Figure 3] It is a diagram for explaining kernel information according to an embodiment. [Figure 4] It is a diagram for explaining a process of selecting any one of a plurality of candidate kernels according to an embodiment. [Figure 5] It is a diagram for explaining a process of selecting any one of a plurality of candidate kernels according to an embodiment. [Figure 6] It is a diagram for explaining a process of selecting any one of a plurality of candidate kernels according to an embodiment. [Figure 7] It is a diagram showing an example of resource allocation according to an embodiment. [Figure 8] It is a diagram showing an example of resource allocation according to an embodiment. [Figure 9] It is a diagram for explaining resource allocation considering a memory access pattern according to an embodiment. <00001​​​​​​​​​​​​​​​​​​​​The specific structural or functional descriptions disclosed herein are illustrative for the purpose of illustrating embodiments, and embodiments can be carried out in various different forms. The present invention is not limited to the embodiments described herein, and the scope of the present invention includes modifications, equivalents, or substitutions that are included in the technical ideas described in the embodiments.

[0028] Terms such as "first" or "second" may be used to describe multiple components, but such terms should be interpreted solely for the purpose of distinguishing one component from others. For example, the first component can be named the second component, and similarly, the second component can also be named the first component.

[0029] When it is said that one component is “linked” or “connected” to another component, it should be understood that it is directly linked to or connected to the other component, but that other components may be present in between.

[0030] A singular expression includes plural expressions unless the context clearly indicates otherwise. In this specification, terms such as “includes” or “has” indicate the presence of features, figures, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood not to preemptively exclude the possibility of the presence or addition of one or more other features, figures, steps, actions, components, parts, or combinations thereof.

[0031] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as those generally understood by a person of ordinary skill in the art to which this embodiment belongs. Commonly used, predefined terms should be interpreted as having the meaning consistent with their meaning in the context of the relevant art, and should not be interpreted in an ideal or overly formal sense unless expressly defined herein.

[0032] The embodiments will be described in detail below with reference to the attached drawings. In describing with reference to the attached drawings, the same reference numerals will be used for the same components regardless of the reference numerals in the drawings, and redundant explanations will be omitted.

[0033] Figure 1 is a diagram illustrating an electronic device according to one embodiment.

[0034] Referring to Figure 1, an electronic device 100 according to one embodiment includes a host processor 110 and an accelerator 120. The host processor 110 and the accelerator 120 communicate via a bus, NoC (Network on a Chip), PCIe (Peripheral Component Interconnect Express), etc.

[0035] The host processor 110 is a device that controls the operation of components included in the electronic device 100, and may include, for example, a central processing unit (CPU). The host processor 110 receives one or more requests for processing a neural network in the accelerator 120, and in response to the requests, generates a kernel containing instructions that can be executed in the accelerator 120. The requests are for data inference based on the neural network, for example, to cause the accelerator 120 to execute the neural network and obtain the results of data inference for object recognition, pattern recognition, computer vision, speech recognition, machine translation, machine interpretation, recommendation services, personalized services, image processing, autonomous driving, etc. The requests may also be for training the neural network, to train a neural network that performs the operations described above based on training data. In this specification, for the sake of convenience of explanation, data inference has been described as the basis, but such explanations may be applied similarly without excluding data learning. The host processor 110 can transmit the data to be inferred and / or the kernel for executing the neural network to the accelerator 120.

[0036] The accelerator 120 is an AI accelerator (Artificial Intelligence accelerator) that executes a neural network and infers data based on a kernel transmitted from the host processor 110, and may be a separate processor distinct from the host processor 110. For example, the accelerator 120 may be an NPU (Neural Processing Unit), GPU, TPU (Tensor Processing Unit), DSP (Digital Signal Processor), etc.

[0037] Due to the characteristics of computations performed by neural networks, the accelerator 120 can handle tasks more efficiently when processed by a separate dedicated processor (in other words, the accelerator 120) than when processed by a general-purpose host processor 110. Here, the accelerator core 131 and on-chip memory 132 included in the accelerator chip 130 may be utilized.

[0038] The accelerator core 131 may include one or more processing elements (PEs) that perform neural network calculations. In Figure 1, for convenience of explanation, the accelerator chip 130 is shown to contain one accelerator core 131, but depending on the embodiment, multiple accelerator cores may be included in the accelerator chip 130 to process one or more required neural network calculations. In this specification, the accelerator core 131 may be referred to as a core for convenience of explanation.

[0039] The on-chip memory 132 is a device included within the accelerator chip 130 that stores data necessary for the execution of calculations by the accelerator core 131 and stores the results of those calculations, and is separate from the off-chip memory 140 located outside the accelerator chip 131. For example, the on-chip memory 132 may include scratchpad memory, SRAM (Static Random Access Memory), etc., that can be accessed via the address space.

[0040] The off-chip memory 140 is a memory located outside the accelerator chip 130 and may be, for example, DRAM (Dynamic Random Access Memory) or HBM (High Bandwidth Memory), but is not limited to these. The off-chip memory 140 stores the data to be inferred and / or the parameters of the neural network to be executed on the accelerator core 131, and the stored data may then be transmitted to the accelerator core 131 for inference execution. The off-chip memory 140 may also be utilized when the on-chip memory 132 within the accelerator chip 130 is insufficient to execute the neural network on the accelerator core 131.

[0041] The off-chip memory 140 has a larger memory capacity than the on-chip memory 132 within the accelerator core 131, but the cost for the accelerator core 131 to access the off-chip memory 140 when running a neural network is greater than the cost for accessing the internal on-chip memory 132. The memory access cost represents the power and / or time required when accessing the memory to read or write data.

[0042] Neural networks can provide the optimal output corresponding to an input by mapping non-linearly related inputs and outputs to each other based on deep learning. Deep learning is a machine learning method for solving problems given from big datasets, and it is a process of optimizing the neural network by finding a model or parameters (e.g., weights) that describe the structure of the neural network. A neural network includes multiple layers (e.g., an input layer, multiple hidden layers, and an output layer). Each layer includes multiple nodes, also called artificial neurons. Each node represents a computation unit with one or more inputs and outputs, and the nodes may be interconnected. Weights are assigned to the connections between nodes, and such weights may be adjusted or changed. The weights determine the degree to which the relevant data values ​​have an influence on the final result by amplifying, decreasing, or retaining the relevant data values. Each node in the output layer may be input with multiple weighted inputs from nodes in the previous layer. The process by which weighted data is input from any layer to the next can be called propagation.

[0043] In one embodiment, if the host processor 110 receives multiple requests, the accelerator 120 can execute multiple neural networks using kernels generated by the host processor 110. Here, the multiple neural networks executed by the accelerator 120 may be neural networks with different structures, or the same neural network may be executed several times. If multiple neural networks are simply executed in the order of the requests, it will be difficult to reduce idle time during which the hardware resources of the accelerator 120 are not used, due to the characteristics of the workload of each neural network, and tail latency may occur, where later received requests are delayed while earlier requests are being processed. To prevent such a decrease in the utilization rate of the accelerator 120, scheduling for multiple neural networks can be performed by the electronic device 100 and / or the host processor 110. In particular, when execution on multiple neural networks is requested, the utilization rate of the accelerator 120 is improved by the electronic device 100 and / or host processor 110 by minimizing idle time that occurs during execution by allocating resources to process each neural network according to the status of the accelerator 120. In this specification, a neural network may be referred to as a model for convenience of explanation.

[0044] The embodiments described herein generate several candidate kernels for running a specific layer on the accelerator 120, and select one of the candidate kernels that can most efficiently utilize the resources of the accelerator 120 to run. This allows for the optimization of performance by minimizing contention between models while maximizing resource utilization of the accelerator 120, even when multiple models are running on the accelerator 120. Although the utilization of the accelerator 120 may decrease when multiple models are driven simultaneously, according to the embodiments, by selecting the kernel that can most efficiently utilize the accelerator 120 during runtime, contention between models can be minimized, resource usage maximized, and performance can be improved.

[0045] Figure 2 illustrates the resource allocation process for running multiple models in a multi-core accelerator according to one embodiment.

[0046] Referring to Figure 2, an example is shown in which the compiler 210 generates multiple kernels at design time, and the scheduler 220 assigns one of these kernels to the accelerator 230 at runtime, depending on the state of the accelerator 230. In Figure 2, for the sake of explanation, we assume that models 1 to 3 are each requested at specific times.

[0047] At design time, when an execution request for Model 1 is received, the compiler 210 generates multiple candidate kernels that can be executed by the accelerator 230 for each layer included in Model 1. There are various ways to execute a particular layer on the accelerator 230, and the kernel may change accordingly. For example, a particular layer can be processed quickly by utilizing all the cores included in the accelerator 230, or it can be processed slowly by utilizing only one core included in the accelerator 230. Alternatively, the operating mode of the accelerator 230 executing a particular layer may change depending on how the data reuse pattern and tailing factor are set when performing calculations by that particular layer. Even if the operating mode of the accelerator 230 is diverse, the output data for the corresponding layer can be determined identically because the same layer is being processed.

[0048] In summary, the compiler 210 can generate multiple candidate kernels that differ in one or more of the number of cores used, data reuse patterns, and tailing factors when the corresponding layers run on the accelerator 230.

[0049] Multiple candidate kernels can be generated for the number of cores available when running on accelerator 230, where the predicted execution times differ from those of the other cores. For example, if accelerator 230 has 10 cores, 1 to 10 cores can be used when running a particular layer. Here, even if the predicted execution time decreases as the number of cores increases from 1 to 6, and increases from 7 or more cores, if the change in predicted execution time is small, multiple candidate kernels may be generated only for the number of cores from 1 to 6.

[0050] Furthermore, the data reuse patterns underlying the generation of multiple candidate kernels may include patterns that reuse the input feature map (most), patterns that reuse the weights (most), and patterns that reuse the output feature map (most) when executed on accelerator 230. Here, the input feature map represents the data input to the corresponding layer, and the output feature map represents the data output from the corresponding layer. In addition, the tailing factor underlying the generation of multiple candidate kernels can be expressed as a data processing unit determined based on the data reuse pattern, for example, as the size of 4-dimensional data (width, height, channels, tensor).

[0051] The compiler 210 can generate kernel information for each candidate kernel. The kernel information may include resource usage information used when the corresponding candidate kernel is executed on the accelerator 230. For example, the kernel information may include one or more of the following: the number of cores required to run the corresponding candidate kernel, the data reuse pattern, the tailing factor, the expected execution time, the memory access pattern, and the memory usage.

[0052] The compiler 210 generates multiple candidate kernels and corresponding kernel information for each layer included in the model in question, and transmits them to the scheduler 220 each time a request is received. In one embodiment, the compiler 210 can be implemented as the host processor described above.

[0053] During runtime, the scheduler 220 can select one of several candidate kernels based on the corresponding kernel information and the state information of the accelerator 230, and assign the selected candidate kernel to the accelerator 230. The scheduler 220 can select one of several candidate kernels for a particular layer that can maximize the utilization rate of the accelerator 230 based on the resource status of the accelerator 230, and assign it to the remaining resources of the accelerator 230. The state information of the accelerator 230 is transmitted to the scheduler 220 by the accelerator 230 and may include, for example, one or more of the following: resource usage information of the accelerator 230, kernel information of the layer currently running in the accelerator 230, and information about the subsequent layers of the layer currently running in the accelerator 230.

[0054] The scheduler 220 allocates the resources required for execution (e.g., cores, memory, memory traffic bandwidth, etc.) to the selected kernel on the accelerator 230, depending on the resource status of the accelerator 230. Once the execution of the kernel is complete, the scheduler 220 can continuously track the available resources of the accelerator 230 while reclaiming the allocated resources.

[0055] The scheduler 220 may be called each time a layer currently running in the accelerator 230 completes to perform scheduling. However, since the time required to run the kernel in the accelerator 230 can be predicted in advance based on kernel information received from the compiler 210, scheduling may also be performed in advance even when there are no additional model requests. However, even if all scheduling has been done in advance, if a new request is received, scheduling must be performed again. The timing of such scheduling may be selected depending on the resource status of the hardware driving the scheduler 220. For example, if hardware resources are sufficient, scheduling all in advance is beneficial from the perspective of overall throughput. On the other hand, if hardware resources are insufficient or there are constraints such as hardware power requirements (for example, if reducing power consumption is important), scheduling may be performed each time the execution of each layer is completed. To illustrate with another example, even if a model is already running, if the execution of a new model is requested, and the accelerator 230 has sufficient remaining resources to run the new model, the scheduler 220 can be called to perform scheduling. Here, whether a new model can be run using the remaining resources of accelerator 230 can be determined based on whether the remaining resources can run all or part of the layers of the new model, but various other criteria can also be applied without limitation.

[0056] In one embodiment, the scheduler 220 may be implemented in or included in the host processor described above, or if the accelerator 230 includes a multilevel core, it may be implemented in or included in the top-level core within the accelerator 230. In this specification, hardware devices that implement the compiler 210 and / or scheduler 220 may be referred to as data processing devices.

[0057] Figure 3 is a diagram illustrating kernel information according to one embodiment.

[0058] Referring to Figure 3, kernel information generated by a compiler (e.g., compiler 210 shown in Figure 2) according to one embodiment is illustrated. The kernel information for each candidate kernel may include one or more of the following: the number of cores required to execute the corresponding candidate kernel, the data reuse pattern, the tailing factor, the expected execution time, the memory access pattern, and the memory usage capacity. The kernel information shown in Figure 3 may include the number of cores, the data reuse pattern, and the tailing factor. In Figure 3, IF represents a pattern for reusing the input feature map, W represents a pattern for reusing the weighted values, and OF represents a pattern for reusing the output feature map. In Figure 3, the tailing factor (e.g., "8,1,2,2") corresponds to the size of the 4-dimensional data (kernel width, height, channels, and tensor). In Model 2, Layer 1 and Layer 2 are executed via a single kernel in a fused configuration, and multiple kernels are generated for Layer 1 and Layer 2, thereby determining the kernel information.

[0059] Figures 4 to 6 illustrate the process of selecting one of several candidate kernels according to one embodiment.

[0060] Referring to Figure 4, a flowchart is shown in which a scheduler according to one embodiment (for example, the scheduler 220 shown in Figure 2) first selects the layer to be executed in the accelerator, and then selects one of several candidate kernels corresponding to the selected layer.

[0061] In step S410, the scheduler determines whether there are enough available resources to run one layer. In other words, it determines whether there are enough resources available for the accelerator to run one kernel.

[0062] In step S420, the scheduler checks whether there are any executable layers in the queue. The queue may contain layers to be executed. If there are executable layers in the queue, step S430 is executed next; conversely, if there are no executable layers in the queue, the scheduler's operation terminates.

[0063] In step S430, the scheduler selects a layer to execute from the executable layers stored in the queue. For example, if multiple models have different priorities, the layer belonging to the highest-priority model is selected, but the scheduler's selection of a layer to execute is not limited to this. In the example shown on the right in Figure 4, the executable layers include layer 2 of model 1 and fused layers 1 and 2 of model 2, and any one of these may be selected as the layer to be executed by the accelerator.

[0064] In step S440, the scheduler selects one of several candidate kernels for the selected layer that is predicted to have the highest performance. The scheduler combines the current accelerator status and the kernel information of the candidate kernels and selects the candidate kernel that is expected to have the highest accelerator performance when each candidate kernel is executed. Depending on the situation or the user's choice, the performance criteria may be one or more of the following for each model: throughput, latency, fairness, power consumption, and accelerator utilization. As an example shown on the right in Figure 4, one of several candidate kernels for Layer 2 of Model 1, which was selected earlier, may be selected.

[0065] For example, the selection of the candidate kernel expected to have the highest performance is based on one of the following methods: greedy, resource equalization, total system performance priority, or power consumption minimization.

[0066] The greedy method refers to a method of selecting the kernel that is expected to have the shortest execution time from among the candidate kernels that can be executed within the accelerator's residual resources. Here, residual resources may refer only to currently idle resources that are not actually being used, or, depending on the case, may also include resources that are currently being used for other kernel executions but are expected to soon become residual resources. For example, the greedy method can repeatedly perform the process of selecting the one with the fastest expected execution time from among several candidate kernels for one or more layers that must be executed next.

[0067] The resource equalization method divides the accelerator's total available resources by the number of layers that can run simultaneously, distributing resources equally among them. Within the allocated resources, it selects the kernel that is expected to have the shortest execution time among the executable candidate kernels for a particular layer. For example, if the accelerator has a total of 20 cores and there are 4 models that need to be run, 5 cores can be allocated to each model. Based on the 5 allocated cores, the kernel expected to have the fastest execution time can be selected from among the multiple candidate kernels for the layers that need to be run in each model.

[0068] The total system performance priority method is a scheduling method that selects the combination of kernels that is expected to have the shortest average relative execution time (e.g., geometric mean) when selecting candidate kernels for each currently executable layer. For example, unlike the greedy method which selects the kernel expected to have the fastest execution time within a relatively short time range, the total system performance priority method is a scheduling method that searches for the combination of kernels that will complete execution in the shortest time from the outset, from an overall perspective.

[0069] The power consumption minimization method selects the kernel that uses the fewest resources from among the candidate kernels that satisfy the requested or determined QoS. For example, the power consumption minimization method selects the kernel that requires the fewest resources from among the kernels that satisfy the predetermined QoS, rather than the kernel that can be expected to have the fastest execution time.

[0070] Therefore, by adaptively changing the number of cores running each layer at runtime according to the accelerator's resource status, inter-model contention can be minimized and accelerator resource utilization can be maximized. For example, in general situations without specific requirements, the overall system throughput can be increased via a total system performance priority method, and if an urgent execution is required for a particular model, a greedy method can be arbitrarily applied to that model, and it can be scheduled to complete quickly.

[0071] Referring to Figure 5, a flowchart is shown in which, instead of first selecting the layer to be executed by the scheduler according to one embodiment (for example, the scheduler 220 shown in Figure 2), the scheduler selects the kernel to be executed by the accelerator from among all executable candidate kernels.

[0072] In step S510, the scheduler determines whether there are enough available resources to run one layer. In step S520, the scheduler checks whether there are any executable candidate kernels in the queue. The queue may contain candidate kernels to be executed. If there are executable candidate kernels in the queue, step S530 is executed next; conversely, if there are no executable candidate kernels in the queue, the scheduler's operation terminates. In step S530, the scheduler selects one of the executable candidate kernels that is predicted to have the highest performance. In the example shown on the right in Figure 5, one of the multiple candidate kernels for layer 2 of model 1 and the multiple candidate kernels for fused layers 1 and 2 of model 2 may be selected. For example, if there is no priority among the multiple models, the candidate kernel predicted to have the highest performance will be selected. The above explanation can be directly applied to the selection of candidate kernels, so a more detailed explanation is omitted.

[0073] Referring to Figure 6, a flowchart is shown illustrating the process by which the scheduler selects a kernel considering memory access patterns, according to one embodiment.

[0074] In step S610, the scheduler determines whether the number of idle cores in the accelerator is greater than or equal to the number of cores requested by the candidate kernel. If the number of idle cores in the accelerator is less than the number of cores requested by the candidate kernel, the scheduler excludes the candidate kernel in step S660.

[0075] In step S620, the scheduler extracts combinations of cores to be mapped per candidate kernel. Even if a kernel runs on the same number of cores, the way it accesses memory may differ depending on the accelerator structure. If the accelerator has a hierarchical structure of cores and memory, combinations are extracted for how many cores per cluster may be used.

[0076] For example, if five cores make up one cluster, and the accelerator contains four clusters, the total number of cores is 20. Now, if 10 cores in two clusters are already in use, and only two cores are in use in another cluster, then the remaining 8 idle cores are understood to be distributed as 3 and 5 cores in the two clusters, respectively.

[0077] In step S630, the scheduler checks the amount of memory available per core combination. In other words, the scheduler can determine the amount of memory available to idle cores in each cluster.

[0078] In step S640, the scheduler compares the memory access patterns of the available cores. The scheduler checks the memory access patterns for each candidate kernel based on the tailing factor and data reuse patterns. It is necessary to compare whether the memory access patterns of the candidate kernels conflict with the memory access patterns of the models currently running in the accelerator. If a conflict occurs, the extent to which it affects the QoS of the running and planned models is analyzed.

[0079] In step S650, the scheduler selects the best executable candidate kernel according to the scheduling strategy. For example, if there is a candidate kernel that does not conflict with the data access pattern of the currently running model, that candidate kernel is selected. If there is no candidate kernel that does not conflict, the candidate kernel that causes the fewest conflicts is selected.

[0080] In one embodiment, when various models are executed, the bus corresponding to the shared resource must be classified and used, and therefore, memory access patterns must be considered. Memory access patterns are examined for each input feature map, weighted value, and output feature map, and these memory accesses are performed at regular intervals. This converts these memory access patterns into real-time tasks (e.g., period, access time), and a demand-bound function used in real-time task studies is determined. Using such a demand-bound function, it is possible to understand how different memory access patterns affect each other, thereby allowing for prior confirmation of whether memory access by other models will negatively impact QoS, one of the user's requirements, and preventing such impacts in advance.

[0081] Figures 7 and 8 show examples of resource allocation according to one embodiment.

[0082] Referring to Figure 7, an example of dynamically allocating resources in one embodiment is shown. The table shown at the top of Figure 7 shows a list of executable candidate kernels for each layer, and the core allocation section below shows how the eight cores are allocated to compute each layer.

[0083] Before Model 1 is executed, five cores are already allocated to run Layer 1 of Model 1, and the remaining three cores are idle. In this situation, after Model 2's Layer 1 is started, kernel 2, which can make the most use of the remaining three cores, can be selected to run Layer 1 of Model 2. In this way, by adaptively allocating resources in response to the accelerator's current state, the accelerator's utilization rate can be efficiently improved.

[0084] Referring to Figure 8, another example of dynamically allocating resources according to one embodiment is shown. In the example shown in Figure 8, it is assumed that the execution of Model 1 is requested first, and the execution of Model 2 is requested while Model 1 is running on the accelerator.

[0085] When only Model 1 is running in the accelerator, all 10 cores included in the accelerator are allocated to Layer 1 of Model 1 (S810). To run Layer 2-1 and Layer 2-2, respectively, to which the output data from Layer 1 of Model 1 has been transmitted, 5 of the 10 cores are first allocated to Layer 2-1 (S820). If execution of Model 2 is requested before execution of Layer 2-2 begins, some of the remaining 5 cores (for example, 3) are allocated to Layer 2-2 of Model 1 (S830), and the remaining 2 cores are allocated to Layer 1 of Model 2 (S840). 5 cores are allocated for execution of Layer 3, to which the output data from Layers 2-1 and 2-2 of Model 1 has been transmitted, and the remaining 5 cores are allocated to Model 2. Here, since Layers 2 and 3 of Model 2 have already been fused in the compilation step and can operate as a single layer, the remaining 5 cores are allocated to the corresponding fused layer (S850). After Model 1 has finished running, all 10 cores are allocated to Model 2 (S860). Here, the cores allocated to each model can minimize migration costs by maximizing their allocation unless a separate new model is required to run.

[0086] In this way, even during the runtime when the deep learning model is executed, by adaptively changing resource allocation according to the accelerator status, a candidate kernel corresponding to the optimal number of cores and other appropriate resource allocations can be selected for each layer executed, thereby minimizing accelerator idle time and improving utilization.

[0087] Figure 9 is a diagram illustrating resource allocation considering a memory access pattern according to one embodiment.

[0088] Referring to Figure 9, one embodiment shows an example in which resources are allocated considering whether or not collisions occur between memory access patterns of simultaneously executed models. Layer i of Model 1 and layer j of Model 2 each have various memory access patterns, and the most efficient memory access patterns may be determined to be a and α, respectively. However, memory access patterns a and α represent the most efficient patterns when the respective models are executed independently. In this embodiment, if layer i of Model 1 and layer j of Model 2 are executed simultaneously using memory access patterns a and α, a collision occurs between the memory access patterns (S910). That is, the patterns that utilize memory access resources between Models 1 and 2 overlap, resulting in a collision where one of the models is unable to utilize the memory access resources. Such collisions have a negative impact on accelerator utilization and should therefore be minimized. Accordingly, in accelerator resource allocation, it is necessary to consider whether or not collisions of memory access patterns occur between simultaneously driven models. In the embodiment shown in Figure 9, memory access pattern a of layer i of Model 1 and memory access pattern β of layer j of Model 2, which does not cause a collision, are selected. In other words, a candidate kernel corresponding to memory access pattern a may be selected to execute layer i of model 1, and a candidate kernel corresponding to memory access pattern β may be selected to execute layer j of model 2. In this way, when driving multiple models, the kernel that has the least impact on memory access is selected.

[0089] Figure 10 is a diagram illustrating an electronic device including multiple accelerators according to another embodiment.

[0090] Referring to Figure 10, the electronic device according to another embodiment includes a CPU 1010, a plurality of NPUs 1020, and a host DRAM 1030. Here, the CPU 1010 corresponds to the host process described earlier, and the NPUs correspond to accelerators.

[0091] Unlike the embodiments described earlier, when one or more models are executed on multiple NPU1020s, the kernel information generated by the compiler may include additional resource information that is consumed to use multiple NPU1020s. Furthermore, the scheduler must perform scheduling not only considering the contention between kernels running on multiple NPU1020s, but also taking into account performance differences related to changes in the system structure.

[0092] The electronic device model shown in Figure 10 is executed via two or more NPUs, in which case the NPUs can exchange data via the PCIe bus. Furthermore, if the on-chip memory (e.g., SRAM) or off-chip memory capacity within the NPU is insufficient, the host DRAM 1030 may be used additionally.

[0093] In summary, among the candidate kernels generated by the compiler, candidate kernels that use various NPUs may have information regarding the amount of data transmitted between NPUs added to their kernel information, and candidate kernels that use the host DRAM1030 may have information regarding the host memory usage added to their kernel information.

[0094] Based on this kernel information, the scheduler can select the appropriate kernel for the accelerator status at runtime. For example, in the case of layers with extremely long computation times, it is more beneficial to distribute the computation across various NPUs rather than processing it on a single NPU, considering the cost of data transfer between NPUs. In such cases, a kernel that uses various NPUs can be selected.

[0095] Figure 11 is a diagram illustrating the hierarchical structure of an accelerator according to one embodiment.

[0096] Referring to Figure 11, an example of an accelerator 1100 having a hierarchical structure according to one embodiment is shown. The accelerator 1100 includes a plurality of processing elements and a multilevel memory accessible by at least one of the plurality of processing elements. The multilevel memory is a term referring to the level 0 memory 1111, level 1 memory 1121, and level 2 memory 1131, which correspond to the on-chip memory of the accelerator 1100.

[0097] Processing element 1110, which is one of several processing elements, includes a level 0 memory 1111, a level 0 DMA 1113, a MAC (Multiplier Accumulator) 1115, and a level 0 controller 1117.

[0098] The level 0 memory 1111 may be a memory accessible by the corresponding processing element 1110. The level 0 DMA 1113 controls the input and / or output data of the level 0 memory 1111 in accordance with instructions from the level 0 controller 1117.

[0099] MAC1115 performs calculations on the workload assigned to processing element 1110. MAC1115 may also perform multiply-accumulation operations on given data or apply activation functions to given data. Activation functions may include, for example, sigmoid, hyperbolic tangent (tanh), and ReLU (rectified linear unit).

[0100] The Level 0 controller 1117 controls components included in the processing element 1110, such as the Level 0 memory 1111, Level 0 DMA 1113, and MAC 1115.

[0101] The description of processing element 1110 mentioned above applies equally to each of the multiple processing elements included in accelerator 1100.

[0102] In one embodiment, a plurality of processing elements are clustered into groups of n processing elements, where n is a natural number greater than 1 and less than the number of processing elements contained in the accelerator 1100. In other words, a portion of the plurality of processing elements contained in the accelerator 1100 are clustered, and this will be explained in relation to the clustered processing elements 1120.

[0103] The clustered processing elements 1120 share a single level 1 memory 1121. In other words, the level 1 memory 1121 may be accessed by the clustered processing elements 1120. For example, even if the operations performed by the first processing element and the second processing element within the clustered processing elements 1120 are different, some of the data required for these operations can be shared. Rather than storing such common data in the level 0 memory 1111 of the first and second processing elements respectively, storing it in the level 1 memory 1121 and sharing it between the first and second processing elements can increase the efficiency of the accelerator 1100. In the conceptual example shown in Figure 11, each processing element can access the level 1 memory 1121 adjacent to itself.

[0104] Furthermore, the entire set of processing elements 1130 shares the Level 2 memory 1131. In other words, the Level 2 memory 1131 may be accessed by multiple processing elements included in the accelerator 1100. For example, among the multiple processing elements included in the accelerator 1100, there may be processing elements that are not clustered in the same group but share some of the data necessary for the operations they perform. Such processing elements cannot share the relevant data via the Level 1 memory 1121, but can efficiently share the common data via the Level 2 memory 1131, thereby increasing the efficiency of the accelerator 1100.

[0105] In summary, each processor unit can access its own level 0 memory 1111, its adjacent level 1 memory 1121, and the accelerator 1100's level 2 memory 1131, and utilize these memories when performing its assigned workload. Thus, the accelerator 1100 may include multilevel memory, and the included memory may be hierarchical. Furthermore, the DMA and controller included in the accelerator 1100 may also have hierarchical multilevel memory.

[0106] In the example shown in Figure 11, the multiple processing elements included in the accelerator 1100 can handle four workloads simultaneously. For example, computationally intensive workloads may be assigned to multiple processing elements for processing, while workloads with relatively low computational requirements may be assigned to fewer processing elements for processing.

[0107] In Figure 11, for the sake of explanation, 64 processing elements are clustered in groups of eight, and three levels of memory are utilized when four workloads are running. However, various other numbers of processing elements, workloads, and levels can be applied without limitation.

[0108] Figure 12 shows a data processing method according to one embodiment.

[0109] Referring to Figure 12, a data processing method performed by one or more processors according to one embodiment includes the steps of: receiving a request to run a model in an accelerator (S1210); generating a plurality of candidate kernels for each layer included in the model (S1220); and assigning one of the plurality of candidate kernels for a layer to be run in the accelerator to the accelerator, selected based on the corresponding kernel information and accelerator state information (S1230).

[0110] The embodiments described herein may be comprehensively applied to a family of server products to which one or more SoCs (System on a Chip) are applied, in which multiple accelerator cores and a hierarchically structured shared memory are connected to a cluster infrastructure structure, for the purpose of accelerating AI processing.

[0111] Since the steps shown in Figure 12 are directly governed by the same principles described above with reference to Figures 1 to 11, a more detailed explanation will be omitted.

[0112] Figures 13 and 14 show examples of an electronic device according to one embodiment.

[0113] Referring to Figure 13, the electronic device according to one embodiment is implemented as a server 1300.

[0114] Server 1300, as a separate device distinct from user terminals controlled by users, can communicate with one or more user terminals via wired and / or wireless networks. Server 1300 receives requests simultaneously sent by multiple users through their own terminals. The host processor 1310 included in Server 1300 generates multiple candidate kernels for each layer included in the requested model, and the accelerator 1320 executes the layer based on one of the multiple candidate kernels selected based on the corresponding kernel information and accelerator state information for the layer to be executed. The inference results obtained by executing the model in accelerator 920 may be returned to the corresponding user terminals. For example, user terminals may include various computing devices such as smartphones, tablets, laptops, and personal computers; various wearable devices such as smartwatches and smart glasses; various home appliances such as smart speakers, smart TVs, and smart refrigerators; smart cars, smart kiosks, IoT (Internet of Things) devices, drones, and robots.

[0115] Referring to Figure 14, the electronic device according to one embodiment may be implemented as a user terminal 1400. In Figure 14, the user terminal 1400 is shown as a smartphone for convenience of explanation, but it can be applied to any other device controlled by the user without limitation. The user terminal 1400 directly receives requests from the user, the host processor 1410 included in the user terminal 1400 generates multiple candidate kernels for each layer included in the requested model, and the accelerator 1420 executes the layer based on one of the multiple candidate kernels selected based on the corresponding kernel information and accelerator state information for the layer to be executed. The inference results obtained through this execution process are provided to the user, or subsequent operations of the user terminal 1400 are performed based on them.

[0116] In Figures 13 and 14, the server 1300 and user terminal 1400, which are embodied as electronic devices, can process the operations described above, so a more detailed explanation is omitted. Furthermore, in this specification, an electronic device may be represented as an accelerator system including a host processor and an accelerator.

[0117] The embodiments described above are embodied in hardware components, software components, or combinations of hardware and software components. For example, the devices and components described in these embodiments are embodied using one or more general-purpose or special-purpose computers, such as a processor, controller, ALU (arithmetic logic unit), digital signal processor, microcomputer, FPA (field programmable array), PLU (programmable logic unit), microprocessor, or different devices that execute and respond to instructions. The processing device can run an operating system (OS) and one or more software applications run on the OS. The processing device can also access data in response to software execution, and can store, manipulate, process, and generate data. For convenience of understanding, the processing device may sometimes be described as being used as a single unit, but a person with ordinary skill in the art will see that the processing device includes multiple processing elements and / or multiple types of processing elements. For example, the processing device may include multiple processors or one processor and one controller. Other processing configurations, such as parallel processors, are also possible.

[0118] Software includes computer programs, code, instructions, or a combination of one or more of these, which can configure a processing unit to operate as desired, or instruct the processing unit independently or in combination. Software and / or data can be permanently or temporarily embodied in any type of machine, component, physical device, virtual device, computer storage medium or device, or transmitted signal wave, for interpretation by a processing unit or for providing instructions or data to a processing unit. Software can be distributed across a network of computer systems, stored and executed in a distributed manner. Software and data can be stored on a recording medium readable by one or more computers.

[0119] The method according to this embodiment is embodied in the form of program instructions that are implemented via various computer means and recorded on a computer-readable recording medium. The recording medium includes program instructions, data files, data structures, etc., individually or in combination. The recording medium and program instructions may be specifically designed and configured for the purposes of the present invention, or they may be known and usable by those skilled in the art who have technology in the field of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floppy disks, and hardware devices specifically configured to store and execute program instructions, such as ROMs, RAMs, and flash memory. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code executed by a computer using an interpreter or the like.

[0120] The hardware device described above may be configured to operate as one or more software modules to perform the operations shown in the present invention, and vice versa.

[0121] As described above, although embodiments have been illustrated with limited drawings, a person with ordinary skill in the art can apply various technical modifications and variations based on the above description. For example, the described techniques may be performed in a different order than described, and / or the components of the described systems, structures, devices, circuits, etc. may be combined or assembled in a different manner than described, or replaced or substituted with other components or equivalents, and still achieve suitable results.

[0122] Therefore, the scope of the present invention is not limited to the disclosed embodiments, but is defined by the claims and equivalents thereof. [Explanation of symbols]

[0123] 100 Electronic equipment 110 host processors 120 Accelerator 130 Accelerator Chips 131 Accelerator Core 132 On-Chip Memory 140 Off-Chip Memory 210 Compiler 220 Scheduler 230 Accelerator 1010 CPU 1020 NPU 1030 Host DRAM 1100 accelerator 1110 Processing Element 1111 Level 0 Memory 1113 Level 0DMA 1115 MAC 1117 Level 0 Controller 1120 Processing Element 1121 Level 1 Memory 1130 Overall Processing Element 1131 Level 2 Memory 1300 servers 1400 user terminals 1410 Host Processor 1420 Accelerator

Claims

1. A data processing method performed by one or more processors, The steps include receiving a request to run a neural network model in an accelerator, The steps include generating multiple candidate kernels for each of the multiple layers included in the aforementioned model, The process includes the step of assigning to the accelerator one of several candidate kernels selected from among a plurality of candidate kernels for a layer to be executed in the accelerator, based on the corresponding kernel information and the state information of the accelerator, The aforementioned assignment step involves selecting a candidate kernel and assigning it to the accelerator based on whether or not a collision occurs with the data access pattern of the layer currently running in the accelerator. Data processing method.

2. The data processing method according to claim 1, wherein the generation step generates a plurality of candidate kernels that run each layer with different accelerator resource usage.

3. The data processing method according to claim 1, wherein the generation step generates a plurality of candidate kernels in which the corresponding layer differs in one or more of the following: the number of cores used in the accelerator during execution, the data reuse pattern, and the tailing factor.

4. The data processing method according to any one of claims 1 to 3, wherein the number of cores among the plurality of cores included in the accelerator that execute the model is variable at runtime according to the resource status of the accelerator.

5. The data processing method according to claim 1, wherein the allocation step involves selecting one of the plurality of candidate kernels that can maximize the utilization rate of the accelerator in the resource status of the accelerator and allocating it to the remaining resources of the accelerator.

6. The data processing method according to claim 1, wherein the state information of the accelerator includes one or more of the following: resource usage information of the accelerator, kernel information of the layer currently running in the accelerator, and information of a subsequent layer of the layer currently running in the accelerator.

7. The data processing method according to claim 1, wherein the assignment step, if the collision occurs, selects from the plurality of candidate kernels a candidate kernel that satisfies the QoS (Quality of Service) of the layer currently being executed and the layer scheduled to be executed, or a candidate kernel that has the least impact on the QoS, and assigns the selected candidate kernel to the accelerator.

8. The data processing method according to claim 1, wherein the allocation step involves allocating the candidate kernel in response to the completion of any one of the one or more layers currently running in the accelerator, or allocating the candidate kernel in response to the availability of the remaining resources of the accelerator to run the model according to the new request when a new request is received.

9. The data processing method according to claim 1, wherein the allocation step is performed on a host processor that generates the plurality of candidate kernels or on an accelerator that executes the model.

10. The data processing method according to claim 1, wherein the assignment step involves assigning one of the multiple candidate kernels selected to the accelerator based on one of the following: a greedy method, a resource equal distribution method, a total system performance priority method, or a power consumption minimization method.

11. The data processing method according to any one of claims 1 to 10, wherein the accelerator includes a plurality of cores and runs one or more models.

12. The data processing method according to any one of claims 1 to 11, further comprising the step of generating at least a portion of the output feature map by running the layers of the model based on the assigned candidate kernel using the accelerator.

13. A computer-readable storage medium on which a program for performing the method according to any one of claims 1 to 12 is recorded.

14. Includes one or more processors, The one or more processors described above are: We receive a request to run a neural network model on an accelerator. Multiple candidate kernels are generated for each of the multiple layers included in the aforementioned model. From among a plurality of candidate kernels for the layer to be executed in the accelerator, one candidate kernel selected based on the corresponding kernel information and the state information of the accelerator is assigned to the accelerator. The assignment selects a candidate kernel and assigns it to the accelerator based on whether or not a collision occurs with the data access pattern of the layer currently running in the accelerator. Data processing device.

15. A host processor receives a request to run a neural network model on an accelerator and generates multiple candidate kernels for each of the multiple layers included in the model. An accelerator that executes a layer based on one of several candidate kernels selected from among the candidate kernels for the layer to be executed, based on the corresponding kernel information and the state information of the accelerator, The candidate kernel is selected based on whether or not a collision occurs with the data access pattern of the layer currently running in the accelerator. electronic equipment.

16. The electronic device according to claim 15, wherein the host processor generates a plurality of candidate kernels that cause each layer to run with different accelerator resource usage.

17. A data processing method performed by a processor, The steps include generating multiple kernels for each of the first neural network model and the second neural network model, The steps include: executing the kernel of the first neural network model based on the number of cores available resources of the accelerator; In response to the execution, the step of executing the kernel of the second neural network model based on the remaining cores of the available resources of the accelerator, The execution of the kernel includes the step of generating one or more output feature maps, The step of executing the kernel of the second neural network model is to execute the kernel of the second neural network model in response to the determination that no collisions between memory access patterns occur between the execution of the kernel of the first neural network model and the execution of the kernel of the second neural network model. Data processing method.

18. For each of the kernels of the first neural network model and the second neural network model, the kernel is: The number of accelerator cores used to execute the kernel, When the kernel is executed, a data reuse pattern is determined that includes one or more of the following: reuse of input feature maps, reuse of weighted values, and reuse of output feature maps. The tailing factor includes the size of the multidimensional data of the kernel, Based on kernel information including at least one of the above, one of the plurality of kernels is selected to be executed. The data processing method according to claim 17.

Images