Data processing method and apparatus

Abstract

Data processing methods and apparatus are disclosed. A processor-implemented data processing method includes receiving a request for executing a neural network model on an accelerator, generating a plurality of candidate kernels for each of a plurality of layers included in the model, and assigning a single candidate kernel to the accelerator, the single candidate kernel being selected from the plurality of candidate kernels generated for a layer to be run on the accelerator based on corresponding kernel information and state information of the accelerator.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2020-0136623, filed on October 21, 2020, with the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference for all purposes. Technical Field

[0002] The following description relates to data processing methods and equipment. Background Technology

[0003] Artificial intelligence (AI) technology can perform inference and learning through specific arithmetic operations. Various devices can be dedicated hardware used to implement and execute AI.

[0004] Specialized hardware for AI can be implemented using, for example, central processing units (CPUs) and graphics processing units (GPUs), and can also be implemented using field programmable gate arrays (FPGAs) and application-specific integrated circuits (ASICs). Summary of the Invention

[0005] This summary is provided to present, in a simplified form, the selection of concepts further described below in the detailed embodiments. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to help determine the scope of the claimed subject matter.

[0006] In one general aspect, a processor-implemented data processing method includes: receiving a request to execute a neural network model on an accelerator; generating a plurality of candidate kernels for each of a plurality of layers included in the model; and assigning a single candidate kernel to the accelerator, the single candidate kernel being selected from the plurality of candidate kernels for the layer to be run on the accelerator based on corresponding kernel information and accelerator state information.

[0007] The generation process may include generating multiple candidate kernels that run each layer using different accelerator resource usage.

[0008] The generation step may include: generating the plurality of candidate kernels such that for each of the plurality of candidate kernels, the number of kernels used when the corresponding layer of the candidate kernel is running on the accelerator, any one or any combination of the number of kernels, the data reuse mode and the tiling factor is different from the number of kernels used when the corresponding layer of another candidate kernel is running on the accelerator.

[0009] The number of cores used to execute the model, which are included among the multiple cores in the accelerator, can vary depending on the accelerator's resource availability at runtime.

[0010] The allocation step may include: selecting a single candidate kernel from the plurality of candidate kernels that is configured to maximize the utilization of the accelerator given the accelerator's resources as the single candidate kernel, and allocating the selected candidate kernel to the remaining resources of the accelerator.

[0011] The accelerator's status information may include any one or any combination of the accelerator's resource usage information, kernel information of the layers running on the accelerator, and information about subsequent layers of the layers running on the accelerator.

[0012] The allocation process may include: selecting candidate kernels based on determining whether a conflict has occurred or not with the data access patterns of the layers of the model running on the accelerator, and allocating the selected candidate kernels to the accelerator.

[0013] The allocation process may include: in response to a conflict, selecting any one of the following as a candidate kernel: a candidate kernel that satisfies the Quality of Service (QoS) of either the layer currently running on the accelerator or the layer to be run on the accelerator, and the candidate kernel among the plurality of candidate kernels that has the least impact on QoS; and allocating the selected candidate kernel to the accelerator.

[0014] The allocation steps may include any one of the following: allocating a single candidate kernel in response to the completion of at least one of the layers of a model running on the accelerator; and allocating a single candidate kernel in response to receiving a new request and using the remaining resources of the accelerator to execute the model corresponding to the new request.

[0015] The allocation step can be performed by any one of the host processors and accelerators configured to generate the plurality of candidate cores and execution models.

[0016] The allocation process may include: assigning a single candidate kernel selected from the plurality of candidate kernels based on any one of the greedy scheme, the equal resource allocation scheme, the overall system performance priority sorting scheme, and the power consumption minimization scheme as a single candidate kernel.

[0017] An accelerator may include multiple cores and may be configured to execute at least one model.

[0018] The method may include generating at least a portion of the output feature map by running the model's layers using an accelerator based on a single assigned candidate kernel.

[0019] A non-transitory computer-readable recording medium can store instructions that, when executed by one or more processors, configure the one or more processors to perform the method.

[0020] In another general aspect, a data processing device includes: one or more processors configured to: receive a request to execute a neural network model on an accelerator; generate a plurality of candidate kernels for each of a plurality of layers included in the model; and assign a single candidate kernel to the accelerator, the single candidate kernel being selected from the plurality of candidate kernels for the layer to be run on the accelerator based on corresponding kernel information and accelerator state information.

[0021] In another general aspect, an electronic device includes: a host processor configured to: receive a request to execute a neural network model on an accelerator and generate a plurality of candidate kernels for each of a plurality of layers included in the model; and an accelerator configured to: run a layer determined to be run based on a single candidate kernel, the single candidate kernel being selected from the plurality of candidate kernels for the layer to be run based on corresponding kernel information and accelerator state information.

[0022] For the generation step, the host processor can be configured to generate multiple candidate kernels that run each layer using different amounts of accelerator resource usage.

[0023] In another general aspect, a processor-implemented data processing method includes: generating multiple kernels for each of a first neural network model and a second neural network model; running the kernels of the first model on a certain number of cores of available resources of an accelerator; in response to the start of said running, running the kernels of the second model on the remaining number of cores of available resources of the accelerator; and generating one or more output feature maps based on the running of the kernels.

[0024] The steps of running the kernel of the second model may include: running the kernel of the second model in response to determining that there will be no conflict between memory access modes during the operation of the kernel of the first model and the kernel of the second model.

[0025] For each of the kernels in the first model and the second model, a kernel for operation can be selected from the plurality of kernels based on kernel information, which includes any one or more of the following: the number of accelerator cores used to run the kernel; the data reuse pattern, including any one or more of the reuse of input feature maps, weights, and output feature maps when running the kernel; and the tiling factor, including the multidimensional data size of the kernel.

[0026] Other features and aspects will become clearer from the following detailed description, drawings and claims. Attached Figure Description

[0027] Figure 1 An example of an electronic device is shown.

[0028] Figure 2 An example of resource allocation processing for performing multi-model operations on a multi-core accelerator is shown.

[0029] Figure 3 An example of kernel information is shown.

[0030] Figures 4 to 6 An example of the process of selecting a single candidate kernel from multiple candidate kernels is shown.

[0031] Figure 7 and Figure 8 An example of resource allocation is shown.

[0032] Figure 9 An example of resource allocation based on memory access patterns is shown.

[0033] Figure 10 An example of an electronic device including multiple accelerators is shown.

[0034] Figure 11 An example of a hierarchical structure for an accelerator is shown.

[0035] Figure 12 An example of a data processing method is shown.

[0036] Figure 13 and Figure 14 An example of an electronic device is shown.

[0037] Throughout the accompanying drawings and detailed embodiments, unless otherwise described or provided, the same reference numerals will be understood to denote the same elements, features, and structures. The drawings may not be to scale, and for clarity, illustration, and convenience, the relative sizes, proportions, and depictions of elements in the drawings may be exaggerated. Detailed Implementation

[0038] The following detailed embodiments are provided to aid the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding the disclosure of this application. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but may be changed as will become clear upon understanding the disclosure of this application, except for operations that must occur in a specific order. Furthermore, for clarity and brevity, descriptions of features known in the art upon understanding the disclosure of this application may be omitted.

[0039] The features described herein may be implemented in different forms and should not be construed as limited to the examples described herein. Rather, the examples described herein are provided only to illustrate some of the many feasible ways of implementing the methods, apparatus, and / or systems described herein that will become clear upon understanding the disclosure of this application.

[0040] The following structural or functional descriptions of the examples disclosed in this disclosure are intended only for illustrative purposes, and the examples may be implemented in various forms. These examples are not intended to be limiting, but various modifications, equivalents, and alternatives are also intended to be covered within the scope of the claims.

[0041] Although the terms "first" or "second" are used to explain various components, assemblies, regions, layers, or parts, the components, assemblies, regions, layers, or parts are not limited to these terms. These terms should only be used to distinguish one component, assembly, region, layer, or part from another component, assembly, region, layer, or part. For example, within the scope of the claims according to the concept of this disclosure, a first component, first assembly, first region, first layer, or first part referred to in the examples described herein may be referred to as a second component, second assembly, second region, second layer, or second part, or similarly, a second component, second assembly, second region, second layer, or second part may be referred to as a first component, first assembly, first region, first layer, or first part.

[0042] Throughout this specification, it will be understood that when a component or element is referred to as being "on," "connected to," or "joined to" another component or element, the component or element may be directly on, connected to, or joined to the other component or element, or there may be one or more intermediate elements located between them. Conversely, when a component or element is referred to as being "directly on," "directly connected to," or "directly joined to" another component or element, there are no intermediate components or elements. Similarly, expressions such as "between" and "immediately between," as well as "adjacent to" and "closely adjacent to," can also be interpreted as described above.

[0043] The terminology used herein is for the purpose of describing particular examples only and is not intended to limit the scope of this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. As used herein, the term “and / or” includes any combination of any one or any two or more of the associated listed items. As used herein, the terms “comprising,” “including,” and “having” indicate the presence of the stated features, integrals, steps, operations, elements, components, quantities, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, quantities, and / or combinations thereof. The term “may” (e.g., regarding what an example or embodiment may include or may implement) is used herein with respect to examples or embodiments to indicate the presence of at least one example or embodiment that includes or implements such features, and all examples are not limited thereto.

[0044] Unless otherwise defined, all terms used herein (including technical or scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which the examples pertain. It will also be understood that, unless expressly defined herein, terms (such as those defined in common dictionaries) shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and in this disclosure, and shall not be interpreted in an idealized or overly formal sense.

[0045] In the following description, examples will be given in detail with reference to the accompanying drawings, and the same reference numerals in the drawings always denote the same elements. Further descriptions related to them are omitted.

[0046] Figure 1 An example of an electronic device is shown.

[0047] Reference Figure 1 The electronic device 100 may include a host processor 110 and an accelerator 120. The host processor 110 and the accelerator 120 may communicate with each other via a bus, a network on-chip (NoC), a peripheral component interconnect (PCIe), or the like.

[0048] The host processor 110 may be a device configured to control the operation of components included in the electronic device 100, and may be, for example, a central processing unit (CPU). The host processor 110 may receive at least one request for processing a neural network at the accelerator 120, and in response to the request generate a kernel including instructions executable at the accelerator 120. The request may be used for data inference based on the neural network, and may cause the accelerator 120 to execute (i.e., run) (hereinafter, the terms “execute” and “run” and their modifications are used interchangeably) the neural network and obtain data inference results (e.g., data inference results for object recognition, pattern recognition, computer vision, speech recognition, machine translation, machine interpretation, recommendation services, personalization services, video processing, and / or autonomous driving). Furthermore, the request may be used for learning the neural network and training the neural network to perform the aforementioned operations based on learning data. Although described based on data inference for clarity, this description is also applicable to situations where data learning is not excluded. The host processor 110 may transmit the data to be inferred and / or the kernel for executing the neural network to the accelerator 120.

[0049] Accelerator 120 may be an artificial intelligence (AI) accelerator configured to infer data by executing neural networks based on kernels transferred from host processor 110, and may be a separate processor distinct from host processor 110. For example, accelerator 120 may be a neural processing unit (NPU), graphics processing unit (GPU), tensor processing unit (TPU), and / or digital signal processor (DSP).

[0050] Due to the characteristics of neural network operations, accelerator 120 can be a separate dedicated processor that processes specific tasks more efficiently than general-purpose host processor 110. Here, accelerator core 131 and on-chip memory 132 included in accelerator chip 130 can be used.

[0051] Accelerator core 131 may include one or more processing elements (PEs) configured to perform operations according to a neural network. Although for clarity of description... Figure 1 A single accelerator core 131 is shown included in accelerator chip 130, but multiple accelerator cores may be included in accelerator chip 130 and may be used to process operations according to one or more requested neural networks. In a non-limiting example, accelerator core 131 may include multiple cores. Here, for clarity of description, accelerator core 131 may also be referred to as a core.

[0052] On-chip memory 132 may be a means including globally shared buffers and / or local buffers contained in accelerator chip 130, and is configured to store data required for accelerator core 131 to perform operations or to store the results of operation execution, and may be distinguishable from off-chip memory 140. For example, on-chip memory 132 may include static random access memory (SRAM) and / or temporary storage memory accessible via address space. In a non-limiting example, on-chip memory 132 may include a plurality of on-chip memories, each corresponding to a corresponding one of the plurality of cores.

[0053] Off-chip memory 140 may be memory located outside of accelerator chip 130, and may be, for example, dynamic random access memory (DRAM) and / or high bandwidth memory (HBM). However, off-chip memory 140 is provided only as an example and is not limited thereto. Off-chip memory 140 may store data to be inferred by accelerator core 131 and / or parameters of the neural network to be executed by accelerator core 131. The stored data may be transferred to accelerator core 131 to perform inference. Off-chip memory 140 may be used when the on-chip memory 132 included in accelerator chip 130 is insufficient to execute the neural network on accelerator core 131.

[0054] Off-chip memory 140 may have a storage capacity greater than that of on-chip memory 132 in accelerator chip 130. However, when executing neural networks, the cost for accelerator core 131 to access off-chip memory 140 may be greater than the cost for accelerator core 131 to access on-chip memory 132. Memory access cost may represent the power and / or time required to read or write data by accessing the corresponding memory.

[0055] Neural networks can provide the optimal output corresponding to the input by mapping inputs and outputs with a non-linear relationship based on deep learning. Deep learning can be a machine learning scheme used to solve a given problem on a large dataset, and can be the process of finding a model and parameters (e.g., kernel weights) that represent the structure of the neural network and optimizing the processing of the neural network. A neural network may include multiple layers (e.g., an input layer, multiple hidden layers, and an output layer). Each layer may include multiple nodes. Each node may be a computational unit with at least one input and output, and the nodes are interconnected. Weights can be set as connections between nodes, and the weights can be adjusted or changed. The influence of corresponding data values ​​on the final result can be determined by using weights to increase, decrease, or maintain corresponding data values. Weighted inputs to nodes included in previous layers can be input to each node included in the output layer. The processing of weighted data from one layer of input to subsequent layers is called propagation.

[0056] In one example, in response to multiple requests received at host processor 110, accelerator 120 may execute multiple neural networks based on kernels generated by host processor 110. Here, the multiple neural networks executed on accelerator 120 may be neural networks with different structures. Optionally, the same neural network may be executed multiple times. In the case of simply executing multiple neural networks in the order of requests, due to the workload characteristics of each neural network, it may be difficult for typical electronic devices and / or host processors to reduce idle time during execution when the hardware resources of accelerator 120 are not used, and tail-latency may occur when later requests are delayed while processing a previous request. To prevent a decrease in the utilization of accelerator 120, scheduling for multiple neural networks can be performed by electronic device 100 and / or host processor 110 of one or more embodiments. Specifically, in response to a request to execute multiple neural networks, electronic device 100 and / or host processor 110 of one or more embodiments may allocate resources for processing each neural network based on the status of accelerator 120, thereby minimizing idle time occurring during execution and improving the utilization of accelerator 120. For clarity, neural networks can also be referred to as models.

[0057] Examples of one or more embodiments described herein may generate multiple candidate kernels for running a specific layer on accelerator 120, and a single candidate kernel that can most efficiently utilize the resources of accelerator 120 may be selected and run from among the multiple candidate kernels. In this way, even when multiple models are executed on accelerator 120, the electronic device 100 and / or host processor 110 of one or more embodiments may improve performance by minimizing contention between models and maximizing the resource utilization of accelerator 120. Because multiple models are executed concurrently, the utilization of accelerator 120 is improved. Examples may select the kernel that can most efficiently utilize accelerator 120 at runtime, thereby minimizing contention between models and maximizing resource utilization, thus improving performance.

[0058] Figure 2 An example of resource allocation processing for performing multi-model operations on a multi-core accelerator is shown.

[0059] Figure 2 An example is shown where compiler 210 can generate multiple kernels at design time and scheduler 220 can assign one of the multiple kernels to accelerator 230 based on the runtime state of accelerator 230. Figure 2 For clarity, it can be assumed that each of Model 1 to Model 3 is requested at a specific time.

[0060] During design, in response to a request to execute Model 1, compiler 210 can generate multiple candidate kernels for each layer included in Model 1 that can run on accelerator 230. Various methods can be used to run a specific layer on accelerator 230, and the kernels can vary accordingly. For example, a specific layer can be processed quickly using all kernels included in accelerator 230, or it can be processed slowly using only a single kernel included in accelerator 230. Optionally, the form of operation of accelerator 230 for running a specific layer can vary depending on the method of operation for that specific layer, based on a set data reuse pattern or tiling factor. Although accelerator 230 can have various forms of operation, it can process the same layer. Therefore, the output data of corresponding layers can be determined to be identical.

[0061] The compiler 210 can generate multiple candidate kernels, in which the number of kernels used when the corresponding layer is running on the accelerator 230, the data reuse mode, and any one or any combination of the tiling factor are different.

[0062] Multiple candidate kernels can be generated based on multiple kernels that are estimated to have different runtimes among the multiple kernels available when running on accelerator 230. For example, when ten kernels are included in accelerator 230, one to ten kernels can be used to run a specific layer. Here, when the estimated runtime decreases significantly as the number of available kernels increases from 1 to 6, and the change in estimated runtime is not significant regardless of whether the number of kernels increases from 7 to 10, multiple candidate kernels can be generated only for the number of kernels from 1 to 6.

[0063] Furthermore, the data reuse patterns used to generate multiple candidate kernels may include patterns for reusing input feature maps, reusing weights, and reusing output feature maps when running on accelerator 230. Here, the input feature maps may be data input to the corresponding layers, and the output feature maps may be data output from the corresponding layers. Additionally, the tiling factor used to generate multiple candidate kernels may represent the data processing unit determined based on the data reuse patterns, and may be represented as, for example, a four-dimensional (4D) data size (such as the kernel's width, height, channels, and tensors).

[0064] The compiler 210 can generate kernel information for each candidate kernel. The kernel information may include resource usage information regarding the resources used when the corresponding candidate kernel is run on the accelerator 230. For example, the kernel information may include the number of cores used to run the corresponding candidate kernel, data reuse mode, tiling factor, estimated runtime, memory access mode, and / or memory usage.

[0065] Whenever a request to execute a corresponding model is received, compiler 210 can generate multiple candidate kernels and corresponding kernel information for each layer included in the corresponding model, and transmit the multiple candidate kernels and corresponding kernel information to scheduler 220. In one example, compiler 210 can be implemented as the aforementioned host processor 110, or included in the aforementioned host processor 110.

[0066] During runtime, scheduler 220 can select a single candidate kernel from multiple candidate kernels based on the status information of accelerator 230 and the corresponding kernel information, and can allocate the selected candidate kernel to accelerator 230. Scheduler 220 can also select a single candidate kernel from multiple candidate kernels for a specific layer that maximizes the utilization of accelerator 230 given its resources, and can allocate the selected candidate kernel to the remaining resources of accelerator 230. Status information of accelerator 230 can be transmitted from accelerator 230 to scheduler 220. The status information of accelerator 230 may include, for example, any one or any combination of: resource usage information of accelerator 230, kernel information of the layer running on accelerator 230, and information about subsequent layers running on accelerator 230.

[0067] Scheduler 220 can allocate resources (e.g., cores, memory, and / or memory bandwidth) required to run a selected kernel on accelerator 230 based on the resource availability of accelerator 230. When the kernel finishes running, scheduler 220 can continuously track the current available resource status of accelerator 230 while retrieving the allocated resources.

[0068] Scheduler 220 can be invoked and performs scheduling whenever a layer running on accelerator 230 completes. Furthermore, scheduler 220 can pre-estimate the amount of time needed to run kernels on accelerator 230 based on kernel information received in advance from compiler 210. Therefore, scheduler 220 can pre-execute scheduling without additional model requests. Here, although all scheduling can be pre-executed, rescheduling may be necessary when a new request is received. The timing of scheduling can be selected based on the resource status of the hardware executing scheduler 220. For example, when hardware resources are sufficient, total throughput can be achieved by pre-executing all scheduling. Conversely, when hardware resources are insufficient or if constraints exist (such as hardware power requirements) (e.g., when reducing power consumption is important), scheduling can be performed whenever each layer's execution completes. As another example, scheduler 220 can be invoked to perform scheduling when the execution of a new model is requested concurrently with the execution of a previous model, and the corresponding new model can be executed using the remaining resources of accelerator 230. Here, whether the new model can be executed using the remaining resources of the accelerator 230 can be determined based on whether all or a portion of the layers of the new model can be run using the remaining resources. Furthermore, various determination criteria can be used without limitation.

[0069] In one example, scheduler 220 may be implemented as or included in the aforementioned host processor 110, or when accelerator 230 includes multi-level cores, scheduler 220 may be implemented as or included in the top core of accelerator 230. The hardware device implementing compiler 210 and / or scheduler 220 may herein be referred to as a data processing device.

[0070] Figure 3 An example of kernel information is shown.

[0071] Figure 3 An example of kernel information generated by a compiler (e.g., compiler 210) is shown. Kernel information for each candidate kernel may include any one or any combination of the following: the number of kernels used to run the corresponding candidate kernel, data reuse mode, tiling factor, estimated runtime, memory access mode, and memory usage. Figure 3 Kernel information includes the number of cores, data reuse pattern, and tiling factor. Figure 3 In this context, IF represents the mode of reusing the input feature map, W represents the mode of reusing the weights, and OF represents the mode of reusing the output feature map. Figure 3 In this model, the tiling factor (e.g., "(8,1,2,2)") can correspond to the 4D data size (such as the kernel's width, height, channels, and tensors). In Model 2, Layer 1 and Layer 2 can be run in a fused form through a single kernel, multiple kernels can be generated for Layer 1 and Layer 2, and kernel information corresponding to multiple kernels can be determined.

[0072] Figures 4 to 6 An example of the process of selecting a single candidate kernel from multiple candidate kernels is shown.

[0073] Figure 4 This is a flowchart illustrating an example of a scheduler (e.g., scheduler 220) initially selecting a layer to run on the accelerator and then selecting a single candidate kernel from a plurality of candidate kernels corresponding to the selected layer.

[0074] Reference Figure 4 In operation 410, the scheduler can verify the existence of sufficient idle resources to run a single layer. That is, for example, it can determine whether there are sufficient resources to guarantee that a single kernel can run on the accelerator.

[0075] In operation 420, the scheduler verifies whether a runnable layer exists in the queue. The layer to be run can be included in the queue. If a runnable layer exists in the queue, operation 430 can be executed. If no runnable layer exists in the queue, the scheduler's operation can be terminated.

[0076] In operation 430, the scheduler can select the layer to run from the runnable layers stored in the queue. For example, when multiple models have different priorities, the layer belonging to the highest priority model can be selected. However, this is provided only as an example. See reference. Figure 4 The example shown on the right includes a runnable layer consisting of layer 2 of model 1 and a fusion layer 1 and 2 of model 2. One of these can be selected as the layer to run on the accelerator.

[0077] In operation 440, the scheduler can select a single candidate core estimated to have the best performance from multiple candidate cores for the selected layer. The scheduler can select the candidate core estimated to have the best accelerator performance when running each candidate core, based on the current state of the accelerator and the kernel information of the candidate cores. Performance criteria, based on user selection or circumstances, can include any one or any combination of throughput, latency, fairness, power consumption, and accelerator utilization for each model. Figure 4 In the example shown on the right, a single candidate kernel can be selected from multiple candidate kernels of layer 2 of the selected model 1.

[0078] For example, a candidate kernel estimated to have the best performance can be selected based on any one of the following: a greedy scheme, an equal resource allocation scheme, an overall system performance priority ranking scheme, and a power minimization scheme.

[0079] A greedy scheme can represent a method of selecting the kernel estimated to have the shortest runtime from among the runnable candidate kernels within the accelerator's remaining resources. Here, remaining resources can refer only to currently free resources that are not actually used, or to resources currently in use in another running kernel but estimated to be remaining resources. For example, a greedy scheme can repeatedly select a single candidate kernel estimated to have the shortest runtime from multiple candidate kernels in one or more layers for subsequent processing.

[0080] An equal resource allocation scheme can be represented as dividing all the accelerator's idle resources into multiple concurrently runnable layers, allocating resources equally to each layer, and selecting the kernel estimated to have the relatively shortest runtime from the runnable candidate kernels within a specific layer of allocated resources. For example, if the total number of cores in the accelerator is 20 and four models are to be run (i.e., executed), then five cores can be allocated to each model, and each model can be run based on a single candidate kernel from the multiple candidate kernels of the layer, where the allocated five cores are estimated to have the shortest runtime.

[0081] A global system performance prioritization scheme can represent a scheme for selecting a combination of kernels estimated to have the shortest relative average runtime (e.g., geometric mean) when selecting candidate kernels for each currently runnable layer. For example, unlike a greedy scheme that selects kernels estimated to have the shortest runtime, a global system performance prioritization scheme can be a scheme that finds a combination of kernels to use the shortest execution-to-completion time from an overall perspective.

[0082] A power minimization scheme can be defined as a scheme for selecting the kernel that uses the least resources from candidate kernels that meet the required or defined Quality of Service (QoS). For example, a power minimization scheme might select the kernel that requires the least resources from kernels that meet a preset QoS, rather than selecting the kernel that is estimated to have the shortest runtime.

[0083] Thus, at runtime, the scheduler of one or more embodiments can adaptively change the number of cores used to run each layer based on the accelerator's resource availability to minimize contention between models and maximize accelerator resource utilization. For example, overall system throughput can be increased by prioritizing overall system performance in the absence of specific requirements, and scheduling can be performed to quickly complete a particular model by selectively applying a greedy scheme to a particular model in response to an urgent request to execute that particular model.

[0084] Figure 5 This is a flowchart illustrating an example of a process in which a scheduler (e.g., scheduler 220) selects a kernel to run on the accelerator from all available candidate kernels, rather than initially selecting the layer to run.

[0085] Reference Figure 5 In operation 510, the scheduler verifies the existence of sufficient idle resources to run a single layer. In operation 520, the scheduler verifies the existence of runnable candidate kernels in the queue. Candidate kernels to be run can be included in the queue. When runnable candidate kernels exist in the queue, operation 530 can be executed. When no runnable candidate kernels exist in the queue, the scheduler operation can be terminated. In operation 530, the scheduler selects a single candidate kernel from the runnable candidate kernels that is estimated to have the best performance. (See reference...) Figure 5 The example shown on the right allows for the selection of a single candidate kernel from multiple candidate kernels in layer 2 of model 1 and multiple candidate kernels in the fusion layers 1 and 2 of model 2. If there is no priority among the multiple models, the candidate kernel estimated to have the best performance can be selected. The selection of candidate kernels (e.g., referring to...) Figure 4 The foregoing description of operation 440) can be applied to Figure 5 Example operation 530 is provided, and further description is omitted.

[0086] Figure 6 This is a flowchart illustrating an example of a scheduler (e.g., scheduler 220) performing kernel selection based on memory access patterns.

[0087] Reference Figure 6 In operation 610, the scheduler can determine whether the number of idle cores in the accelerator is greater than or equal to the number of cores required by the candidate cores. When it is determined in operation 610 that the number of idle cores in the accelerator is less than the number of cores required by the candidate cores, the scheduler can exclude the corresponding candidate cores in operation 660.

[0088] When it is determined in operation 610 that the number of idle cores in the accelerator is greater than or equal to the number of cores required by the candidate cores, in operation 620, the scheduler may extract combinations of cores that can be mapped to each candidate core. Although a single core can be executed by the same number of cores, the memory access scheme can vary based on the accelerator's architecture. When the accelerator includes hierarchical cores and memory, combinations regarding the number of cores to be used for each cluster can be extracted.

[0089] For example, if five cores form a single cluster and the accelerator comprises four clusters, a total of 20 cores can be used. Here, when using ten cores included in two clusters and two cores in another cluster, it can be verified that three and five of the remaining eight idle cores are included in the two clusters, respectively.

[0090] In operation 630, the scheduler can verify the amount of available memory for each core combination extracted in operation 620. The scheduler can verify the amount of memory that can be used by the idle cores of each cluster.

[0091] In operation 640, the scheduler compares memory access patterns across runnable cores. The scheduler can verify the memory access pattern for each candidate core based on tiling factors and data reuse patterns. A comparison can be performed to determine if there are conflicts between the memory access patterns of the candidate cores and those of the models currently running on the accelerator. If conflicts occur, the extent of the impact on the QoS of both the running and upcoming models can be analyzed.

[0092] In operation 650, the scheduler may select the best runnable candidate kernel based on a scheduling policy. For example, if it is determined in operation 640 that there is no conflict between the memory access mode of the candidate kernel and the memory access mode of the current running model, a candidate kernel may be selected in operation 650. Conversely, if it is determined in operation 640 that there is a conflict between the memory access mode of the candidate kernel and the memory access mode of the current running model, the candidate kernel with the least conflict may be selected in operation 650.

[0093] In one example, when executing multiple models, these models can share a bus corresponding to shared resources, thus memory access patterns can be considered. Memory access patterns can be verified for each of the input feature map, weights, and output feature map. Memory accesses of the models can be performed at desired intervals. The model's memory access patterns can be translated into real-time tasks (e.g., cycles and access times) to determine the demand constraint function used in real-time task studies. If a demand constraint function is used, the impact of different memory access patterns can be verified. Thus, it is possible to pre-verify whether memory accesses of another model have a malicious impact on QoS, which is one of the user requirements, and memory accesses can be prevented accordingly.

[0094] Figure 7 and Figure 8 An example of resource allocation is shown.

[0095] Figure 7 An example of dynamically allocating resources is shown. Figure 7 The upper part shows a list of runnable candidate kernels for each layer. Figure 7 The core allocation section shown in the lower part illustrates the method of allocating eight cores to compute each layer.

[0096] In the case of execution only in Model 1 (e.g., before Layer 1 of Model 1 begins running), five of the eight cores are allocated to run Layer 1 of Model 1, and the remaining three cores are idle. In the case of execution "after" of Layer 1 of Model 1 beginning running under the above conditions, Layer 1 of Model 2 can be run by selecting Core 2, which can make maximum use of the remaining three cores. As described above, the utilization of the accelerator can be effectively improved by adaptively allocating resources based on the current state of the accelerator.

[0097] Figure 8 This illustrates another example of dynamically allocating resources. Figure 8 In the example, it is assumed that the execution of Model 2 is requested in the case that the execution of Model 1 is initially requested and is running on the accelerator.

[0098] When only Model 1 is executed on the accelerator, all ten cores included in the accelerator are allocated to Layer 1 of Model 1 (810). Five of the ten cores are initially allocated to Layer 2-1 to run each of Layers 2-1 and 2-2, and the output data of Layer 1 is transferred in Model 1 to each of Layers 2-1 and 2-2 (820). When execution of Model 2 is requested before running Layer 2-2, a portion of the remaining five cores (e.g., three cores) are allocated to Layer 2-2 of Model 1 (830), and the remaining two cores are allocated to Layer 1 of Model 2 (840). To run Layer 3, where the output data of Layers 2-1 and 2-2 of Model 1 is transferred, five cores are allocated to Model 1, and the remaining five cores are allocated to Model 2. Here, Layers 2 and 3 of Model 2 are merged during the compilation phase and operate as a single merged layer, and the remaining five cores are allocated to the merged layer (850). After the execution of Model 1 is complete, all ten cores are allocated to Model 2 (860). Here, unless a separate new model is requested to be executed, the migration cost can be minimized by keeping the allocation position of the cores assigned to each model at the maximum value.

[0099] As described above, even during the runtime of executing a deep learning model, the methods of one or more embodiments can minimize accelerator idle time and improve accelerator utilization by adaptively changing resource allocation based on accelerator state and by selecting candidate kernels corresponding to appropriate resource allocation (such as the optimal number of kernels when running each layer).

[0100] Figure 9 An example of resource allocation based on memory access patterns is shown.

[0101] Figure 9An example is shown of allocating resources based on the existence or absence of conflicts between memory access patterns of concurrently running models. Each of layer i of model 1 and layer j of model 2 may have various memory access patterns, where a and α may be determined as the most efficient (or optimal) memory access patterns, respectively. When the corresponding models are run independently, each of memory access patterns a and α may represent the most efficient pattern. According to the example, in the case of concurrently running layer i of model 1 and layer j of model 2 based on memory access patterns a and α, respectively, conflicts may occur between memory access patterns (910). That is, due to the overlap of memory access resource usage patterns between model 1 and model 2, conflicts may occur that could lead to one model not using memory access resources. Since conflicts have a malicious impact on accelerator utilization, examples of one or more embodiments can minimize conflicts. Therefore, examples of one or more embodiments can take into account the existence of conflicts between memory access patterns of concurrently running models when allocating accelerator resources. Figure 9 In the example, when a conflict is determined between memory access modes a and α, memory access mode a of layer 1 of model 1 and memory access mode β of layer j of model 2, which does not conflict with model 1, can be selected (920). A candidate kernel corresponding to memory access mode a can be selected to run layer i of model 1, and a candidate kernel corresponding to memory access mode β can be selected to run layer j of model 2. Therefore, a kernel with less impact on memory access can be selected when executing multiple models.

[0102] Figure 10 An example of an electronic device including multiple accelerators is shown.

[0103] Reference Figure 10 The electronic device may include a CPU 1010, multiple NPUs 1020, and a host DRAM 1030. Here, the CPU 1010 may correspond to or be included in the aforementioned host processor 110, and the NPU 1020 may correspond to or be included in the aforementioned accelerator 120.

[0104] In addition to the kernel information in the examples above, when executing at least one model on multiple NPU 1020s, the kernel information generated by the compiler may also include information on the additional resources consumed by using multiple NPU 1020s. Furthermore, the scheduler may perform scheduling based on contention between kernels to be run on multiple NPU 1020s and performance differences due to changes in system architecture.

[0105] exist Figure 10In the electronic device, a model can be executed by at least two NPUs 1020. In this case, the NPUs 1020 can exchange data via the Peripheral Component Interconnect Fast (PCIe) bus. In addition, when the capacity of the on-chip memory (e.g., SRAM) or off-chip memory in the NPU 1020 is insufficient, a host DRAM 1030 can be used.

[0106] For candidate kernels generated by the compiler to utilize multiple NPU 1020s, information regarding the amount of data transfer between NPU 1020s can be added to the corresponding kernel information. For candidate kernels generated to utilize host DRAM 1030s, information regarding host memory usage can be added to the corresponding kernel information.

[0107] Based on this kernel information, the scheduler can select the appropriate kernel for the accelerator situation at runtime. For example, in the case of layers requiring significant operation time, even considering the data migration costs between NPU 1020s, it may be beneficial (or result in faster runtime) to process the corresponding operations in a distributed manner using multiple NPU 1020s, rather than using a single NPU 1020. Therefore, in this case, a kernel with multiple NPU 1020s can be selected.

[0108] Figure 11 An example of a layered structure for an accelerator is shown.

[0109] Figure 11 An example of a layered accelerator 1100 is shown. (See reference...) Figure 11 The accelerator 1100 may include a plurality of processing elements (PEs) and a multilevel memory accessible by at least one of the plurality of processing elements. The multilevel memory may be a common representation of a level 0 (LV0) memory 1111, a level 1 (LV1) memory 1121, and a level 2 (LV2) memory 1131 corresponding to the on-chip memory of the accelerator 1100.

[0110] As one of a plurality of processing elements, processing element 1110 may include LVO memory 1111, LVO direct memory access (DMA) 1113, multiplier-accumulator (MAC) 1115 and LVO controller 1117.

[0111] LV0 memory 1111 can be a memory accessible by the corresponding processing element 1110. LV0 DMA 1113 can control the input and / or output data of LV0 memory 1111 in response to instructions from LV0 controller 1117.

[0112] MAC 1115 can perform operations assigned to the workload of processing element 1110. Furthermore, MAC 1115 can perform multiplication-accumulation operations on given data, or apply activation functions to given data. For example, activation functions may include sigmoid, hyperbolic tangent (tanh), and corrected linear unit (ReLU).

[0113] The LV0 controller 1117 may be a device configured to control components included in the processing element 1110. For example, the LV0 controller 1117 may control the LV0 memory 1111, the LV0 DMA 1113, and the MAC 1115.

[0114] The foregoing description relating to processing element 1110 may be applied as is to each of the plurality of processing elements included in accelerator 1100.

[0115] In one example, multiple processing elements can be clustered into n processing elements. Here, n represents a natural number greater than 1 and less than the number of processing elements included in accelerator 1100. A portion of the multiple processing elements included in accelerator 1100 can be clustered, described based on clustered processing elements 1120.

[0116] The cluster's processing elements 1120 can share a single LV1 memory 1121. The LV1 memory 1121 can be accessed by the cluster's processing elements 1120. For example, although the operations performed in the first and second processing elements of the cluster's processing elements 1120 may be different from each other, a portion of the data used for the corresponding operations may be common. The efficiency of the accelerator 1100 is improved by storing common data in the LV1 memory 1121 for sharing between the first and second processing elements, rather than sharing common data in the LV0 memory 1111 of each of the first and second processing elements. Figure 11 In the example, each of the processing elements adjacent to LV1 memory 1121 has access to LV1 memory 1121.

[0117] Furthermore, the multiple processing elements 1130 as a whole can share the LV2 memory 1131. The LV2 memory 1131 can be accessed by the multiple processing elements included in the accelerator 1100. For example, among the multiple processing elements included in the accelerator 1100, there may be processing elements that are not clustered but have common data required to perform operations. Such processing elements can efficiently share common data through the LV2 memory 1131 instead of sharing corresponding data through the LV1 memory 1121, thereby improving the efficiency of the accelerator 1100.

[0118] Each of the processing elements may include and access LV0 memory 1111, LV1 memory 1121 adjacent to the processing element, and LV2 memory 1131 of accelerator 1100, and such memory may be used when performing assigned workloads. Accelerator 1100 may include multi-level memory and hierarchical memory structure. Furthermore, the DMA and controller included in accelerator 1100 may have hierarchical multi-level structures.

[0119] exist Figure 11 In the example, multiple processing elements included in accelerator 1100 can concurrently execute four workloads (e.g., a first workload, a second workload, a third workload, and a fourth workload). For example, workloads using relatively large amounts of computation can be assigned to and processed by a relatively large number of processing elements, and workloads using relatively small amounts of computation can be assigned to and processed by a relatively small number of processing elements.

[0120] although Figure 11 The illustration shows 64 processing elements clustered into eight and three levels of memory used to execute four workloads; however, this is provided as an example only for clarity of description. Various numbers of processing elements, workloads, and levels can be applied without limitation.

[0121] Figure 12 An example of a data processing method is shown.

[0122] Reference Figure 12 The data processing method executed by at least one processor includes: operation 1210 of receiving a request to execute a model on an accelerator, operation 1220 of generating multiple candidate kernels for each layer included in the model, and operation 1230 of assigning a single candidate kernel to the accelerator, the single candidate kernel being selected from multiple candidate kernels for the layer to be run on the accelerator based on corresponding kernel information and accelerator state information.

[0123] The examples described herein can be broadly applied to server-oriented product lines that employ at least one System-on-Chip (SoC), in which multiple accelerator cores and shared memory in a hierarchical structure are connected in a cluster-based architecture to accelerate AI processing.

[0124] The above reference Figures 1 to 11 The description can be applied as is. Figure 12 Each of operations 1210 to 1230 is described, and further description is omitted.

[0125] Figure 13 and Figure 14 An example of an electronic device is shown.

[0126] Reference Figure 13 The electronic device can be a server 1300.

[0127] Server 1300 may be a separate device distinct from user terminals controlled by users, and may communicate with at least one user terminal via wired and / or wireless networks. Server 1300 may receive requests concurrently from multiple users using their own terminals. Host processor 1310 included in server 1300 may generate multiple candidate kernels for each layer included in the requested model. Accelerator 1320 may run the desired layer based on a single candidate kernel selected from multiple candidate kernels for that layer based on corresponding kernel information and state information of accelerator 1320. Each inference result obtained by executing the model on accelerator 1320 may be returned to each corresponding user terminal. Examples of user terminals may include various computing devices (such as smartphones, tablets, laptops, and personal computers), various wearable devices (such as smartwatches and smart glasses), various home appliances (such as smart speakers, smart TVs, and smart refrigerators), smart vehicles, smart self-service machines, Internet of Things (IoT) devices, drones, and / or robots.

[0128] Reference Figure 14 The electronic device may be a user terminal 1400. Although for the sake of clarity, Figure 14 User terminal 1400 is shown as a smartphone, but this is provided as an example only. Any user-controllable device can be used without limitation. User terminal 1400 can directly obtain requests from the user, and host processor 1410 included in user terminal 1400 can generate multiple candidate kernels for each layer included in the requested model. Accelerator 1420 can run the desired layer based on a single candidate kernel, which is selected from the multiple candidate kernels for that layer based on corresponding kernel information and accelerator 1420 state information. The inference results obtained through the above-described execution process can be provided to the user, or subsequent operations of user terminal 1400 can be performed based on the inference results.

[0129] They are implemented as Figure 13 and Figure 14 The server 1300 and user terminal 1400 of the electronic device can process Figures 1 to 12 The aforementioned operations. Therefore, further description is omitted. Furthermore, the electronic device used herein may be referred to as an accelerator system including a host processor and an accelerator.

[0130] Refer to this Figures 1 to 14The described electronic devices include: a host processor, an accelerator, an accelerator chip, an accelerator core, on-chip memory, off-chip memory, a compiler, a scheduler, a CPU, an NPU, host DRAM, cluster processing elements, LV1 memory, processing elements, LV0 memory, LV0 DMA, a MAC, an LV0 controller, a server, and a user terminal. Specifically, electronic device 100 includes a host processor 110, an accelerator 120, an accelerator chip 130, an accelerator core 131, on-chip memory 132, off-chip memory 140, a compiler 210, a scheduler 220, an accelerator 230, a CPU 1010, an NPU 1020, host DRAM 1030, an accelerator 1100, cluster processing elements 1120, LV1 memory 1121, processing elements 1110, LV0 memory 1111, and LV0 DMA. 1113, MAC 1115, LV0 controller 1117, server 1300, host processor 1310, accelerator 1320, user terminal 1400, host processor 1410, accelerator 1420, devices, units, modules, apparatuses, and other components are implemented or represented by hardware components. Examples of hardware components that can be used to perform the operations described in this application include, where appropriate, controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components performing the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer may be implemented by one or more processing elements, such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field-programmable gate arrays, programmable logic arrays, microprocessors, or any other means or combination of means configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, the processor or computer includes or is connected to one or more memories storing instructions or software executed by the processor or computer. Hardware components implemented by the processor or computer may execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) to perform the operations described in this application. The hardware components may also access, manipulate, process, create, and store data in response to the execution of instructions or software. For the sake of brevity, the singular terms “processor” or “computer” may be used in the description of the examples described in this application, but in other examples, multiple processors or computers may be used, or a processor or computer may include multiple processing elements or multiple types of processing elements or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller.One or more hardware components may be implemented by one or more processors or processors and controllers, and one or more other hardware components may be implemented by one or more other processors, or additional processors and additional controllers. One or more processors, or processors and controllers, may implement a single hardware component or two or more hardware components. Hardware components may have any one or more different processing configurations, examples of which include: a single processor, a discrete processor, a parallel processor, Single Instruction Single Data (SISD) multiple processing, Single Instruction Multiple Data (SIMD) multiple processing, Multiple Instruction Single Data (MISD) multiple processing, and Multiple Instruction Multiple Data (MIMD) multiple processing.

[0131] Figures 1 to 14 The methods for performing the operations described in this application, as shown, are executed by computing hardware (e.g., by one or more processors or a computer), which is implemented as described above to execute instructions or software to perform the operations performed by the methods described in this application. For example, a single operation or two or more operations may be executed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be executed by one or more processors or a processor and a controller, and one or more other operations may be executed by one or more other processors, or additional processors and additional controllers. One or more processors, or a processor and a controller, may execute a single operation or two or more operations.

[0132] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above can be written as computer programs, code segments, instructions, or any combination thereof to individually or collectively instruct or configure one or more processors or computers to operate as a machine or special-purpose computer to perform the operations performed by the hardware components and methods described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) that is directly executed by one or more processors or computers. In another example, the instructions or software include high-level code that is executed by one or more processors or computers using an interpreter. The instructions or software can be written in any programming language based on the block diagrams and flowcharts shown in the accompanying drawings and the corresponding descriptions in the specification, which disclose algorithms for performing the operations performed by the hardware components and methods described above.

[0133] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above, as well as any associated data, data files, and data structures, may be recorded, stored, or fixed in, or may be recorded, stored, or fixed on, one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage devices, hard disk drives (HDDs), solid-state drives (SSDs), flash memory, card-type storage devices (such as multimedia cards or microcards (e.g., Secure Digital (SD) or Extreme Digital (XD))), magnetic tape, floppy disks, magneto-optical data storage devices, optical data storage devices, hard disks, solid-state drives, and any other means configured to store instructions or software and any associated data, data files, and data structures in a non-transitory manner and to provide said instructions or software and any associated data, data files, and data structures to one or more processors or computers so that one or more processors or computers can execute the instructions. In one example, the instructions or software and any associated data, data files, and data structures are distributed across a networked computer system, such that the instructions and software and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner by one or more processors or computers.

[0134] Although this disclosure includes specific examples, it will be clear upon understanding this disclosure that various changes in form and detail may be made to these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are to be considered descriptive only and not for limiting purposes. The description of features or aspects in each example will be considered applicable to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order and / or if components in the described system, architecture, apparatus, or circuit are combined in a different manner and / or replaced or supplemented by other components or their equivalents. Therefore, the scope of the disclosure is not limited by the specific embodiments but by the claims and their equivalents, and all variations within the scope of the claims and their equivalents will be construed as included in the disclosure.

Claims

1. A data processing method implemented by a processor, the data processing method comprising: Receive a request to execute the model on the accelerator; Multiple candidate kernels are generated for each of the multiple layers included in the model; as well as A single candidate kernel is assigned to the accelerator. This single candidate kernel is selected from multiple candidate kernels generated for the layer to be run on the accelerator, based on corresponding kernel information and accelerator state information. The kernel includes instructions that are executed at the accelerator. The generation step includes: generating the plurality of candidate kernels such that for each of the plurality of candidate kernels, the number of kernels used when the corresponding layer of the candidate kernel is running on the accelerator, any one or any combination of the number of kernels, the data reuse mode and the tiling factor is different from the number of kernels used when the corresponding layer of another candidate kernel is running on the accelerator.

2. The data processing method according to claim 1, wherein, The generation process includes generating multiple candidate kernels that run each layer using different accelerator resource usage.

3. The data processing method according to claim 1, wherein, The number of cores used to execute the model, which are included among the multiple cores in the accelerator, varies based on the accelerator's resource availability at runtime.

4. The data processing method according to claim 1, wherein, The allocation steps include: selecting a single candidate kernel from the plurality of candidate kernels that is configured to maximize the utilization of the accelerator given the accelerator's resources, and allocating the selected candidate kernel to the remaining resources of the accelerator.

5. The data processing method according to claim 1, wherein, The accelerator's status information includes any one or any combination of the accelerator's resource usage information, kernel information of the layers running on the accelerator, and information about subsequent layers of the layers running on the accelerator.

6. The data processing method according to claim 1, wherein, The allocation process includes: selecting candidate kernels based on determining whether a conflict has occurred or not with the data access patterns of the layers of the model running on the accelerator, and allocating the selected candidate kernels to the accelerator.

7. The data processing method according to claim 6, wherein, The allocation process includes: in response to a conflict, selecting any one of the following as a candidate kernel: Candidate kernels that satisfy the quality of service (QoS) of either the layer currently running on the accelerator or the layer to be run on the accelerator; and The candidate kernel with the least impact on service quality among the plurality of candidate kernels; and Assign the selected candidate kernels to the accelerator.

8. The data processing method according to any one of claims 1 to 7, wherein, The allocation steps include any one of the following: A single candidate kernel is assigned in response to the completion of at least one of the layers of the model running on the accelerator; as well as In response to receiving a new request, the accelerator uses its remaining resources to execute the model corresponding to the new request and allocate a single candidate kernel.

9. The data processing method according to any one of claims 1 to 7, wherein, The allocation steps are performed by any one of the host processors and accelerators configured to generate the plurality of candidate cores.

10. The data processing method according to any one of claims 1 to 7, wherein, The allocation process includes: allocating a single candidate kernel to the accelerator based on any one of the multiple candidate kernels, such as a greedy scheme, an equal resource allocation scheme, an overall system performance priority sorting scheme, and a power minimization scheme.

11. The data processing method according to any one of claims 1 to 7, wherein, The accelerator comprises multiple cores and is configured to execute at least one model.

12. A non-transitory computer-readable recording medium storing instructions, which, when executed by one or more processors, configure the one or more processors to perform the data processing method of any one of claims 1 to 11.

13. A data processing apparatus, the data processing apparatus comprising: One or more processors, The one or more processors are configured to: Receive a request to execute the model on the accelerator; Multiple candidate kernels are generated for each of the multiple layers included in the model; as well as A single candidate kernel is assigned to the accelerator. This single candidate kernel is selected from multiple candidate kernels generated for the layer to be run on the accelerator, based on corresponding kernel information and accelerator state information. The kernel includes instructions that are executed at the accelerator. The generation step includes: generating the plurality of candidate kernels such that for each of the plurality of candidate kernels, the number of kernels used when the corresponding layer of the candidate kernel is running on the accelerator, any one or any combination of the number of kernels, the data reuse mode and the tiling factor is different from the number of kernels used when the corresponding layer of another candidate kernel is running on the accelerator.

14. An electronic device, the electronic device comprising: The host processor is configured to receive requests for executing a model on the accelerator and to generate multiple candidate kernels for each of the multiple layers included in the model. as well as The accelerator is configured to run a layer determined to be run based on a single candidate kernel, which is selected from multiple candidate kernels generated for the layer to be run based on corresponding kernel information and accelerator state information. The kernel includes instructions that are executed at the accelerator. In the generation step, the host machine is configured to generate the plurality of candidate kernels such that for each of the plurality of candidate kernels, the number of kernels used when the corresponding layer of the candidate kernel is running on the accelerator, any one or any combination of the number of kernels, the data reuse mode and the tiling factor is different from the number of kernels used when the corresponding layer of another candidate kernel is running on the accelerator.

15. The electronic device according to claim 14, wherein, For the generation step, the host is configured to generate multiple candidate kernels that run each layer using different accelerator resource usage.

16. A processor-implemented data processing method, the data processing method comprising: Multiple kernels are generated for each of the multiple layers included in each of the first and second models; Assign a single kernel of the first model to the accelerator. The single kernel of the first model is selected from multiple kernels generated for the layer of the first model to be run on the accelerator based on the corresponding kernel information and the state information of the accelerator. Assign a single kernel of the second model to the accelerator. The single kernel of the second model is selected from multiple kernels generated for the layer of the second model to be run on the accelerator based on the corresponding kernel information and the state information of the accelerator. A single kernel of the first model is allocated and run on a certain number of cores available in the accelerator's resources; In response to the start of the run, a single core of the allocated second model is run on the remaining number of cores of the accelerator's available resources; Generate one or more output feature maps based on kernel execution. The kernel includes instructions that are executed at the accelerator. The generation steps include: generating the plurality of kernels for each of the plurality of layers included in each of the first model and the second model, such that for each of the plurality of kernels, the number of kernels used when the corresponding layer of the kernel is running on the accelerator, any one or any combination of the number of kernels, the data reuse mode and the tiling factor is different from the number of kernels used when the corresponding layer of another kernel is running on the accelerator.

17. The data processing method according to claim 16, wherein, The steps for running the kernel of the second model include: running the kernel of the second model in response to determining that there will be no conflict between memory access modes during the operation of the kernel of the first model and the kernel of the second model.

18. The data processing method according to claim 16, wherein, For each of the kernels in the first model and the second model, a kernel for execution is selected from the plurality of kernels based on kernel information, which includes any one or more of the following: The number of accelerator cores used to run the kernel; Data reuse patterns include any one or more of the reuse of input feature maps, weights, and output feature maps during kernel runtime; and Tiling factor, including the multidimensional data size of the kernel.

Images