Accelerator, method of operating accelerator, and accelerator system

Abstract

Disclosed are an accelerator, an operation method of the accelerator, and an accelerator system. The method includes receiving one or more workloads allocated by a host controller, determining reuse data of the workloads based on hardware resource information and / or memory access cost of the accelerator when a plurality of processing units included in the accelerator execute the workloads, and providing a result of executing the workloads.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2020-0023750, filed with the Korean Intellectual Property Office on February 26, 2020, the entire disclosure of which is incorporated herein by reference for all purposes. Technical Field

[0002] The following description relates to the operation of the accelerator and the system including the accelerator. Background Technology

[0003] With the development of artificial intelligence (AI) technology, there is a demand for dedicated AI hardware that can perform inference and learning through computation. Various devices are being developed as hardware specifically for implementing AI.

[0004] Such dedicated hardware for AI can be implemented by, for example, central processing units (CPUs) and graphics processing units (GPUs), or by application-specific integrated circuits (ASICs) and field-programmable gate arrays (FPGAs) that can be repurposed. Summary of the Invention

[0005] This summary is provided to introduce, in a simplified form, the selection of concepts that will be further described in the detailed embodiments below. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to help determine the scope of the claimed subject matter.

[0006] In one general aspect, an accelerator operating method includes: receiving one or more workloads assigned by a host controller, the host controller being configured to control an accelerator comprising a plurality of processing elements; determining reusable data of the workloads based on at least one of hardware resource information of the accelerator and memory access costs while the plurality of processing elements are executing the workloads; and providing the results of executing the workloads to the host controller.

[0007] The steps to determine reusable data may include: determining the accelerator's access to external memory with the fewest reusable data as the processing element performs its workload.

[0008] The hardware resource information of the accelerator may include at least one of the following: information on the use of multi-level memory in the accelerator, information on the use of processing elements, or information on system cache.

[0009] The multilevel memory may include at least one of the following: level 0 memory, accessible by one of the processing elements; level 1 memory, accessible by a subset of the processing elements, wherein the subset is a number greater than one; or level 2 memory, accessible by the processing elements.

[0010] The steps to determine reusable data may also include: dynamically determining reusable data based on workload characteristics.

[0011] Memory access costs can include the access costs of external memory and the access costs of multi-level memory included in the accelerator.

[0012] The cost of accessing the external memory of an accelerator can be greater than the cost of accessing multi-level memory.

[0013] For a portion of memory in a multilevel memory that is shared by a larger number of processing elements, the access cost of multilevel memory can increase.

[0014] At least one of hardware resource information and memory access cost can be determined by offloading to an extension of Direct Memory Access (DMA), which is configured to control data input to or output from multilevel memory.

[0015] The steps for determining reusable data may include: determining a chunking method based on accelerator hardware resource information for the data input to be applied to perform workloads across multiple workloads; and determining the size of the reusable data based on the determined chunking method.

[0016] The steps of determining reusable data may include: determining reusable data for each layer of a neural network that corresponds to at least one of the multi-level memory and the workload.

[0017] Reusable data can be stored in a multi-level memory within the accelerator and may not be sent to the accelerator's external memory.

[0018] Each of the processing elements may include: a Level 0 memory accessible by the corresponding processing element; a Level 0 DMA configured to: control data input and output to the Level 0 memory, and monitor and / or parse data input to or output from the Level 0 memory; a multiplier-accumulator (MAC) configured to perform operations related to a workload among a plurality of workloads assigned to the processing element; and a Level 0 controller configured to control one or a combination of the Level 0 memory, Level 0 DMA, and MAC.

[0019] The accelerator may be located in a user terminal or a server. Based on the workload, a neural network is used to input the data to be identified into the user terminal, and the server is configured to receive the data to be identified from the user terminal.

[0020] In another general aspect, an accelerator includes: a plurality of processing elements configured to execute one or more workloads assigned by a host controller; and a multi-level memory configured to be accessed by at least one of the processing elements. When the processing elements execute workloads, reused workload data may be determined based on at least one of the accelerator's hardware resource information and memory access costs, and may be stored in the multi-level memory.

[0021] In another general aspect, an accelerator system includes: an accelerator comprising a plurality of processing elements configured to perform one or more workloads, and a multi-level memory with different access costs; external memory of the accelerator; and a host controller configured to distribute workloads to the accelerator. When the processing elements perform a workload, the accelerator may determine reusable data for the workload based on at least one of the accelerator's hardware resource information and memory access costs.

[0022] In another general aspect, an accelerator operating method includes: receiving a workload assigned by a host controller, the host controller being configured to control an accelerator including processing elements; dynamically determining reusable data for the workload based on any two of the following: hardware resource information of the accelerator, memory access costs of the accelerator, and characteristics of the workload, while the processing elements execute consecutive workloads; and providing the results of executing the workloads to the host controller. The reusable data is utilized in subsequent operations.

[0023] The steps to determine reusable data include: when the processing element performs a workload, determining that the accelerator accesses external memory with the fewest reusable data.

[0024] The hardware resource information of the accelerator may include: usage information of the multi-level memory contained in the accelerator. The multi-level memory may include: Level 0 memory, accessible by one of the processing elements; Level 1 memory, accessible by a subset of the processing elements, wherein the subset is a number greater than one; and Level 2 memory, accessible by the processing elements.

[0025] Memory access costs can include the access costs of the accelerator’s external memory and the access costs of the multilevel memory included in the accelerator, with the access costs of external memory being greater than those of multilevel memory.

[0026] The access cost of multilevel memory can increase based on the number of processing elements accessing each of the level 0, level 1, and level 2 memories in the multilevel memory.

[0027] Other features and aspects will become clear from the following detailed description, drawings, and claims. Attached Figure Description

[0028] Figure 1 This is a diagram illustrating an example of an accelerator system.

[0029] Figure 2 This is a diagram showing an example of an accelerator.

[0030] Figure 3 This is a diagram illustrating examples of multi-level memory and external memory.

[0031] Figure 4 This is a diagram illustrating an example of determining how to reuse data.

[0032] Figure 5 and Figure 6 This is a diagram illustrating an example of an accelerator system.

[0033] Figure 7 This is a flowchart illustrating an example of how an accelerator operates.

[0034] Throughout the accompanying drawings and detailed embodiments, unless otherwise described or provided, the same reference numerals will be understood to denote the same elements, features, and structures. The drawings may not be to scale, and for clarity, illustration, and convenience, the relative sizes, proportions, and depictions of elements in the drawings may be exaggerated. Detailed Implementation

[0035] The following detailed embodiments are provided to aid the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to the order of those operations set forth herein, but may be changed as will become clear upon understanding this disclosure, except for operations that must occur in a specific order.

[0036] The features described herein may be implemented in different forms and should not be construed as limited to the examples described herein. Rather, the examples provided herein are merely to illustrate some of the many feasible ways of implementing the methods, apparatus, and / or systems described herein, which will be clear upon understanding the disclosure of this application.

[0037] The terminology used herein is for the purpose of describing various examples only and is not intended to limit disclosure. Unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. The terms “comprising,” “including,” and “having” indicate the presence of the stated features, quantities, operations, components, elements, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, quantities, operations, components, elements, and / or combinations thereof.

[0038] Although terms such as “first,” “second,” and “third” may be used herein to describe various components, assemblies, regions, layers, or parts, these components, assemblies, regions, layers, or parts are not limited by these terms. Rather, these terms are used only to distinguish one component, assembly, region, layer, or part from another. Thus, without departing from the teaching of the examples described herein, the first component, first assembly, first region, first layer, or first part referred to in the examples may also be referred to as a second component, second assembly, second region, second layer, or second part.

[0039] Throughout this specification, when a component is described as "connected to" or "bonded to" another component, that component may be directly "connected to" or directly "bonded to" another component, or there may be one or more other components in between. In contrast, when an element is described as "directly connected to" or "directly bonded to" another element, there may not be any other element in between. Similarly, similar expressions, such as "between," "immediately between," "adjacent to," and "closely adjacent to," should be interpreted in the same manner. As used herein, the term "and / or" includes any one and any combination of any two or more of the associated listed items.

[0040] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains and as commonly understood based on an understanding of the disclosure of this application. Unless expressly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and in the disclosure of this application, and shall not be interpreted in an idealized or overly formalized sense.

[0041] Furthermore, in the description of the exemplary embodiments, descriptions will be omitted where a detailed description of a structure or function known therefrom after understanding the disclosure of this application would lead to a vague interpretation of the exemplary embodiments. Hereinafter, the examples will be described in detail with reference to the accompanying drawings, in which the same reference numerals always denote the same elements.

[0042] Figure 1 This is a diagram illustrating an example of an accelerator system.

[0043] Reference Figure 1 The accelerator system 100 may include a host controller 110, an accelerator 120, an external memory controller 130, and an external memory 140. The host controller 110, the accelerator 120, the external memory controller 130, and the external memory 140 may communicate with each other via a bus 150.

[0044] Host controller 110 may be a means configured to control the operation of various components included in accelerator system 100, and may include, for example, a central processing unit (CPU). In one example, host controller 110 may assign one or more workloads to accelerator 120. Workloads may be instructions to accelerator 120 to execute neural networks for, for example, object recognition, speech recognition, pattern recognition, computer vision, and machine translation. Host controller 110 may assign workloads to accelerator 120 based on one or more requested jobs or tasks.

[0045] Accelerator 120 may be an artificial intelligence (AI) accelerator configured to execute neural networks and infer from input data based on assigned workloads. Accelerator 120 may be a separate processor distinct from host controller 110. That is, accelerator 120 may execute one or more workloads assigned by host controller 110 simultaneously. Accelerator 120 may process workloads more efficiently than host controller 110, which is available for general computing purposes.

[0046] A neural network may include multiple layers. In one example, a neural network may include an input layer, multiple hidden layers, and an output layer. Each layer may include multiple nodes, each node being called an artificial neuron. Each node may represent a computational unit with at least one input and output, and nodes may be connected to each other. Weights may be set for the connections between nodes, and these weights may be adjusted or changed. Weights may increase, decrease, or maintain relevant data values ​​to determine the impact of those data values ​​on the final result. Weighted inputs from nodes included in a previous layer may be fed into each node included in the output layer. The processing of weighted data from one layer to subsequent layers of that layer may be referred to as propagation.

[0047] Neural network-based operations can be performed in accelerator 120. To perform these operations, multiple processing units (or multiple processing elements) and multi-level memory included in accelerator 120 can be used. Multi-level memory can be memory accessible by at least one of the processing units and may include, for example, scratch memory and / or system cache. Scratch memory can be on-chip memory included in accelerator 120 (e.g., static random access memory (SRAM) accessible via address space). For example, system cache may be last-level cache (LLC). In terms of memory capacity, scratch memory may not be larger than dynamic RAM (DRAM); however, scratch memory has lower access costs than DRAM.

[0048] Based on the characteristics of the neural network described above, the result of the current operation can be used for subsequent operations. Therefore, during operation execution, fast operation processing can be achieved by storing data that will be reused in subsequent layers (e.g., intermediate operation results) in multilevel memory and minimizing access costs. However, due to the relatively small capacity of multilevel memory, it is practically impossible to store the entire set of required data in multilevel memory, and a portion of the data can be stored in external memory (e.g., Figure 1 In the external memory 140 shown.

[0049] External memory 140 may refer to memory located outside of accelerator 120, and may be, for example, DRAM used as the main memory of accelerator system 100. External memory 140 is accessible via external memory controller 130. External memory 140 may be used when the memory present inside accelerator 120 is insufficient for accelerator 120 to perform workloads.

[0050] External memory 140 may have a larger capacity than the multi-level memory within accelerator 120. However, the memory access cost from accelerator 120 to external memory 140 may be greater than the memory access cost for such internal multi-level memory. Such memory access cost can indicate the amount of power and / or time required to access, read, or write data from or in memory.

[0051] Therefore, to improve the performance of accelerator system 100, it is desirable to minimize access from accelerator 120 to external memory 140. That is, access to external memory 140 can be minimized by storing the data required by accelerator 120 to perform its workload and / or data that will be reused in subsequent layers (e.g., intermediate operation result data) in multi-level memory instead of external memory 140. Such data stored in multi-level memory within accelerator 120 may be referred to herein as reuse data.

[0052] The example will be described in further detail below.

[0053] Figure 2 This is a diagram showing an example of an accelerator.

[0054] Reference Figure 2 The accelerator 200 may include a plurality of processing units (or processing elements) and a multi-level memory accessible by at least one of the processing units. The multi-level memory may be a common description of the level 0 (LV) memory 211 (e.g., LVO SRAM), LV1 memory 221, and LV2 memory 231 in the accelerator 200.

[0055] Processing unit 210, as one of the processing units, may include LVO memory 211, LVO direct memory access (DMA) 213, multiplier-accumulator (MAC) 215, and LVO controller 217. Processing unit 210 may be a neural processor (NPU), graphics processing unit (GPU), CPU, tensor processor (TPU), etc.

[0056] The LVO memory 211 can be a memory that can be accessed by the corresponding processing unit 210. That is, the LVO memory 211 can be accessed only by the processing unit 210, which is one of the multiple processing units included in the accelerator 200.

[0057] LV0 DMA 213 can control the input and / or output data of LV0 memory 211 according to instructions from LV0 controller 217. LV0 DMA 213 can read data from LV0 memory 211 or write data to LV0 memory 211 based on information associated with the source, destination, and data size included in the instructions from LV0 controller 217.

[0058] In this example, data that can be monitored and / or profiled (e.g., analyzed) will be input to or output from LV0 memory 211. Monitoring and / or profiling can be performed via LV0 DMA 213 or a separate component. Through monitoring and / or profiling, the access cost of LV0 memory 211, usage information of LV0 memory 211, and the type of data stored in LV0 memory 211 can be verified. For example, LV0 DMA 213 can verify the percentage of usage indicated by the usage information of LV0 memory 211, and which workload involves the data stored in LV0 memory 211. In the following, for ease of description, examples will be further described based on the case where monitoring and / or profiling is performed by LV0 DMA 213.

[0059] MAC 215 can perform operations related to the workload allocated to processing unit 210. For example, MAC 215 can perform a multiplication-accumulation operation on given data. Furthermore, MAC 215 can apply an activation function to the given data. For example, the activation function can be a sigmoid function, a hyperbolic tangent function, or a linear rectified function (also known as a modified linear unit, ReLU).

[0060] The LVO controller 217 may be a device configured to control components included in the processing unit 210. For example, the LVO controller 217 may control the LVO memory 211, the LVO DMA 213, and the MAC 215.

[0061] The foregoing description of processing unit 210 can be applied to each processing unit included in accelerator 200. That is, accelerator 200 may include processing units that perform operations independently.

[0062] In one example, every n processing units within the processing unit may be clustered together. In this example, n is a natural number greater than 1 and less than the number of processing units included in accelerator 200. That is, a subset of the processing units included in accelerator 200 may work together to form a cluster (e.g., processing unit cluster 220).

[0063] Processing units included in cluster 220 can share a single LV1 memory 221. That is, the LV1 memory 221 can be accessed by the processing units within cluster 220. For example, even if the operations performed by the first and second processing units within cluster 220 are different, a portion of the data required by the two operations can be common. This common data is stored in the LV1 memory 221, rather than in the LV0 memory included in each of the first and second processing units. Therefore, the first and second processing units share common data, thereby improving the overall system operating efficiency. Figure 2 In the example, each processing unit can access the LV1 memory adjacent to each processing unit.

[0064] In one example, an LV1 DMA configured to control the data input and output of LV1 memory 221 can monitor and / or analyze data input to or output from LV1 memory 221. Additionally, an LV1 controller may be present for controlling LV1 memory 221 and the LV1 DMA.

[0065] Furthermore, the entire processing unit 230 can share the LV2 memory 231. That is, the LV2 memory 231 can be accessed by the processing units included in the accelerator 200. For example, among the processing units included in the accelerator 200, there may be processing units that, although not clustered together to form the same cluster, share a portion of the data required to perform operations. In this example, such processing units can efficiently share common data through the LV2 memory 231 instead of sharing data through the LV1 memory, thereby improving overall operational efficiency. Similarly, in one example, an LV2 DMA configured to control the data input and output of the LV2 memory 231 can monitor and / or analyze data input to or output from the LV2 memory 231. Additionally, an LV2 controller is present for controlling the LV2 memory 231 and the LV2 DMA.

[0066] As described above, each processing unit can access its corresponding LV0 memory, the LV1 memory adjacent to each processing unit, and the LV2 memory of the accelerator 200, and uses these memories to perform assigned workloads respectively. The accelerator 200 may include a multi-level memory with hierarchical memory. In one example, each of the LV0, LV1, and LV2 memories may be a temporary storage memory and / or a system cache with lower access costs than DRAM as external memory.

[0067] Furthermore, the DMA and controller included in the accelerator 200 can be of a hierarchical, multi-level type.

[0068] exist Figure 2 In the example, the processing units included in accelerator 200 can execute four workloads simultaneously. For example, workloads with relatively large operations can be assigned to a larger number of processing units and processed therein, while workloads with relatively small operations can be assigned to a smaller number of processing units and processed therein.

[0069] For ease of description, in a non-limiting example, in Figure 2 The diagram shows eight processing units grouped together out of 64 to form a total of eight clusters, and three levels of memory are used to execute four workloads (Workload 1, Workload 2, Workload 3, and Workload 4). However, there is no limitation on the number of processing units, workloads, and levels that can be applied.

[0070] Figure 3 This is a diagram illustrating examples of multi-level memory and external memory.

[0071] exist Figure 3 In this document, for ease of description, LV0 memory 310, LV1 memory 320, LV2 memory 330, external memory 340, and DMA 350 are shown according to their functions.

[0072] LV0 memory 310, LV1 memory 320, and LV2 memory 330 can be configured as a global buffer (GLB) in the accelerator. LV2 memory 330 can be memory shared by multiple processing units included in the accelerator, and LV1 memory 320 can be memory shared by some of the multiple processing units. LV0 memory 310 can be included in a processing unit and is not shared with another processing unit. In the accelerator, there are LV0 memory 310 in a number corresponding to the number of processing units included in the accelerator, LV1 memory 320 in a number corresponding to the number of clusters of processing units, and LV2 memory 330 configured as a single LV2 memory.

[0073] External memory 340 may be off-chip memory located outside the accelerator and includes, for example, DRAM, three-dimensional (3D) memory (such as high-bandwidth memory (HBM)), and processing in memory (PIM). For ease of description, external memory 340 may also be referred to herein as LV3 memory.

[0074] When performing a workload in the processing unit, LV0 memory 310, LV1 memory 320, LV2 memory 330, and external memory 340 can be used, and the memory access cost can vary for each level. For example, the memory access cost can increase with the level. That is, the access cost of LV0 memory 310 can be the lowest, and the access cost of external memory 340 can be the highest.

[0075] DMA 350 is also shown according to its function. DMAs can be configured individually for each level and are used to read data from or write data to the memory at the corresponding level. For example, there are LV0 DMA, LV1 DMA, LV2 DMA, and LV3 DMA, respectively. LV0 DMA is configured to control the input and / or output data of LV0 memory 310, LV1 DMA is configured to control the input and / or output data of LV1 memory 320, LV2 DMA is configured to control the input and / or output data of LV2 memory 330, and LV3 DMA is configured to control the input and / or output data of external memory 340. LV0 memory 310, LV1 memory 320, LV2 memory 330, and external memory 340 can exchange data with each other through DMAs configured for their respective levels.

[0076] Figure 4 This is a diagram illustrating an example of determining how to reuse data.

[0077] exist Figure 4 In the example, for ease of description, LV0 memory 410, LV1 memory 420, LV2 memory 430, external memory 440, DMA (or DMA device) 450, data reuse policy maker 460 and host controller 470 are shown in accordance with their functions.

[0078] In one example, through an extension to DMA 450, data input to or output from a corresponding level of memory can be monitored and / or profiled, and the results of the monitoring and / or profiling can be sent to host controller 470. The extension can indicate the additional provisioning or offloading of this monitoring and / or profiling functionality for input and output data to the corresponding level of memory within DMA 450. In one example, hardware resource information and / or memory access costs can be determined by offloading the extension to DMA 450, which can be configured to control data input to or output from a multi-level memory included in an accelerator. For ease of description, in Figure 4 The diagram illustrates the performance of such monitoring and / or profiling operations within an extension of the DMA 450. However, separate components for performing monitoring and / or profiling operations can be applied without limitation. The following description will focus on the scenario of performing monitoring and / or profiling operations via an extension of the DMA 450. Figure 4 Examples.

[0079] DMA 450 can verify usage information of the corresponding level of memory. For example, LV2 DMA can verify usage information 431 of LV2 memory 430 for each workload. Usage information 431 may include, for example, information related to the utilization and amount of LV2 memory 430 for each workload. Furthermore, LV1 DMA can verify usage information 421 of LV1 memory 420. Figure 4 In another example, LV0 DMA can verify the usage information of LV0 memory 410. Such memory usage information verified by the corresponding level of DMA can be sent to the host controller 470.

[0080] The host controller 470 can allocate one or more workloads to the accelerator based on memory usage information for each level. For example, when there is a relatively large amount of available multilevel memory due to the accelerator's multilevel memory not being heavily used, the host controller 470 can allocate a larger number of workloads or workloads with greater operational volume. Conversely, when there is a relatively small amount of available multilevel memory due to the accelerator's multilevel memory being heavily used, the host controller 470 can allocate a smaller number of workloads or workloads with smaller operational volume.

[0081] When a workload is allocated via the host controller 470, the reuse data policy maker 460 determines the reuse data for the workload. The reuse data policy maker 460 can operate within a controller configured to control DMA 450 and can be implemented as a controller included separately for each level in DMA 450. For example, an LV2 controller configured to control LV2 DMA can determine the reuse data to be stored in LV2 memory 430. Similarly, an LV1 controller configured to control LV1 DMA can determine the reuse data to be stored in LV1 memory 420. Likewise, an LV0 controller configured to control LV0 DMA can determine the reuse data to be stored in LV0 memory 410.

[0082] Reusable data can be determined based on accelerator hardware resource information and / or memory access costs. Reusable data can represent data that is reused in subsequent operations when performed sequentially according to workload and will be stored in multi-level memory within the accelerator. That is, reusable data may not be sent to the accelerator's external memory. For example, data most frequently used in subsequent operations may be stored in memory with lower access costs and then used later.

[0083] The accelerator's hardware resource information may include: usage information of multi-level memory included in the accelerator, usage information of multiple processing units included in the accelerator, and / or system cache information. Multi-level memory usage information may indicate the availability and / or usage information of memory for each level. Processing unit usage information may include usage efficiency information for each processing unit.

[0084] In one example, based on the accelerator's hardware resource information, a tiling method can be determined for applying the data input to perform the workload. A tiling method can represent a approach where, when the input data is too large to process all at once, it is divided into multiple blocks and each block is processed. When dividing the input data used to perform the workload into blocks, the block size can be determined based on the accelerator's hardware resource information. For example, when there is a large amount of available hardware resources for the accelerator, the block size can be determined to be large. Conversely, when there is a small amount of available hardware resources for the accelerator, the block size can be determined to be small. Based on the tiling method determined as described above, the size of the reused data can be determined. For example, the size of the reused data can be determined to correspond to the block size.

[0085] Memory access costs may include the access costs of the accelerator's external memory 440 and the access costs of the accelerator's multilevel memory. For example, the access cost of external memory 440 may be greater than the access cost of multilevel memory. The access cost of multilevel memory can increase for memory shared by a larger number of processing units. For example, LV2 memory 430 may be shared by all processing units in the accelerator, therefore, LV2 memory 430 has the highest access cost among multilevel memory systems. In contrast, LV0 memory 410 may not be shared by another processing unit, but can only be accessed by the corresponding processing unit; therefore, LV0 memory 410 has the lowest access cost among multilevel memory systems.

[0086] For example, data that is frequently used in subsequent operations or tasks may be identified as reusable data that will be stored in a lower-level memory, even though only a small number of processing units can access the lower-level memory. Therefore, it is also possible to determine whether the data is to be identified as reusable data and at which level of memory the reusable data will be stored, based on the number of processing units that need the data and whether these processing units are grouped together to form the same cluster.

[0087] Furthermore, the characteristics of the assigned workloads can be considered when determining the reused data. For example, one workload assigned to the accelerator may need to be processed quickly, while another workload may only need to be processed within a preset time. Therefore, to enhance overall system efficiency, reused data can be allocated with higher priority in the initial steps for workloads that need to be processed as quickly as possible, and then with higher priority closer to the preset time for workloads that need to be processed within the preset time. As mentioned above, processing scheduling for each workload can be additionally considered, thus allowing for dynamic determination of reused data.

[0088] As described above, reusable data can be determined for each memory in a multi-level memory system. For example, reusable data can be determined for each of the LV0 memory 410, LV1 memory 420, and LV2 memory 430. Furthermore, reusable data can be determined for each layer of a neural network corresponding to a workload. For example, when operations are performed sequentially along multiple layers included in a neural network, reusable data suitable for each of the multiple layers can be determined. For example, data that is reused more frequently in subsequent layers can be determined as reusable data with higher priority. In this example, data associated with neural network-based operations may include input feature maps (IFM), output feature maps (OFM), and weights (WGHT). Furthermore, the type of data selected as reusable data can be dynamically changed when operations are performed sequentially along multiple layers. For example, reusable data determined in earlier layers may have a higher proportion of IMF. In contrast, reusable data determined in later layers may have a higher proportion of WGHT. Therefore, the type of data selected as reusable data can be dynamically changed based on the workload, the neural network, and / or the operation.

[0089] Based on the above factors, reusable data that minimizes access to the accelerator's external memory can be determined when multiple processing units of the accelerator execute one or more workloads. The reusable data can be determined such that access to external memory is minimized for all workloads allocated to the accelerator and for all processing units, rather than for a specific workload or a portion of a processing unit.

[0090] Figure 5 and Figure 6 This is a diagram illustrating an example of an accelerator system.

[0091] Reference Figure 5 Server 500 may include host controller 510, accelerator 520, and external storage 530. Host controller 510 may distribute one or more workloads to accelerator 520. When accelerator 520 performs workloads across multiple processing units, accelerator 520 may determine at least one set of reusable data based on hardware resource information and / or memory access costs. Accelerator 520 may use the reusable data to perform workloads and provide the results of performing workloads. In one example, server 500 may be an accelerator system.

[0092] Reference Figure 6 The user terminal 600 may include a host controller 610, an accelerometer 620, and external memory 630. Each of these components can perform the respective operations described herein. Although for the sake of description, Figure 6 The Lieutenant General's User Terminal 600 is shown as a smartphone, but the above reference... Figure 6The description provided can also be applied to various computing devices (such as personal computers (PCs), tablet PCs and laptops), various wearable devices (such as smartwatches and smart glasses), various home appliances (such as smart speakers, smart TVs and smart refrigerators) and other devices (such as smart vehicles, smart self-service terminals and Internet of Things (IoT) devices), without limitation.

[0093] As described above, accelerators (e.g., accelerators 520 and 620) may be included in a user terminal (e.g., user terminal 600) or in a server (e.g., server 500). Based on the workload, the data to be identified is input to the user terminal using a neural network, and the server is configured to receive the data to be identified from the user terminal.

[0094] Figure 7 This is a flowchart illustrating an example of how an accelerator operates.

[0095] The operating methods described below can be performed by the accelerator.

[0096] Reference Figure 7 In operation 710, the accelerator may receive one or more workloads assigned by the host controller.

[0097] In operation 720, when a workload is executed across multiple processing units included in the accelerator, the accelerator may determine reusable data for the workload based on the accelerator's hardware resource information and / or memory access costs. As the processing units execute the workload, the accelerator may determine reusable data that minimizes access to the accelerator's external memory. The accelerator may additionally determine reusable data dynamically based on the characteristics of the workload. Reusable data may represent data stored in the accelerator's internal multi-level memory and that may not be sent to the accelerator's external memory.

[0098] The accelerator's hardware resource information may include usage information for multi-level memory included in the accelerator and usage information for processing units. Memory access costs may include access costs for external memory and access costs for multi-level memory included in the accelerator.

[0099] In operation 730, the accelerator provides the results of executing the workload. The accelerator can then send the results of executing the workload to the host controller.

[0100] Reference above Figures 1 to 6 The content described can be applied to the above references. Figure 7 The operations described are omitted here for brevity; more detailed and repetitive descriptions will be omitted.

[0101] Accelerators, acceleration systems 100, 200, accelerators 120, 520, 620, host controllers 110, 470, 510, 610, external memory controllers 130, external memory 140, 530, 630, processing unit 210, LV0 memory 310, 410, LV1 memory 320, 420, LV2 memory 330, 430, external memory 340, 440, DMA 350, 450, data reuse policy maker 460, server 500, and more. Figures 1 to 7Other devices, units, modules, apparatuses, and other components described are implemented through hardware components. Examples of hardware components that can be used to perform the operations described in this application include, where appropriate, controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components performing the operations described in this application are implemented through computing hardware (e.g., through one or more processors or computers). A processor or computer can be implemented through one or more processing elements (such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field-programmable gate arrays, programmable logic arrays, microprocessors, or any other means or combination of means configured to respond to and execute instructions in a defined manner to achieve a desired result). In one example, the processor or computer includes or is connected to one or more memories storing instructions or software executed by the processor or computer. Hardware components implemented by a processor or computer can execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) for performing the operations described in this application. Hardware components can also access, manipulate, process, create, and store data in response to the execution of instructions or software. For simplicity, the singular terms "processor" or "computer" are used in the description of the examples described in this application; however, in other examples, multiple processors or computers may be used, or a processor or computer may include multiple processing elements or multiple types of processing elements or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or additional processors and additional controllers. One or more processors, or a processor and a controller, may implement a single hardware component or two or more hardware components. Hardware components may have any one or more different processing configurations, examples of which include: a single processor, a standalone processor, a parallel processor, single-instruction single-data (SISD) multiple processing, single-instruction multiple-data (SIMD) multiple processing, multiple-instruction single-data (MISD) multiple processing, and multiple-instruction multiple-data (MIMD) multiple processing.

[0102] Figures 1 to 7The methods for performing the operations described in this application, as shown, are executed by computing hardware (e.g., by one or more processors or a computer), which is implemented to execute instructions or software as described above to perform the operations performed by the methods described in this application. For example, a single operation or two or more operations may be executed by a single processor or two or more processors, or a processor and a controller. One or more operations may be executed by one or more processors, or a processor and a controller, and one or more other operations may be executed by one or more other processors, or additional processors and additional controllers. One or more processors, or a processor and a controller, may execute a single operation or two or more operations.

[0103] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above can be written as computer programs, code segments, instructions, or any combination thereof to individually or collectively instruct or configure one or more processors or computers, such as machines or special-purpose computers, to perform the operations performed by the hardware components and methods described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) that is directly executed by one or more processors or computers. In another example, the instructions or software include high-level code that is executed by one or more processors or computers using an interpreter. The instructions or software can be written in any programming language based on the block diagrams and flowcharts shown in the accompanying drawings and the corresponding descriptions in the specification, which disclose algorithms for performing the operations performed by the hardware components and methods described above.

[0104] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above, along with any associated data, data files, and data structures, may be recorded, stored, or fixed in, or on, one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards or microcards (e.g., Secure Digital (SD) or Extreme Digital (XD)), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store instructions or software and any associated data, data files, and data structures in a non-transitory manner and to provide said instructions or software and any associated data, data files, and data structures to one or more processors or computers, such that one or more processors or computers can execute the instructions. In one example, the instructions or software and any associated data, data files, and data structures are distributed across a networked computer system, such that the instructions and software and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0105] While this disclosure includes specific examples, it will be clear upon understanding this disclosure that various changes in form and detail may be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are to be considered descriptive only and not for limiting purposes. The description of features or aspects in each example will be considered applicable to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order, and / or if components in the described system, architecture, apparatus, or circuit are combined in a different manner, and / or replaced or supplemented by other components or their equivalents.

[0106] Therefore, the scope of the disclosure is not limited by the specific embodiments, but by the claims and their equivalents, and all variations within the scope of the claims and their equivalents should be interpreted as included in the disclosure.

Claims

1. A method of operating an accelerator, comprising: Receives one or more workloads assigned by a host controller, which is configured to control an accelerator comprising multiple processing elements; When the plurality of processing elements execute the one or more workloads, reuse data for the one or more workloads is determined based on at least one of the accelerator's hardware resource information and memory access costs; as well as The results of executing the one or more workloads will be provided to the host controller. Reused data refers to data that is reused in subsequent operations. Specifically, by offloading an extension to a direct memory access device to determine at least one of hardware resource information and memory access cost, the direct memory access device is configured to control data input to or output from a multi-level memory included in the accelerator.

2. The operating method according to claim 1, wherein, The steps to determine reusable data include: When the plurality of processing elements execute the one or more workloads, reused data is determined that minimizes the accelerator's access to external memory.

3. The operating method according to claim 1, wherein, The hardware resource information of the accelerator includes at least one of the following: usage information of multi-level memory in the accelerator, usage information of the plurality of processing elements, and system cache information.

4. The operating method according to claim 3, wherein, Multilevel memory includes at least one of the following: Level 0 memory, accessible by one of the plurality of processing elements; Level 1 memory, accessible by a subset of the plurality of processing elements, wherein the subset includes more than one number of processing elements; and Level 2 memory, accessible by the plurality of processing elements.

5. The operating method according to claim 1, wherein, The steps to determine reusable data also include: The data to be reused is determined dynamically based on the characteristics of the one or more workloads.

6. The operating method according to claim 1, wherein, Memory access costs include: access costs to the accelerator's external memory and access costs to the multilevel memory included in the accelerator.

7. The operating method according to claim 6, wherein, The access cost of external memory is greater than the access cost of multi-level memory.

8. The operating method according to claim 6, wherein, The access cost of multilevel memory increases as the number of shared processing elements in the multilevel memory increases.

9. The operating method according to any one of claims 1 to 8, wherein, The steps to determine reusable data include: Based on accelerator hardware resource information, a chunking method is determined to be applied to data input to execute the one or more workloads; and The size of the reused data is determined based on a defined block partitioning method.

10. The operating method according to any one of claims 1 to 8, wherein, The steps to determine reusable data include: Reusable data is determined for at least one of the multi-level memory included in the accelerator and for each layer of the neural network corresponding to the workload.

11. The operating method according to any one of claims 1 to 8, wherein, Reusable data is stored in a multi-level memory within the accelerator and is not sent to the accelerator's external memory.

12. The operating method according to any one of claims 1 to 8, wherein, Each of the plurality of processing elements includes: Level 0 memory, which can be accessed by the corresponding processing element; A Level 0 direct memory access device is configured to: control data input and / or output to Level 0 memory, and monitor and / or analyze data input to or output from Level 0 memory; Multiplier-accumulator, configured to perform operations related to the one or more workloads allocated to the processing element; and The Level 0 controller is configured to control one of the Level 0 memory, the Level 0 direct memory access device, and the multiplier-accumulator, or a combination of the Level 0 memory, the Level 0 direct memory access device, and the multiplier-accumulator.

13. The operating method according to any one of claims 1 to 8, wherein, The accelerator is included in a user terminal or a server, where first data is input to the user terminal and the server is configured to receive the first data from the user terminal, wherein the first data is data to be identified based on a workload using a neural network.

14. A non-transitory computer-readable storage medium for storing commands, which, when executed by a processor, cause the processor to perform the operating method according to any one of claims 1 to 13.

15. An accelerator, comprising: Multiple processing elements are configured to execute one or more workloads assigned by the host controller; as well as A multi-level memory is configured to be accessible by at least one of the plurality of processing elements. When the plurality of processing elements execute the one or more workloads, the reuse data of the one or more workloads is determined based on at least one of the accelerator's hardware resource information and memory access costs, and is stored in multi-level memory. Specifically, by offloading an extension to a direct memory access device to determine at least one of hardware resource information and memory access cost, the direct memory access device is configured to control data input to or output from a multi-level memory included in the accelerator.

16. The accelerator according to claim 15, wherein, When the plurality of processing elements execute the one or more workloads, reused data is determined that minimizes access to the accelerator's external memory.

17. The accelerator according to claim 15, wherein, The hardware resource information of the accelerator includes at least one of the following: usage information of the multi-level memory in the accelerator, usage information of the plurality of processing elements, and system cache information.

18. An accelerator system, comprising: An accelerator includes multiple processing elements configured to perform one or more workloads, and multi-level memory with different access costs; The accelerator's external memory; as well as The host controller is configured to distribute the one or more workloads to the accelerator. Wherein, when the plurality of processing elements execute the one or more workloads, the accelerator is configured to: determine the reuse data for the one or more workloads based on at least one of the accelerator's hardware resource information and memory access costs. Specifically, by offloading an extension to a direct memory access device to determine at least one of hardware resource information and memory access cost, the direct memory access device is configured to control data input to or output from a multi-level memory included in the accelerator.

Images