Iterative compilation method and electronic device for deep learning

Abstract

A method and an electronic device for iterative compilation for deep learning are disclosed. The electronic device includes a deep learning compiler configured to receive a hardware representation corresponding to a target system including a hierarchical structure, extract a plurality of hierarchies from the target system based on the received hardware representation, and perform iterative compilation on the extracted plurality of hierarchies.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2023-0073587 filed on June 8, 2023, in the Korean Intellectual Property Office, and Korean Patent Application No. 10-2023-0115322 filed on August 31, 2023, in the Korean Intellectual Property Office, the disclosures of which are incorporated herein by reference in their entirety for all purposes. Technical Field

[0002] The following description relates to methods and apparatus with iterative compilation for deep learning. Background Art

[0003] Deep learning models may include convolutional neural networks (CNNs), transformer models, graph neural network (GNN) models, etc. Hardware may include graphics processing units (GPUs), neural processing units (NPUs), and processing in memory (PIMs). Systems may include mobile devices, vehicles, data centers, and supercomputers. Deep learning models or hardware may be optimized based on hints, users, or heuristic algorithms. Optimization of deep learning models or hardware may be automatically performed by a compiler. Summary of the invention

[0004] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0005] In one or more general aspects, an electronic device includes: a deep learning compiler configured to: receive a hardware representation corresponding to a target system including a hierarchical structure; extract a plurality of hierarchies from the target system based on the received hardware representation; and perform iterative compilation on the extracted plurality of hierarchies.

[0006] For the process of performing iterative compilation, the deep learning compiler may be configured to sequentially apply a path pipeline indicating a sequence of paths to each of the extracted multiple levels from an upper level to a lower level.

[0007] For the process of performing iterative compilation, the deep learning compiler can be configured to perform graph-level optimization, partition optimization, scheduling optimization, memory optimization, and communication optimization on each of the extracted multiple levels by applying a path pipeline to the extracted multiple levels.

[0008] The path pipeline may include: a graphics-level optimized path, a partition optimized path, a computation scheduling optimized path, a memory or communication metered path, and a memory or communication scheduling optimized path.

[0009] Each of the plurality of pathways constituting the pathway pipeline may be configured to be applicable to a hierarchy of a plurality of systems including a target system without being dependent on a single system among the plurality of systems.

[0010] For the process of performing iterative compilation, the deep learning compiler may be configured to perform optimization on the extracted multiple levels using a graphics dialect, a scheduling dialect, a data movement dialect, a memory dialect, and a communication dialect.

[0011] For a process of performing iterative compilation, the deep learning compiler may be configured to: calculate a count of layers constituting the target system according to the received hardware representation; and iteratively apply the path pipeline to the extracted plurality of layers with the calculated count of layers.

[0012] An electronic device may include one or more processors including a deep learning compiler.

[0013] In one or more general aspects, a processor-implemented method includes receiving a hardware representation corresponding to a target system including a hierarchy structure; extracting a plurality of hierarchies from the target system based on the received hardware representation; and performing iterative compilation on the extracted plurality of hierarchies.

[0014] The process of performing iterative compilation may include sequentially applying a path pipeline indicating a sequence of paths to each of the extracted plurality of levels from an upper level to a lower level.

[0015] The process of performing iterative compilation may include performing graph level optimization, partition optimization, scheduling optimization, memory optimization, and communication optimization on each of the extracted multiple levels by applying a path pipeline to the extracted multiple levels.

[0016] The path pipeline may include: a graphics-level optimized path, a partition optimized path, a computation scheduling optimized path, a memory or communication metered path, and a memory or communication scheduling optimized path.

[0017] Each of the plurality of pathways constituting the pathway pipeline may be configured to be applicable to a hierarchy of a plurality of systems including a target system without being dependent on a single system among the plurality of systems.

[0018] The process of performing iterative compilation may include performing optimizations on the extracted plurality of hierarchies using a graphics dialect, a scheduling dialect, a data movement dialect, a memory dialect, and a communication dialect.

[0019] The process of performing iterative compilation may include: calculating a count of levels constituting the target system according to the received hardware representation; and iteratively applying the path pipeline to the extracted plurality of levels with the calculated count of levels.

[0020] In one or more general aspects, a non-transitory computer-readable storage medium stores instructions that, when executed by a processor, configure the processor to perform any one, any combination, or all of the operations and / or methods described herein.

[0021] In one or more general aspects, an electronic device includes: one or more processors configured to: receive a hardware representation corresponding to a target system including a hierarchical structure; extract a plurality of hierarchies from the target system based on the received hardware representation; and perform iterative compilation on the extracted plurality of hierarchies by applying the same path pipeline to each of the plurality of hierarchies.

[0022] The extracted plurality of hierarchies may include: an upper hierarchy including a plurality of nodes of a target system; and a lower hierarchy including components of nodes among the plurality of nodes.

[0023] For processing that performs iterative compilation, the one or more processors may be configured to: apply the same path pipeline to an upper level by mapping information onto a node instead of using information of an internal configuration of the node; and apply the same path pipeline to a lower level by mapping the information mapped onto the node onto a component of the node based on information of an internal configuration of the node.

[0024] The information mapped to the node may include: operations mapped to the node and data of a deep learning model mapped to the node.

[0025] Other features and aspects will be apparent from the following detailed description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a flowchart briefly illustrating a process in which a deep learning compiler performs compilation on a target system according to one or more embodiments.

[0027] Figure 2 An example of an overall configuration of a framework of a Universal Deep Learning Compiler (UDLC) according to one or more embodiments is shown.

[0028] Figure 3 An example of a compilation process of a deep learning compiler according to one or more embodiments is shown.

[0029] Figure 4 An example of pseudo code is shown that determines a pass for data placement in a memory hierarchy in accordance with one or more embodiments.

[0030] Figure 5 Examples of the operation of a deep learning runtime and a deep learning compiler are shown in accordance with one or more embodiments.

[0031] Figure 6 A table showing an example of a path to configure a deep learning optimization field according to one or more embodiments is shown.

[0032] Figure 7 An example of a process in which a deep learning compiler performs iterative compilation on a target system according to one or more embodiments is shown.

[0033] Figure 8 An electronic device according to one or more embodiments is shown.

[0034] Throughout the drawings and detailed description, unless otherwise described or provided, the same figure reference numerals will be understood to refer to the same elements, features and structures. For clarity, illustration and convenience, the drawings may not be to scale, and the relative sizes, proportions and depictions of the elements in the drawings may be exaggerated. DETAILED DESCRIPTION

[0035] The following specific embodiments are provided to help the reader obtain a comprehensive understanding of the methods, devices and / or systems described herein. However, after understanding the disclosure of the present application, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be clear. For example, the order of operations described herein is merely an example and is not limited to the order of operations set forth herein, but the order of operations can be changed as will be clear after understanding the disclosure of the present application, except for the operations that must occur in a specific order. In addition, for greater clarity and simplicity, the description of features known after understanding the disclosure of the present application may be omitted.

[0036] Although terms such as "first", "second" and "third" or A, B, (a), (b), etc. may be used herein to describe various components, components, regions, layers or parts, these components, components, regions, layers or parts are not limited by these terms. Each of these terms is not used to define, for example, the essence, order or sequence of the corresponding component, component, region, layer or part, but is only used to distinguish the corresponding component, component, region, layer or part from other components, components, regions, layers or parts. Therefore, without departing from the teaching of the example, the first component, first component, first region, first layer or first part mentioned in the examples described herein may also be referred to as the second component, second component, second region, second layer or second part.

[0037] Throughout the specification, when a component or element is described as being "connected to," "coupled to," or "engaged to" another component or element, it may be "connected to," "coupled to," or "engaged to" another component or element directly (e.g., in contact with the other component or element), or one or more other components or elements may reasonably exist between them. When a component or element is described as being "directly connected to," "directly coupled to," or "directly engaged to" another component or element, there may not be other elements between them. Similarly, expressions such as "between" and "directly between," as well as "adjacent to" and "immediately adjacent to" may also be interpreted as described above.

[0038] The terms used herein are only used to describe various examples and are not used to limit the present disclosure. Unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. As used herein, the term "and / or" includes any one of the associated listed items and any combination of any two or more. As a non-limiting example, the terms "include", "comprise", and "have" illustrate the existence of the features, quantities, operations, components, elements and / or combinations thereof, but do not exclude the existence or addition of one or more other features, quantities, operations, components, elements and / or combinations thereof. Herein, with respect to examples or embodiments (for example, with respect to what an example or embodiment may include or realize), the term "may" is used to represent at least one example or embodiment of the situation in which such features are included or realized, but all examples are not limited thereto.

[0039] Unless the corresponding description and embodiments require that such a list (e.g., "at least one of A, B, and C") be interpreted as having a conjunctive meaning, the phrases "at least one of A, B, and C," "at least one of A, B, or C," etc. are intended to have a disjunctive meaning, and these phrases "at least one of A, B, and C," "at least one of A, B, or C," etc. also include examples in which one or more of each of A, B, and / or C may be present (e.g., any combination of one or more of each of A, B, and C).

[0040] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as would be generally understood after understanding the present disclosure. It will also be understood that, unless expressly so defined herein, terms (such as those defined in general dictionaries) should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and not interpreted in an idealized or overly formal sense.

[0041] Hereinafter, examples will be described in detail with reference to the accompanying drawings. When describing the embodiments with reference to the accompanying drawings, the same reference numerals denote the same elements, and repeated descriptions thereof will be omitted.

[0042] Figure 1 is a flowchart briefly illustrating a process in which a deep learning compiler performs compilation on a target system according to one or more embodiments.

[0043] Conventional compilers cannot use one compiler to support systems with various hierarchical structures. This is because the different hardware levels that make up the system need to be optimized using different compiler codes implemented by conventional compilers. In contrast, the electronic device of one or more embodiments can perform compilation processing on systems with various hierarchical structures through a deep learning compiler. The deep learning compiler of one or more embodiments can generate a common path pipeline that is commonly applied to the hardware level, and can compile and optimize the entire system by applying the generated common path pipeline to the hardware level that makes up the system. In the following, a detailed description of an example of the operation of a deep learning compiler is provided.

[0044] In operation 110, a deep learning compiler of an electronic device may receive a hardware representation that provides system-level information about a target system having a hierarchical structure. The hardware representation may represent components constituting the hardware (e.g., Figure 8 The physical form or construction method of the target system (components of the electronic device 800). The deep learning compiler can identify the hardware structure of the target system by reading the hardware representation.

[0045] In operation 120, the deep learning compiler may extract a plurality of levels from the target system based on the received hardware representation. The plurality of levels extracted from the target system may indicate a result of abstracting the hardware configuration of the target system at various levels.

[0046] In operation 130, the deep learning compiler may perform iterative compilation on the extracted multiple levels. In this case, the deep learning compiler performing iterative compilation on the multiple levels may be or may include applying the same path pipeline to each of the multiple levels. The deep learning compiler may sequentially apply the same path pipeline to each of the multiple levels from the upper level to the lower level.

[0047] For example, the deep learning compiler may extract a plurality of hierarchies from the target system, the plurality of hierarchies including a first hierarchical level as an upper hierarchical level and a second hierarchical level as a lower hierarchical level. In this case, the deep learning compiler may perform iterative compilation by applying the path pipeline to the first hierarchical level as the upper hierarchical level and the second hierarchical level as the lower hierarchical level.

[0048] In one or more embodiments, a path pipeline may be a sequence of paths. In the present disclosure, a path may be a component of a deep learning compiler that receives code as input and performs conversion, analysis, optimization, etc. on the received code.

[0049] The deep learning optimization field can be divided into five fields, namely, a graph field, a partition field, a scheduling field, a memory field, and a communication field. The deep learning compiler can generate a path pipeline including a path corresponding to each optimization field. The deep learning compiler can perform graph-level optimization, partition optimization, scheduling optimization, memory optimization, and communication optimization by applying the generated path pipeline to each of the multiple levels.

[0050] Figure 2 An example of an overall configuration of a framework of a Universal Deep Learning Compiler (UDLC) according to one or more embodiments is shown.

[0051] In one or more embodiments, the UDLC framework 200 can perform compilation processing for a wide range of target systems. In order to perform the compilation process through the UDLC framework 200, the hardware description about the target system can be used to encapsulate information. For example, the hardware representation can provide a hierarchical composition of the hardware elements (including computing elements, memory elements, and communication elements) of the target system.

[0052] In one or more embodiments, the UDLC framework 200 may include a deep learning compiler 210 , a deep learning runtime module 230 , a graph optimizer 240 , and a performance estimator 250 .

[0053] In one or more embodiments, the deep learning compiler 210 of the UDLC framework 200 may receive a hardware representation that provides system-level information of a target system. In one or more embodiments, the deep learning compiler 210 may load a hardware representation that provides system-level information of a target system from a memory. In addition, the deep learning compiler 210 may receive a deep learning model 211 as input. For example, the deep learning model 211 received as input by the deep learning compiler 210 may be represented by a Torch script or an Open Neural Network Exchange (ONNX). The deep learning compiler 210 may use four dialects to perform iterative compilation, and iterative compilation can manage the heterogeneity and varying scales of the system. The four dialects may be graphics dialects, memory dialects, communication dialects, and scheduling dialects.

[0054] In one or more embodiments, graphics dialects may be used to easily perform graphics-level optimizations including operations such as merging and splitting of hierarchies and execution sorting. Memory dialects may be used to focus on memory allocation and related optimizations. Communication dialects may be used to handle point-to-point and cluster communications. Scheduling dialects may be used to form the core of optimizations to build parallelization techniques and double buffering. These dialects and corresponding paths may be constructed to apply to all system levels.

[0055] In one or more embodiments, the deep learning compiler 210 may mainly use internal heuristics to optimize the deep learning model 211, and may allow the integration of external optimization methods. The deep learning compiler 210 may use linear programming, meta-heuristics, and deep learning methods including reinforcement learning to perform more aggressive optimization.

[0056] In one or more embodiments, fast performance evaluation may be performed to search a large design space by graph optimizer 240. To this end, a performance estimator 250 may be used to perform performance prediction on an intermediate representation (IR) of a schedule for various hardware without assuming specific hardware.

[0057] In one or more embodiments, the deep learning compiler 210 of the UDLC 200 may perform a compilation process, and may generate a target executable binary file 221 corresponding to the deep learning model 211 by performing the compilation process. The target executable binary file 221 may be sent to the deep learning runtime module 230. The deep learning runtime module 230 may execute the target executable binary file 221 received from the deep learning compiler 210. The deep learning runtime module 230 may have a configurable and extensible architecture to accommodate systems of various sizes from mobile devices to large-sized servers.

[0058] In one or more embodiments, the components (e.g., deep learning compiler 210 and deep learning runtime module 230) provided in UDLC framework 200 can be configured to have properties focused on highly reconfigurable hardware representation. By selecting the configuration, UDLC framework 200 of one or more embodiments can provide flexible and efficient compilation processing for various artificial intelligence (AI) accelerators and systems.

[0059] Figure 3 An example of a compilation process of a deep learning compiler according to one or more embodiments is shown.

[0060] In one or more embodiments, the deep learning compiler 310 (e.g., Figure 2 The deep learning compiler 210) can implement various optimizations to execute deep neural network (DNN) models in various target systems (or devices).

[0061] Deep Learning Compiler Configuration Principles

[0062] In one or more embodiments, three configuration principles may be established to configure the reusable deep learning compiler 310 .

[0063] 1) Definition of expressive IR:

[0064] In one or more embodiments, the deep learning compiler 310 may use IR to include information about the deep learning model input to the deep learning compiler 310 and an optimized execution plan for the deep learning model. Therefore, the expressiveness of IR can play a key role in determining the scope of the deep learning compiler. When IR cannot effectively represent a predetermined computing operation or data type, the ability of the compiler may be limited, and thus the ability to compile various deep learning models including predetermined operations or data types may be reduced. In addition, when the ability to represent a predetermined scheduling optimization (such as overlap between computing operations and communication operations) is insufficient, the compiler may not be able to use optimization to reduce execution time. Therefore, the configuration of the deep learning compiler 310 of one or more embodiments that expresses IR can achieve extensive optimization of various deep learning models.

[0065] 2) Configuration of target-agnostic paths:

[0066] In one or more embodiments, in the deep learning compiler 310, a path can be used as a basic element for generating analysis results or optimizing code by receiving input code. However, because optimizations related to low-level hardware features (such as memory hierarchy, dedicated computing operations, and communication bandwidth) are inherently associated with specific hardware types, problems may occur in typical deep learning compilers because low-level hardware features may not be reused on various target architectures. In contrast, by considering such optimizations to be implemented as paths in the deep learning compiler 310 of one or more embodiments, a reusable and generally applicable compiler framework (e.g., Figure 2 UDLC framework 200).

[0067] In one or more embodiments, each of the multiple paths that make up the path pipeline can be agnostic to the target (e.g., system or hardware). A single path may not reflect a specific hardware type or system structure, and may be configured based on hardware abstraction information described in the hardware representation. For example, a path may be configured based on abstract code (such as, "if (deviceType == compute)") rather than code (such as, "if (device == GPU)"). As described above, by configuring the path based on the hardware abstraction information described in the hardware representation, the path of one or more embodiments may not depend on a specific hardware type or system structure.

[0068] 3) Various extended system-aware compiler structures:

[0069] In one or more embodiments, the execution of deep learning models may include a wide range of systems of various sizes. The systems may range from mobile neural processors (NPUs) processing small DNNs for object detection to large graphics processor (GPU) clusters for training various language models.

[0070] The deep learning compiler 310 of one or more embodiments can recognize various sizes and adapt them to ensure compatibility across extended systems. That is, based on the expressive definition of IR, a system-aware compiler using a configuration of a target-agnostic path can be generated. Through this configuration, the deep learning compiler 310 of one or more embodiments can effectively meet the specific requirements of systems of various sizes.

[0071] Dialect Definition

[0072] Dialects can represent concepts including IR, types, attributes, and paths that are tuned to a specific optimization scope. In the following, based on the dialect concept, the concepts of 5 dialects (graph, scheduling, data movement, memory, and communication) are described to facilitate various optimizations for a wide range of DNN models.

[0073] In one or more embodiments, the graph dialect in the deep learning compiler 310 may include various computational operations for deep learning models, including convolution, matrix multiplication (e.g., matmul), and rectified linear units (ReLU), as well as tensor representations. Since the computational operations may be implemented differently in various target devices, only the mathematical semantics of the operations may be captured without detailed information about the execution. In addition to forward computational operations, the graph dialect may include gradient computations and backward computational operations. The graph dialect may represent the execution of a deep learning model via a computational graph by providing a comprehensive set of computational and tensor operations. Through this representation, the deep learning compiler 310 of one or more embodiments may apply high-level optimizations (such as operator fusion).

[0074] In one or more embodiments, the scheduling dialect in the deep learning compiler 310 can play an important role in the deep learning compiler 310 by capturing optimized execution schedules of computational operations, memory operations, and communication operations across various devices and systems. The scheduling dialect may include various operations that represent the sequential and parallel execution of operations, dependencies among operations, and synchronization mechanisms (such as barriers). By using the performance of the scheduling dialect, the deep learning compiler 310 of one or more embodiments can achieve optimal performance and efficient use of resources across various hardware platforms and systems by effectively managing and orchestrating the execution of operations.

[0075] In one or more embodiments, the data movement dialect in the deep learning compiler 310 may include memory operations and communication operations, and may perform a time representation of all data movement occurring between devices. The data movement dialect may include various operations related to peer-to-peer (P2P) and collective data movement, and may be additionally optimized by mapping to memory operations or communication operations based on cost analysis.

[0076] In one or more embodiments, the memory dialect in the deep learning compiler 310 can represent various memory operations (including memory allocation and deallocation, memory object copying, and memory object storage). In addition, the memory dialect can extend support beyond the management of virtual memory objects to include direct management of physical memory (such as scratchpad memory).

[0077] The communication dialect in the deep learning compiler 310 may include various P2P and cluster communication operations to easily perform efficient communication between devices. In the final lowering operation 330, each communication operation may be mapped to a specific communication library function (such as NCLL allreduce) to optimize communication processing. By establishing a communication dialect, the deep learning compiler 310 of one or more embodiments can achieve smooth establishment of communication operations, and can use a dedicated communication library to improve the scalability and performance of deep learning calculations across distributed systems.

[0078] In one or more embodiments, deep learning compiler 310 may perform optimizations on multiple levels of a target system using graphics dialects, scheduling dialects, data movement dialects, memory dialects, and communication dialects.

[0079] Hereinafter, an example of a compilation process of the deep learning compiler 310 is described.

[0080] First, in operation 311, the deep learning compiler 310 may load a hardware representation providing system-level information about a target system. The deep learning compiler 310 may extract a plurality of hierarchies from the target system based on the loaded hardware representation.

[0081] In operation 320 , the deep learning compiler 310 may apply iterative compilation to the extracted multiple levels.

[0082] Hereinafter, an example of a process in which the deep learning compiler 310 applies a path pipeline to a level to perform a compilation process on the level is described. In one or more embodiments, the path pipeline may include a first path of graph-level optimization, a second path of partition optimization, a third path of computational scheduling optimization, a fourth path of memory or communication instrumentation, and a fifth path of memory or communication scheduling optimization. The deep learning compiler 310 may sequentially perform optimization in the order of graph-level optimization, partition optimization, computational scheduling optimization, memory or communication instrumentation, and memory or communication scheduling optimization.

[0083] For example, in operation 321, the deep learning compiler 310 may perform graph-level optimization by applying the first path to the corresponding hierarchy. In operation 322, the deep learning compiler 310 may perform partition optimization by applying the second path to the corresponding hierarchy. In operation 323, the deep learning compiler 310 may perform computation scheduling optimization by applying the third path to the corresponding hierarchy. In operation 324, the deep learning compiler 310 may perform memory or communication metering by applying the fourth path to the corresponding hierarchy. In operation 325, the deep learning compiler 310 may perform memory or communication scheduling optimization by applying the fifth path to the corresponding hierarchy. In operation 330, the deep learning compiler 310 may perform lowering of a high-level representation into a low-level representation.

[0084] Figure 4 An example of pseudo code is shown for determining a path for data placement in a memory hierarchy according to one or more embodiments.

[0085] To easily implement reusable paths across various target systems, deep learning compilers can be developed using hardware abstraction functions represented by the hardware (e.g. Figure 2 The deep learning compiler 210 and / or Figure 3 Each path in the deep learning compiler 310).

[0086] In one or more embodiments, a path may use hardware-represented views, properties, and hardware features to access specific hardware functions including memory allocation policies through memory hierarchies, memory capacities, and hardware. Paths may be used in various scenarios (such as a 2-level memory hierarchy including static random access memory (SRAM) and dynamic RAM (DRAM) in an NPU, or a 3-level memory hierarchy including high bandwidth memory (HBM), DRAM, and non-volatile memory express (NVMe)). A deep learning compiler may effectively generate paths that are agnostic to various targets (e.g., systems or devices) for optimization (including overlap of compute and memory operations, data placement, compute placement, and partitioning). Pseudocode 400 may be pseudocode generated by a deep learning compiler for a path that determines data placement in a memory hierarchy.

[0087] In one or more embodiments, the deep learning compiler may implement a path pipeline to be applied to each level. In this case, each of the multiple paths constituting the path pipeline may not depend on each system and may be constructed as a level applicable to all systems.

[0088] Figure 5 An example of the operation of a deep learning runtime and a deep learning compiler according to one or more embodiments is shown. Figure 5The operations may be performed in the order and manner shown, but without departing from the spirit and scope of the illustrated examples, the order of one or more of the operations may be changed, one or more of the operations may be omitted, and / or one or more of the operations may be performed in parallel or simultaneously.

[0089] The deep learning compiler in the embodiment (e.g., Figure 2 The deep learning compiler 210 and / or Figure 3 The deep learning compiler 310 of the system can use iterative compilation technology to perform compilation on a target system with a hierarchical structure. The core concept of the iterative compilation technology can be to separate the public path pipeline applicable to all levels of the system hierarchy from the path dedicated to a single level.

[0090] In one or more embodiments, the deep learning compiler may load a hardware representation of the target system in operation 511. In addition, the deep learning compiler may extract a plurality of levels from the target system based on the loaded hardware representation.

[0091] In operation 520, the deep learning compiler may perform iterative compilation on multiple levels. Figure 5 As shown in , the deep learning compiler can sequentially perform graph-level optimization, partition optimization, compute scheduling optimization, data movement metering, and data movement scheduling optimization on each level through multiple paths.

[0092] For example, in a GPU cluster as a hierarchy, data-level parallelism optimization can be applied through partition paths and computationally scheduled paths at the inter-node level. When this operation is in progress, the deep learning compiler may not include detailed architectural information about each component in the node (such as the type of GPU included in the GPU cluster, the number of interconnection systems and memory devices). Therefore, a typical deep learning compiler may not determine the data transmission method between devices, but can manage inter-node communication by mapping a small computational graph to each node. To solve this problem, the deep learning compiler of one or more embodiments may use a data movement dialect as an abstract representation that bridges the memory dialect and the communication dialect. For example, the deep learning compiler of one or more embodiments may represent operations related to memory and communication as intermediate representations related to data movement. The deep learning compiler may use a data movement dialect to perform optimization. For example, a deep learning compiler can apply the same path pipeline to an upper level by mapping information to a node instead of using information of the node's internal configuration; and can apply the same path pipeline to a lower level by mapping the information mapped to the node to a component of the node based on the information of the node's internal configuration; wherein the information mapped to the node includes: operations mapped to the node and data of a deep learning model mapped to the node.

[0093] In subsequent iterations, the deep learning compiler can progressively map computations and tensor operations to specific accelerators and memory devices in each node. As described above, the paths applied to the hierarchy can be configured to be agnostic to the system hierarchy. Therefore, by iteratively compiling the system, the compiler of one or more embodiments can support all system hierarchies at various levels from a single device such as a multi-core NPU (e.g., a mobile device, an augmented reality (AR) device, and a virtual reality (VR) device) to a large-scale cluster (e.g., a multi-node GPU cluster, a data center, and a supercomputer).

[0094] In one or more embodiments, in response to performing iterative compilation, in operation 530, the deep learning compiler may perform a reduction in memory or communication. In operation 540, the deep learning compiler may apply system-level dependent optimizations including memory or communication optimizations. In operation 550, the deep learning compiler may reduce common dialects to hardware-specific dialects (such as NPU, GPU, and processing-in-memory (PIM) dialects) to take advantage of hardware-specific optimizations that are not reusable. In addition, in operation 560, the deep learning compiler may generate an executable binary file or a binary executable format (BEF) file for runtime.

[0095] In one or more embodiments, when all optimizations are complete, the deep learning compiler (e.g., Figure 2 The deep learning compiler 210 and / or Figure 3 The deep learning compiler 310 of FIG. 310 may generate a target executable binary file (e.g., Figure 2 The generated target executable binary file may be sent to the deep learning runtime 570 (e.g., Figure 2 In order to achieve reusability in the deep learning runtime 570, the core runtime can be separated from the distributed runtime. The core runtime can be responsible for ensuring the management and efficient use of local resources, and the distributed runtime can focus on facilitating communication with other servers, thereby achieving distributed processing and scalability in the system architecture. This modular construction can promote flexibility and adaptability in deploying deep learning models in various target systems and distributed environments.

[0096] The core runtime can perform robust functions including automatic differentiation, model construction, and GPU acceleration. This functionality may include enhanced resource management, enhanced memory allocation, and advanced runtime monitoring capabilities. On the other hand, the components of the distributed runtime can focus on achieving efficient distributed processing of deep learning tasks by facilitating communication and coordination between multiple servers. Through this modular design, flexible scalability, efficient resource utilization, and smooth integration into a distributed environment can be achieved.

[0097] Figure 6 A table showing an example of a path to configure a deep learning optimization field according to one or more embodiments is shown.

[0098] As described above, the deep learning optimization field can be divided into five fields, which are a graph field, a partition field, a scheduling field, a memory field, and a communication field. Figure 6Table 600, graph-level optimization may include paths such as TypeElegalization, OperationLegalization, OperatorFusion, and Canonicalizer. Partitioning optimization may include paths such as PartitioningPropagation, PartitioningDecision, Inter-operatorPartitioning, and Intra-operatorPartitionin. Scheduling optimization may include paths such as ProgramDependenceAnalysis, ComputationPlacement, TensorPlacement, ExecutionOrder, and DataMovementInstrumentation. Memory optimization may include paths such as Bufferization, Memory Reuse Analysis, Remove Unnecessary MemOp, Static Allocator, FIFO Converter, Copy Channel Allocator, and Fragmentation Optimization. Communication optimization may include paths such as Communication Instrumentation, Explore Communication Patterns, and Communication Conversion.

[0099] Figure 7 An example of a process in which a deep learning compiler performs iterative compilation on a target system according to one or more embodiments is shown.

[0100] In one or more embodiments, a deep learning compiler (e.g., Figure 2 The deep learning compiler 210 and / or Figure 3 The deep learning compiler 310) can receive a hardware representation related to the target system 700, and can extract multiple levels from the target system 700 based on the hardware representation.

[0101] In one or more embodiments, the deep learning compiler may calculate the number of levels (e.g., 2 levels) constituting the target system 700 based on the hardware representation. The deep learning compiler may generate a common path pipeline to be applied to each level, and may iteratively apply the generated common path pipeline to the extracted multiple levels as many times as the calculated number of levels (e.g., 2 levels).

[0102] like Figure 7 As shown in , the hierarchy of the target system 700 may be 2 levels (e.g., a processor cluster (e.g., a GPU cluster) and an operating system (e.g., DGX A100)). For example, the processor cluster 710 (e.g., a GPU cluster) as an upper level may represent a system in which a plurality of GPUs are interconnected through a network. Figure 7 As shown in , GPU cluster 710 may include a switch and 8 DGX nodes connected to the switch. DGX nodes may be optimized high-performance server systems for deep learning tasks and may include multiple GPUs. Operating system 720 (e.g., DGX A100) may include 8 a100s. A100 may be an optimized high-performance server system for deep learning tasks and may include a specific core called a tensor core.

[0103] The deep learning compiler may perform a compilation process by applying a path pipeline to the GPU cluster 710 as an upper layer, and applying the same path pipeline to the DGX A100 720 as a lower layer. The deep learning compiler may generate an executable binary file by performing iterative compilation on multiple layers. The deep learning compiler may send the generated executable binary file to the deep learning runtime, and the deep learning runtime may execute the executable binary file.

[0104] Figure 8 An electronic device according to one or more embodiments is shown.

[0105] Reference Figure 8 , the electronic device 800 (eg, an electronic device) includes a processor 801 (eg, one or more processors), a memory 803 (eg, one or more memories), and a communication module 805. The electronic device 800 may include a processor configured to perform the above-referenced Figures 1 to 7 An apparatus for any one, any combination of any two or more, or all of the operations described. For example, the electronic device 800 may include user devices such as, for example, smart phones, personal computers and tablet PCs, augmented reality (AR) glasses, sensors, and servers.

[0106] In one example, the processor 801 may execute the above reference Figures 1 to 7Any one, any combination of any two or more, or all of the operations described. For example, processor 801 may include any one, any combination of any two or more of deep learning compiler 210, deep learning runtime module 230, graphics optimizer 240, performance estimator 250, deep learning compiler 310, GPU cluster 710, and operating system 720.

[0107] The memory 803 may be a volatile memory or a non-volatile memory and may store the same Figures 1 to 7 The memory 803 may include, for example, random access memory (RAM), dynamic RAM (DRAM), static RAM (SRAM), and / or other types of non-volatile memory known in the art.

[0108] The electronic device 800 according to one aspect may be connected to an external device, such as a server (eg, Figure 6 server 602), and can exchange data through the communication module 805.

[0109] In one example, the memory 803 may store the Figures 1 to 7 The processor 801 may execute the program or instruction stored in the memory 803 and may control the electronic device 800. The code of the program executed by the processor 801 may be stored in the memory 803. For example, the memory 803 may be or may include a non-transitory computer-readable storage medium storing instructions that, when executed by the processor 801, configure the processor 801 to perform Figures 1 to 7 Any one, any combination or all of the operations and methods.

[0110] In one example, the electronic device 800 may also include other components not shown here. For example, the electronic device 800 may also include an input / output (I / O) interface including an input device and an output device as a method for interfacing with the communication module 805. As another example, the electronic device 800 may also include other components (such as a transceiver, various sensors, and a database).

[0111] Here about Figures 1 to 8The disclosed and described UDLC framework, deep learning compiler, deep learning runtime module, graph optimizer, performance estimator, deep learning compiler, target system, processor cluster, operating system, electronic device, processor, memory, communication module, UDLC framework 200, deep learning compiler 210, deep learning runtime module 230, graph optimizer 240, performance estimator 250, deep learning compiler 310, target system 700, processor cluster 710, operating system 720, electronic device 800, processor 801, memory 803, communication module 805 and other devices, devices, units, modules and components are implemented or represent hardware components by hardware components. As described above, or in addition to the above description, examples of hardware components that can be used to perform the operations described in this application include, where appropriate: controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more hardware components in the hardware components for performing the operations described in the present application are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer may be implemented by one or more processing elements (such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field programmable gate arrays, programmable logic arrays, microprocessors, or any other device or combination of devices configured to respond and execute instructions in a limited manner to achieve the desired result). In one example, a processor or computer includes or is connected to one or more memories storing instructions or software executed by a processor or computer. The hardware components implemented by a processor or computer may execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) for performing the operations described in the present application. The hardware components may also access, manipulate, process, create, and store data in response to the execution of instructions or software. For simplicity, the singular term "processor" or "computer" may be used for the description of the examples described in the present application, but in other examples, multiple processors or computers may be used, or a processor or computer may include multiple processing elements or multiple types of processing elements or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller may implement a single hardware component or two or more hardware components.As described above, or in addition to the description above, the example hardware components may have any one or more of different processing configurations, where examples of different processing configurations include: a single processor, independent processors, parallel processors, single instruction single data (SISD) multiprocessing, single instruction multiple data (SIMD) multiprocessing, multiple instruction single data (MISD) multiprocessing, and multiple instruction multiple data (MIMD) multiprocessing.

[0112] Figures 1 to 8 The method for performing the operations described in the present application shown in the embodiment is performed by computing hardware (e.g., by one or more processors or computers), wherein the computing hardware is implemented as an execution instruction or software as described above to perform the operations performed by the method described in the present application. For example, a single operation or two or more operations may be performed by a single processor or two or more processors, or a processor and a controller. One or more operations may be performed by one or more processors, or a processor and a controller, and one or more other operations may be performed by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller may perform a single operation or two or more operations.

[0113] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform methods as described above may be written as computer programs, code segments, instructions, or any combination thereof to individually or collectively instruct or configure one or more processors or computers to operate as a machine or special-purpose computer to perform operations performed by the hardware components and methods as described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) directly executed by one or more processors or computers. In another example, the instructions or software include high-level code executed by one or more processors or computers using an interpreter. Instructions or software may be easily written using any programming language based on the block diagrams and flow charts shown in the accompanying drawings and the corresponding descriptions herein, which disclose algorithms for performing operations performed by the hardware components and methods as described above.

[0114] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above, as well as any associated data, data files, and data structures, may be recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media, rather than the signal itself. As described above, or in addition to the above description, examples of non-transitory computer-readable storage media include any one or more of the following: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-RLTH, BD-RE, Blu-ray or optical disk storage device, hard disk drive (HDD), solid state drive (SSSD), etc. D), flash memory, card memory (such as a multimedia mini card or card (e.g., secure digital (SD) or extreme digital (XD)), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk, and any other device configured to store instructions or software and any associated data, data files and data structures in a non-transitory manner and provide the instructions or software and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the instructions. In one example, the instructions or software and any associated data, data files and data structures are distributed on a networked computer system so that the instructions and software and any associated data, data files and data structures are stored, accessed and executed by one or more processors or computers in a distributed manner.

[0115] Although the present disclosure includes specific examples, it will be clear after understanding the disclosure of the present application that various changes in form and detail may be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein will be considered in a descriptive sense only, not for limiting purposes. The description of the features or aspects in each example will be considered applicable to similar features or aspects in other examples. If the described techniques are performed in a different order, and / or if the components in the described system, architecture, device, or circuit are combined in different ways and / or replaced or supplemented by other components or their equivalents, appropriate results may be obtained.

[0116] Therefore, the scope of the disclosure includes the claims and their equivalents in addition to the disclosure above and all the accompanying drawings, that is, all changes within the scope of the claims and their equivalents should be construed as being included in the disclosure.

Claims

1. An electronic device, include: The deep learning compiler is configured to: receiving a hardware representation corresponding to a target system including a hierarchical structure; extracting a plurality of layers from a target system based on the received hardware representation; as well as Iterative compilation is performed on the extracted multiple levels.

2. The electronic device according to claim 1, in, For the process of performing iterative compilation, the deep learning compiler is configured to sequentially apply a path pipeline indicating a sequence of paths to each of the extracted multiple levels from an upper level to a lower level.

3. The electronic device according to claim 2, in, For the process of performing iterative compilation, the deep learning compiler is also configured to: perform graph-level optimization, partition optimization, scheduling optimization, memory optimization, and communication optimization on each of the extracted multiple levels by applying a path pipeline to the extracted multiple levels.

4. The electronic device according to claim 2, in, The path pipeline includes: a graphics-level optimized path, a partition optimized path, a computation scheduling optimized path, a memory or communication metering path, and a memory or communication scheduling optimized path.

5. The electronic device according to claim 2, in, Each of the plurality of pathways constituting the pathway pipeline is configured to be applicable to a hierarchy of a plurality of systems including a target system without being dependent on a single system among the plurality of systems.

6. The electronic device according to claim 1, in, For the process of performing iterative compilation, the deep learning compiler is configured to perform optimization on the extracted multiple levels using a graphics dialect, a scheduling dialect, a data movement dialect, a memory dialect, and a communication dialect.

7. The electronic device according to any one of claims 1 to 6, in, For the process of performing iterative compilation, the deep learning compiler is configured to: calculating a count of the layers constituting the target system based on the received hardware representation; and The path pipeline is iteratively applied to the extracted plurality of levels as many times as the count of calculated levels.

8. The electronic device according to any one of claims 1 to 6, further comprising one or more processors comprising a deep learning compiler.

9. A method for iterative compilation implemented by a processor, the method include: receiving a hardware representation corresponding to a target system including a hierarchical structure; extracting a plurality of layers from a target system based on the received hardware representation; as well as Iterative compilation is performed on the extracted multiple levels.

10. The method according to claim 9, in, The step of performing iterative compilation includes sequentially applying a path pipeline indicating a sequence of paths to each of the extracted plurality of levels from an upper level to a lower level.

11. The method according to claim 10, in, The step of performing iterative compilation includes performing graph-level optimization, partition optimization, scheduling optimization, memory optimization, and communication optimization on each of the extracted multiple hierarchies by applying a path pipeline to the extracted multiple hierarchies.

12. The method according to claim 10, in, The path pipeline includes: a graphics-level optimized path, a partition optimized path, a computation scheduling optimized path, a memory or communication metering path, and a memory or communication scheduling optimized path.

13. The method according to claim 10, in, Each of the plurality of paths constituting the path pipeline is configured to be applicable to a hierarchy of a plurality of systems including a target system without depending on a single system among the plurality of systems.

14. The method according to claim 9, in, The step of performing iterative compilation includes performing optimization on the extracted plurality of hierarchies using a graphics dialect, a scheduling dialect, a data movement dialect, a memory dialect, and a communication dialect.

15. The method according to any one of claims 9 to 14, in, The steps to perform iterative compilation include: calculating a count of the layers constituting the target system based on the received hardware representation; and The path pipeline is iteratively applied to the extracted plurality of levels as many times as the count of the calculated levels.

16. A non-transitory computer-readable storage medium storing instructions which, when executed by one or more processors, configure the one or more processors to perform the method of any one of claims 9 to 15.

17. An electronic device, include: One or more processors configured to: receiving a hardware representation corresponding to a target system including a hierarchical structure; extracting a plurality of layers from a target system based on the received hardware representation; as well as Iterative compilation is performed on the extracted plurality of levels by applying the same path pipeline to each of the plurality of levels.

18. The electronic device according to claim 17, in, The multiple levels extracted include: an upper level including a plurality of nodes of a target system; and A lower level of a component including nodes in the plurality of nodes.

19. The electronic device according to claim 18, in, For performing the iterative compilation process, the one or more processors are configured to: Applying the same path pipeline to the upper level by mapping information onto the nodes instead of using information from the internal configuration of the nodes; as well as The same path pipeline is applied to the lower level by mapping the information mapped on the node to the components of the node based on the information of the internal configuration of the node.

20. The electronic device according to claim 19, in, The information mapped to the node includes: operations mapped to the node and data of the deep learning model mapped to the node.

Images