Apparatus and method for providing execution plans for multiple mixed-precision deep learning models based on multi-precision NPUs

Abstract

To provide an apparatus and method for providing execution plans of multiple mixed-precision deep learning models based on a multi-precision NPU that provide execution plans of mixed-precision deep learning models that can efficiently execute multiple mixed-precision deep learning models on a multi-precision NPU.SOLUTION: In an execution plans providing system, an execution plans providing apparatus includes a memory, and a processor 210 electrically connected to the memory. The processor is configured to form a multi-precision Neural Processing Unit (NPU) including a Processing Element (PE) composed of multiple Micro-PEs, to generate multiple mixed-precision deep learning models that perform multiple precision operations requiring different degrees of precision in a model execution process, and to generate execution plans for executing the multiple mixed-precision deep learning models on the multi-precision NPU.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an execution planning technique for mixed-precision deep learning models, and more particularly to a technique for providing an execution plan for mixed-precision deep learning models that enables multiple mixed-precision deep learning models to be efficiently executed on a multi-precision NPU. [Background technology]

[0002] A neural processing unit (NPU) is a processor optimized for deep learning algorithm calculations, which are the core of artificial intelligence, and is used to accelerate the learning and inference tasks of neural network models. Unlike existing central processing units (CPUs) and graphics processing units (GPUs), NPUs are specialized for deep learning calculations and can efficiently perform related tasks.

[0003] Mixed precision refers to the representation precision of data used when training or inferring a deep learning model. Precision is expressed as the number of bits used to represent data, and single precision (float32) and half precision (float16) are generally used. In other words, mixed precision is a technology that reduces the amount of calculations performed by deep learning models and optimizes memory usage by performing calculations using both single precision and half precision during model training and inference processes.

[0004] A mixed precision deep-learning model is a deep learning model built through mixed precision training. The training process of a deep learning model is performed on an NPU (Neural Processing Unit), and a multi-precision NPU is a hardware accelerator designed to efficiently support the execution of multiple precision operations, each requiring different precision levels, for model training.

[0005] An execution plan for a deep learning model on an NPU refers to a plan for computation and resource allocation for executing a deep learning model, and corresponds to a plan necessary for the deep learning model to be efficiently executed on the NPU and to achieve optimal performance. The execution plan is established taking into consideration the structure of a given deep learning model, input data, available hardware resources, etc., and includes the execution order for each layer or computation, the data transmission method, the degree of parallelization of computation, memory and storage allocation, etc. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Korean Patent Publication No. 10-2023-0099357 Summary of the Invention [Problem to be solved by the invention]

[0007] The present invention has been made in consideration of the above-mentioned conventional techniques, and an object of the present invention is to provide an apparatus and method for providing an execution plan for multiple mixed-precision deep learning models based on multi-precision NPUs. [Means for solving the problem]

[0008] To achieve the above object, according to one aspect of the present invention, an apparatus for providing an execution plan for multiple mixed-precision deep learning models based on a multi-precision NPU includes a memory and a processor electrically connected to the memory, wherein the processor forms a multi-precision NPU (Neural Processing Unit) having a processing element (PE) composed of a plurality of minimum units (Micro-PEs), and is configured to generate multiple mixed-precision deep learning models that perform multiple precision operations each requiring different precision during model execution, and generate an execution plan for executing the multiple mixed-precision deep learning models on the multi-precision NPU.

[0009] The processor may generate the multiple mixed-precision deep learning models via Hardware-Aware Mixed Precision Quantization (HAWQ). The processor may generate the execution plan based on the structure and characteristics of each of the multiple mixed-precision deep learning models to ensure efficient distribution of precision operations for each model. The processor may generate the execution plan using dynamic programming to minimize the execution time of each of the multiple mixed-precision deep learning models while ensuring efficient resource utilization. The processor may generate the execution plan as a result of applying the dynamic programming based on the results of measuring the execution time of all precision operations for each mixed-precision deep learning model through a pre-simulation method or an actual execution method. The processor may dynamically allocate at least one or more minimum units to execute each of the multiple precision operations in the process of executing the multiple mixed-precision deep learning models according to the execution plan. The processor may control the execution plan so that a precision calculation of a model having a higher execution priority among the plurality of mixed-precision deep learning models is executed first. The processor may track and monitor execution time and resource usage for precision operations of each mixed-precision deep learning model and dynamically adjust the execution plan.

[0010] To achieve the above object, according to one aspect of the present invention, an execution method for a computing device including a memory and a processor electrically connected to the memory for providing an execution plan for a plurality of mixed-precision deep learning models based on a multi-precision NPU includes the steps of: configuring, via the processor, a multi-precision NPU (Neural Processing Unit) including a processing element (PE) configured with a plurality of minimum units (Micro-PEs); generating, via the processor, a plurality of mixed-precision deep learning models that perform a plurality of precision operations, each requiring different precisions during model execution; and generating, via the processor, an execution plan for executing the plurality of mixed-precision deep learning models on the multi-precision NPU.

[0011] Generating the plurality of mixed-precision deep learning models may include generating the plurality of mixed-precision deep learning models via Hardware-Aware Mixed Precision Quantization (HAWQ). The step of generating the execution plan may include generating the execution plan based on the structure and characteristics of each of the multiple mixed-precision deep learning models so as to ensure efficient distribution of precision operations for each model. Generating the execution plan may include generating the execution plan using dynamic programming to minimize execution time for each of the plurality of mixed-precision deep learning models while ensuring efficient resource utilization. The step of generating the execution plan may include a step of generating the execution plan as a result of applying the dynamic programming based on the results of measuring the execution time of all precision operations for each mixed-precision deep learning model through a pre-simulation method or an actual execution method. The step of generating the execution plan may include analyzing data dependencies that occur during execution of each mixed-precision deep learning model, and optimizing the execution plan based on the analyzed data dependencies. The step of generating the execution plan may include detecting errors and exceptional situations that occur during execution of the mixed-precision deep learning model, and adding handling plans for the detected errors and exceptional situations to the execution plan. The method may further include dynamically allocating at least one or more atomic units that execute each of the multiple precision operations during the process of executing the multiple mixed-precision deep learning models according to the execution plan.

[0012] To achieve the above object, one aspect of the present invention provides a computer-readable recording medium storing a computer program, the computer program including instructions for causing a processor to execute a method including the steps of: configuring a multi-precision NPU (Neural Processing Unit) having a processing unit PE (Processing Element) composed of a plurality of minimum units (Micro-PEs); generating a plurality of mixed-precision deep learning models that perform a plurality of precision operations, each requiring different precisions during model execution; and generating an execution plan for executing the plurality of mixed-precision deep learning models on the multi-precision NPU. [Effects of the Invention]

[0013] The present invention has the following effects. However, this does not mean that a particular embodiment should include all or only the following effects, and therefore the scope of the disclosed technology should not be understood as being limited thereby.

[0014] The apparatus and method for providing an execution plan for multiple mixed-precision deep learning models based on a multi-precision NPU according to the present invention can provide an execution plan for mixed-precision deep learning models that enables multiple mixed-precision deep learning models to be efficiently executed on a multi-precision NPU.

[0015] In particular, the present invention does not depend on a mixed-precision deep learning model created through HAWQ, and can execute mixed-precision deep learning models created through other technologies as well as HAWQ. Furthermore, the present invention does not simply execute a mixed-precision deep learning model multiple times, but can execute multiple mixed-precision deep learning models simultaneously to pursue energy efficiency and fast execution times, and has the advantage of not requiring additional programs or hardware to provide an execution plan for such multiple mixed-precision deep learning models. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 is a diagram illustrating an execution plan providing system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram illustrating the system configuration of the execution plan providing device of FIG. [Figure 3] FIG. 3 is a diagram illustrating the functional configuration of the processor in FIG. 2. [Figure 4] 1 is a flowchart illustrating a method for providing an execution plan for a plurality of multi-precision NPU-based mixed-precision deep learning models according to an embodiment of the present invention. [Figure 5] FIG. 1 illustrates the structure of a framework for providing execution plans for multiple mixed-precision deep learning models based on multi-precision NPUs according to an embodiment of the present invention. [Figure 6] FIG. 10 is a diagram illustrating an example of an execution plan according to an embodiment of the present invention. [Figure 7] FIG. 10 is a diagram illustrating another example of an execution plan according to an embodiment of the present invention. [Figure 8] FIG. 10 is a diagram illustrating the performance difference of a multi-precision NPU to which an execution plan according to an embodiment of the present invention is applied. DETAILED DESCRIPTION OF THE INVENTION

[0017] The present invention was filed with the support of the following research project:

[0018] [National research and development project that supported this invention] [Project unique number] 1711198587 [Project number 00277060 [Ministry name] Ministry of Science, ICT and Communication [Name of issue management (specialized) organization] Information and Communications Planning and Evaluation Agency [Research Project Name: AI Semiconductor SW Integrated Platform Technology Development [Research title] Open Edge AI Semiconductor Design and SW Platform Technology Development [Project Implementation Institution Name] Yonsei University [Research period] 2023.06.01~2024.02.09

[0019] [National research and development project that supported this invention] [Project unique number] 1711193986 [Project Number] 2020-0-01361-004 [Ministry name] Ministry of Science, ICT and Communication [Name of issue management (specialized) organization] Information and Communications Planning and Evaluation Agency [Research Project Name] Information, Communications, and Broadcasting Innovation Human Resource Development [Research title] Artificial Intelligence Graduate School Support (Yonsei University) [Project Implementation Organization Name] Yonsei University Industry-Academia Cooperation Group [Research period] 2024.01.01~2024.12.31

[0020] Hereinafter, specific examples of embodiments of the present invention will be described in detail with reference to the drawings.

[0021] The description of the present invention is merely an embodiment for the purpose of structural or functional description, and therefore the scope of the present invention should not be construed as being limited by the embodiments described herein. In other words, since the embodiments can be modified in various ways and can have various forms, the scope of the present invention should be understood to include equivalents that can realize the technical idea. Furthermore, the objectives or effects presented in the present invention do not mean that a particular embodiment should include all of these or only such effects, and therefore the scope of the present invention should not be understood as being limited thereby.

[0022] Meanwhile, the meanings of the terms used in this specification should be understood as follows.

[0023] Terms such as "first" and "second" are used to distinguish one component from another and should not be used to limit the scope of rights. For example, a first component may be named a second component, and similarly, a second component may be named a first component.

[0024] When a component is said to be "connected" to another component, it should be understood that it is directly connected to the other component, but that there may be other components in between. Conversely, when a component is said to be "directly connected" to another component, it should be understood that there are no other components in between. Meanwhile, other expressions describing the relationship between components, such as "between" and "immediately between," or "adjacent to" and "directly adjacent to," should be interpreted in the same way.

[0025] Singular expressions shall be understood to include plural expressions unless the context clearly dictates otherwise, and terms such as "comprise" or "have" shall be understood to specify the presence of embodied features, numbers, steps, operations, components, parts, or combinations thereof, and shall not be understood to preclude the presence or additional possibility of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0026] In each step, the identifiers (e.g., a, b, c, etc.) are used for convenience of explanation and do not describe the order of each step, and each step may occur in a different order than specified unless the context clearly dictates a specific order. That is, each step may occur in the same order as specified, may occur substantially simultaneously, or may occur in the reverse order.

[0027] The present invention is embodied as computer-readable code on a computer-readable recording medium, which includes all types of recording devices that store data readable by a computer system. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. The computer-readable recording media may also be distributed across computer systems connected via a network, so that the computer-readable code is stored and executed in a distributed manner.

[0028] Unless otherwise defined, all terms used herein have the same meaning as commonly understood by a person of ordinary skill in the art to which this invention pertains. Terms defined in commonly used dictionaries should be interpreted to be consistent with the meaning they have in the context of the relevant art, and should not be interpreted as having an ideal or overly formal meaning unless expressly defined herein.

[0029] FIG. 1 is a diagram illustrating an execution plan providing system 100 according to an embodiment of the present invention.

[0030] As shown in FIG. 1, the execution plan providing system 100 includes a user terminal 110, an execution plan providing device 130, and a database 150.

[0031] The user terminal 110 corresponds to a computing device operated by a user. Here, the user corresponds to an entity that executes and manages model execution on a multi-precision NPU according to an execution plan for multiple mixed-precision deep learning models. The user terminal 110 may be implemented as a smartphone, a notebook, or a computer, but is not limited thereto, and may be implemented as various devices such as a tablet PC. The user terminal 110 is implemented as one device constituting the execution plan providing system 100 according to the present invention, and the execution plan providing system 100 may be implemented in various forms depending on the purpose of providing the execution plan.

[0032] The user terminal 110 is connected to the execution plan providing device 130 via a network, and multiple user terminals 110 are simultaneously connected to the execution plan providing device 130. The user terminal installs and executes a dedicated program or application for interfacing with the execution plan providing device 130.

[0033] The execution plan providing device 130 is realized by a computer or a server that executes the method for providing execution plans for a plurality of multi-precision NPU-based mixed-precision deep learning models according to the present invention. Here, the method for providing execution plans for a plurality of multi-precision NPU-based mixed-precision deep learning models according to the present invention includes a plurality of operational steps defined in a process of performing a series of processes for generating an execution plan when a user request for providing an execution plan is received from the user terminal 110, and providing the resulting generated execution plan to the user terminal 110.

[0034] The execution plan providing device 130 is connected to the user terminal 110 via a wired network or a wireless network such as Bluetooth, WiFi, LTE, etc., and transmits and receives data to and from the user terminal 110 via the network. The execution plan providing device 130 may also be implemented to operate while connected to an independent external system (not shown in FIG. 1). For example, the execution plan providing device 130 may operate in conjunction with an independent external system that configures a multi-precision NPU or generates a mixed-precision deep learning model, but is not limited thereto, and may operate in conjunction with one or more external systems that independently perform one or more steps constituting the method according to the present invention.

[0035] The database 150 corresponds to a storage device that stores various information required in the operation process of the execution plan providing device 130. The database 150 stores tools for configuring a multi-precision NPU and stores information for generating a mixed-precision deep learning model, but is not necessarily limited thereto, and may store information collected or processed in various forms in the process of the execution plan providing device 130 performing the method for providing execution plans for a plurality of mixed-precision deep learning models based on a multi-precision NPU according to the present invention.

[0036] Also, in FIG. 1, the database 150 is illustrated as a device independent of the execution plan providing device 130, but this is not necessarily limited to this, and it can of course be realized as a logical storage device included in the execution plan providing device 130.

[0037] FIG. 2 is a diagram illustrating the system configuration of the execution plan providing device 130 in FIG.

[0038] As shown in FIG. 2, the execution plan providing device 130 includes a processor 210, a memory 230, a user input / output unit 250, and a network input / output unit 270.

[0039] The processor 210 executes a procedure for providing execution plans for multiple mixed-precision deep learning models based on a multi-precision NPU according to an embodiment of the present invention, manages the memory 230 that is read or created during this process, and schedules synchronization time between the volatile memory and non-volatile memory in the memory 230. The processor 210 controls the overall operation of the execution plan providing device 130 and is electrically connected to the memory 230, the user input / output unit 250, and the network input / output unit 270 to control the flow of data therebetween. The processor 210 is implemented as a central processing unit (CPU) or a graphics processing unit (GPU) of the execution plan providing device 130. In one embodiment, the processor 210 is implemented to operate in conjunction with one or more multi-precision NPUs.

[0040] The memory 230 includes a secondary storage device implemented as a non-volatile memory such as a solid state disk (SSD) or a hard disk drive (HDD) and used to store all data required for the execution plan providing device 130, and a primary storage device implemented as a volatile memory such as a random access memory (RAM). The memory 230 also stores a set of instructions that are executed by the processor 210 electrically connected thereto to perform the method for providing execution plans for multiple mixed-precision deep learning models based on multi-precision NPUs according to the present invention.

[0041] The user input / output unit 250 includes an environment for receiving user input and an environment for outputting specific information to a user, and includes an input device with an adapter, such as a touchpad, touch screen, visual keyboard, or pointing device, and an output device with an adapter, such as a monitor or touch screen. In one embodiment, the user input / output unit 250 corresponds to a computing device connected via a remote connection, and in such a case, the execution plan providing device 130 is implemented as an independent server.

[0042] The network input / output unit 270 provides a communication environment for connecting to the user terminal 110 via a network, and includes adapters for communication with, for example, a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a value-added network (VAN), etc. The network input / output unit 270 is also implemented to provide a short-range communication function such as WiFi or Bluetooth (registered trademark) or a wireless communication function of 4G or higher for wireless data transmission.

[0043] FIG. 3 is a diagram illustrating the functional configuration of the processor 210 in FIG.

[0044] 3, the execution plan providing device 130 executes the method for providing execution plans for a plurality of multi-precision NPU-based mixed-precision deep learning models according to the present invention. To this end, the processor 210 of the execution plan providing device 130 is implemented to include a plurality of functional components. Specifically, the processor 210 includes an NPU configuration unit 310, a deep learning model generation unit 330, an execution plan generation unit 350, a model execution unit 370, and a control unit 390.

[0045] The processor 210 of the present invention does not necessarily include all of the above functional components at the same time, but may omit some of the above components or selectively include some or all of the above components depending on the embodiment. The operation of each functional component will be specifically described below.

[0046] The NPU configuration unit 310 configures a multi-precision NPU (Neural Processing Unit) including a processing element (PE) configured with a plurality of minimum units (Micro-PEs). Here, the multi-precision NPU corresponds to a hardware device specialized for performing deep learning and artificial neural network operations. In particular, the multi-precision NPU is realized with a structure optimized for efficiently processing matrix operations and operations requiring various precisions performed in the model learning and inference processes. More specifically, the multi-precision NPU is configured with a plurality of processing elements (PEs), each of which is configured with one or more minimum units (Micro-PEs). The performance and functions of the NPU are determined based on the structure between the processing elements and the minimum units.

[0047] The arithmetic unit is one of the core components of a multi-precision NPU and handles the main operations and calculations. The arithmetic unit is designed to perform a specified operation on input data and then generate an operation result. The minimum unit (Micro-PE) is a smaller unit that makes up the arithmetic unit and is designed to perform small and simple operations. The minimum unit is used to perform operations such as multiplication, addition, and activation functions, and the functions of the arithmetic unit are formed by combining the operations of the minimum unit. In other words, the minimum unit is an independent subunit within the arithmetic unit and is used to improve the parallel processing and flexibility of the arithmetic unit.

[0048] The NPU configuration unit 310 also provides an interface for configuring a multi-precision NPU (Neural Processing Unit), and the interface is designed so that a user can set various settings and parameters to define the operation of the multi-precision NPU. That is, a user inputs configuration information for the multi-precision NPU, and the configuration information includes information such as the characteristics of the work performed by the NPU, calculation precision, memory allocation, etc. The NPU configuration unit 310 analyzes the characteristics of the work requested by the user and requirements for necessary resources based on the configuration information input by the user, and then reflects this in the configuration of the NPU to configure the multi-precision NPU, including details of the hardware resources and algorithms required for the NPU to perform the work.

[0049] The deep learning model generation unit 330 generates multiple mixed-precision deep learning models that perform multiple precision operations, each requiring different precision during the model execution process. A mixed-precision deep learning model is a deep learning model that combines various types of precision operations and is designed to include different precision and detailed conditions depending on the characteristics of the operation. Precision operations are classified according to the representation precision of the data used in the operation, and operations that require higher precision produce more accurate results but require more computing resources. For example, a 32-bit floating-point operation requires lower precision than a 64-bit floating-point operation.

[0050] The deep learning model generation unit 330 also analyzes the model structure and the characteristics of each operation. Through this, the deep learning model generation unit 330 identifies the precision required for each operation and distinguishes between operations that require high precision and operations that can use low precision. The deep learning model generation unit 330 also generates a mixed-precision deep learning model using the precision data type corresponding to each operation based on the identified precision of the operation. Meanwhile, the deep learning model generation unit 330 operates in conjunction with the user terminal 110 and can generate multiple mixed-precision deep learning models based on configuration information set by the user.

[0051] In one embodiment, the deep learning model generation unit 330 generates multiple mixed-precision deep learning models through hardware-aware mixed precision quantization (HAWQ). That is, the deep learning model generation unit 330 performs additional optimization operations based on a given model or a model configured by a user to generate a model that minimizes execution time and memory usage while maintaining model accuracy. To this end, the deep learning model generation unit 330 uses hardware-aware mixed precision quantization (HAWQ). However, this is not limited to this, and it goes without saying that the mixed-precision deep learning model can be generated through other techniques besides HAWQ. Meanwhile, detailed description of hardware-aware mixed precision quantization (HAWQ) will be omitted here. The model optimization process includes a process of adjusting and quantizing the operations of each layer to optimal precision.

[0052] The execution plan generator 350 generates an execution plan for executing multiple mixed-precision deep learning models on a multi-precision NPU. To this end, the execution plan generator 350 analyzes the hardware structure of the multi-precision NPU and analyzes the structure and operation characteristics of the mixed-precision deep learning models. For example, the execution plan generator 350 extracts information about the operation amount, memory requirements, operation type, etc. of each model, and extracts information about the NPU's operation speed and resource constraints, etc. The execution plan generator 350 generates an optimal execution strategy based on the analysis results. Here, the execution strategy includes a method for efficiently utilizing the resources of the multi-precision NPU to execute multiple mixed-precision deep learning models in parallel. In addition, the execution plan generator 350 allocates resources within the NPU for processing the operations of each model according to the generated execution strategy, schedules the operations, and arranges the operations of each model in an optimal execution order through this process, thereby minimizing the overall execution time and maximizing resource utilization.

[0053] In one embodiment, the execution plan generator 350 generates an execution plan based on the structure and characteristics of each of the multiple mixed-precision deep learning models to ensure efficient distribution of precision operations for each model. The execution plan generator 350 determines the type of operation required by each model, the possibility of parallelization of the operation, the memory requirements, etc. based on the model structure and characteristics, and determines whether operations can be parallelized between the multiple models, memory access, etc., to generate an efficient distribution strategy for precision operations for each model. The execution plan generator 350 generates an execution plan based on the distribution strategy, including the order and precision of operations to be used when each model is executed, the amount of resource allocation, etc. That is, the execution plan generator 350 establishes an efficient distribution strategy for precision operations based on the structure and characteristics of each mixed-precision deep learning model, and generates an execution plan.

[0054] In one embodiment, the execution plan generator 350 generates an execution plan using dynamic programming to minimize the execution time of each of the multiple mixed-precision deep learning models while ensuring efficient resource utilization. Dynamic programming is an algorithm design technique for solving optimization problems, which divides a problem into multiple subproblems based on input size and generates a solution to the overall problem based on solutions to each subproblem. More specifically, the execution plan generator 350 defines resource allocation and scheduling problems required to perform each precision operation for each mixed-precision deep learning model as subproblems. The execution plan generator 350 generates and combines solutions to the subproblems based on each operation step of the mixed-precision deep learning model. Once the calculations for all operation steps are completed, the execution plan generator 350 generates an optimal execution plan that minimizes the execution time based on the overall operation step while ensuring efficient resource utilization.

[0055] In one embodiment, the execution plan generator 350 generates an execution plan as a result of applying dynamic programming based on the results of measuring the execution time of all precision operations for each mixed-precision deep learning model through a pre-simulation method or an actual execution method. The execution plan generator 350 selectively applies the pre-simulation method or the actual execution method to generate an optimal execution plan. In this case, the execution time of the precision operations is measured differently depending on the method used, and the execution plan generator 350 applies dynamic programming by reflecting the error in the execution time depending on the selected method.

[0056] In one embodiment, the execution plan generator 350 detects errors and exceptional situations that occur during the execution of a mixed-precision deep learning model and adds a handling plan for the detected error or exceptional situation to the execution plan. For example, when a very large value is handled during model calculation, an overflow or an underflow for a very small value occurs, which affects the numerical stability of the model. In this case, the execution plan generator 350 adds a handling plan for limiting the value range using a method such as normalizing the value to the execution plan. As another example, when a NaN (Not a Number) value occurs during model calculation due to an erroneous calculation or data loss, the execution plan generator 350 adds a handling plan for replacing the NaN value with another value or for aborting the calculation to the execution plan.

[0057] In one embodiment, the execution plan generator 350 analyzes data dependencies that occur during the execution of each mixed-precision deep learning model and optimizes the execution plan based on the analyzed data dependencies. Here, data dependencies refer to cases where the result of one operation affects other operations. In the case of deep learning models, each operation is performed on input data, and data dependencies exist between the operations. For example, data dependencies occur between operations when the output of one operation is used as the input of another operation.

[0058] Specifically, the execution plan generator 350 analyzes the structure of the mixed-precision deep learning model to be executed and determines data dependencies between operations. That is, the execution plan generator 350 analyzes data dependencies based on the connection between the input and output of each operation. The execution plan generator 350 generates a dependency graph that represents data dependencies between operations. The dependency graph represents each operation as a node and data dependencies as edges, thereby representing the flow of data between operations within the model. The execution plan generator 350 then optimizes the generated execution plan by applying various techniques based on the generated dependency graph. For example, the execution plan generator 350 optimizes the execution plan by applying techniques such as minimizing data loading and storage, constructing a batch of operations that can be executed in parallel, or minimizing data movement between operations, thereby improving processing speed.

[0059] The model execution unit 370 dynamically allocates at least one minimum unit to execute each of multiple precision operations in the process of executing multiple mixed-precision deep learning models according to an execution plan. The model execution unit 370 dynamically allocates minimum units to process the execution of operations, taking into consideration the type, precision, priority, etc. of each operation specified in the execution plan. The model execution unit 370 determines the precision operation to be executed according to the execution plan and allocates one or more minimum units based on the type of operation and the required precision. In this case, the model execution unit 370 determines the number of minimum units to be allocated, taking into consideration the order of operations, the number of operations that can be executed simultaneously, etc. In addition, the model execution unit 370 manages the minimum units allocated during the execution of operations, monitors the model and resource status, and reallocates or releases resources as necessary.

[0060] In one embodiment, the model execution unit 370 controls the execution of precision calculations of a model with a higher execution priority among multiple mixed-precision deep learning models according to an execution plan. That is, the model execution unit 370 manages the priority among multiple models according to the execution plan, determines the execution order of operations for each model according to the execution priority, and adjusts the execution priority of models based on changes in the execution status and resource status. For example, unlike the execution plan, the model execution unit 370 readjusts the execution priority when a new model is added or the status of a model currently being executed changes, and manages the execution plan to suit an environment that is different from the environment at the time the execution plan was generated.

[0061] In one embodiment, the model executor 370 tracks and monitors the execution time and resource usage for precision operations of each mixed-precision deep learning model and dynamically adjusts the execution plan. The model executor 370 tracks various precision operations required by each mixed-precision deep learning model and measures the execution time of each operation. The model executor 370 also monitors the resource usage of each operation by tracking and quantifying the memory and computing resources used during the operation process of each model. If the model executor 370 detects an operation with a long execution time or high resource consumption, it adjusts the execution plan to maintain optimal performance or minimize resource consumption. The model executor 370 stores the tracked information and dynamic adjustment results in the database 150, and the collected information is subsequently used in the process of generating an execution plan for the mixed-precision deep learning model.

[0062] The control unit 390 controls the overall operation of the processor 210 and manages the flow of control or data between the NPU configuration unit 310, the deep learning model generation unit 330, the execution plan generation unit 350, and the model execution unit 370.

[0063] FIG. 4 is a flowchart illustrating a method for providing an execution plan for multiple mixed-precision deep learning models based on multi-precision NPUs according to an embodiment of the present invention.

[0064] As shown in Fig. 4, the execution plan providing device 130 processes a series of operational steps for executing a method for providing an execution plan via the processor 210. Specifically, the execution plan providing device 130 configures a multi-precision NPU (Neural Processing Unit) including a processing element (PE) composed of a plurality of micro-PEs via the processor 210 (step S410). The execution plan providing device 130 generates a plurality of mixed-precision deep learning models that perform a plurality of precision operations, each requiring different precision levels, during the model execution process via the processor 210 (step S430). The execution plan providing device 130 generates an execution plan for executing the plurality of mixed-precision deep learning models on the multi-precision NPU via the processor 210 (step S450).

[0065] FIG. 5 illustrates a framework structure for providing an execution plan for multiple mixed-precision deep learning models based on a multi-precision NPU according to an embodiment of the present invention.

[0066] As shown in FIG. 5, the execution plan providing device 130 generates a mixed-precision model through HAWQ based on a given model. In this case, this operation is not dependent on HAWQ and is replaced by another technology that provides a similar function. The execution plan providing device 130 then generates an execution plan based on the generated mixed-precision model. The execution plan providing device 130 uses publicly available software called MLIR to perform this operation, but is not necessarily limited to this. As a result, the execution plan providing device 130 efficiently executes multiple mixed-precision models on a multi-precision NPU through the generated execution plan.

[0067] FIG. 6 is a diagram for explaining an example of an execution plan according to one embodiment of the present invention, and FIG. 7 is a diagram for explaining another example of an execution plan according to one embodiment of the present invention.

[0068] 6 and 7, the execution plan providing device 130 provides an execution plan for efficiently executing multiple mixed-precision deep learning models on a multi-precision NPU. In particular, the execution plan providing device 130 eliminates idle time of one or more minimum processing units (Micro-PEs) to efficiently utilize the multi-precision NPU and improve overall execution time.

[0069] 6, the execution plan providing device 130 generates an execution plan that allows up to four operations A that can be executed simultaneously, which allows low accuracy, and generates an execution plan that allows up to two operations B that require a medium level of accuracy to be executed simultaneously. The execution plan providing device 130 generates an efficient execution plan by forming a batch of operations that can be executed simultaneously in order to eliminate idle time of the smallest unit (Micro-PE).

[0070] 7, the vertical length of each rectangle represents the execution time (t1 and t2) of the corresponding operation, with the upper two rectangles taking longer to execute than the operations corresponding to the lower two rectangles. In other words, when constructing a batch of operations that can be executed simultaneously, the execution plan providing device 130 takes into account the execution time of each operation, thereby preventing an increase in idle time before the start of the next operation due to the operation of a specific minimum unit (Micro-PE) being completed first.

[0071] As a result, the execution plan providing device 130 can generate an execution plan that executes multiple mixed-precision models quickly and energy-efficiently on a multi-precision NPU by applying the techniques shown in FIGS. 6 and 7 based on the framework shown in FIG. 5.

[0072] FIG. 8 is a diagram illustrating the performance difference of a multi-precision NPU to which an execution plan according to an embodiment of the present invention is applied.

[0073] As shown in Figure 8, the x-axis of the graph represents execution time, with a shorter x-axis representing faster execution of multiple mixed-precision deep learning models. The x-axis of the graph also represents results normalized based on the case where no optimization was applied. In other words, Figure 8 demonstrates that executing multiple mixed-precision deep learning models according to the execution plan of the present invention allows for more efficient use of the multi-precision NPU.

[0074] More specifically, eight types of mixed-precision models with various levels of precision and detailed conditions are used in the performance comparison experiments, and a total of 12 types of combinations of the mixed-precision model groups are used.

[0075] Experimental results showed that when optimization was not applied, the overall structure of each model could not be understood in multiple mixed-precision models, and a bundle of low-precision operations that could be performed simultaneously could not be constructed, resulting in idle time in the Micro-PE and inefficient resource utilization of the multi-precision NPU.

[0076] In contrast, in the case of "optimized application," the method according to the present invention is applied to construct an appropriate bundle of low-precision operations, thereby eliminating idle time in the Micro-PE, resulting in efficient resource utilization of the multi-precision NPU. In other words, the method according to the present invention can achieve an average execution time improvement of 1.2 times.

[0077] Although the embodiments of the present invention have been described in detail above with reference to the drawings, the present invention is not limited to the above-described embodiments and can be modified in various ways without departing from the technical concept of the present invention. [Explanation of symbols]

[0078] 100 Action Plan Providing System 110 User terminal 130 Execution plan providing device 150 databases 210 processors 230 memory 250 User input / output unit 270 Network Input / Output Unit 310 NPU component 330 Deep Learning Model Generation Unit 350 Execution Plan Generation Unit 370 Model Execution Department 390 Control Unit

Claims

1. An apparatus for providing an execution plan for a plurality of multi-precision NPU-based mixed-precision deep learning models, comprising: Memory and a processor electrically coupled to the memory; The processor: A multi-precision NPU (Neural Processing Unit) is configured having a processing element (PE) composed of a plurality of minimum units (Micro-PEs), Generate multiple mixed-precision deep learning models that perform multiple precision calculations, each requiring different precision during the model execution process, 1. An apparatus configured to generate an execution plan for executing the plurality of mixed-precision deep learning models on the multi-precision NPU.

2. The apparatus of claim 1 , wherein the processor generates the execution plan based on the structure and characteristics of each of the multiple mixed-precision deep learning models to ensure efficient distribution of precision operations for each model.

3. 2. The apparatus of claim 1, wherein the processor uses dynamic programming to generate the execution plan to minimize execution time for each of the plurality of mixed-precision deep learning models while ensuring efficient resource utilization.

4. The device of claim 3, wherein the processor generates the execution plan as a result of applying the dynamic programming based on a result of measuring the execution time of all precision operations for each mixed-precision deep learning model through a pre-simulation method or an actual execution method.

5. The device according to claim 1 , wherein the processor dynamically allocates at least one or more atomic units that execute each of the multiple precision operations in the process of executing the multiple mixed-precision deep learning models according to the execution plan.

6. The device according to claim 5 , wherein the processor controls the execution plan so that a precision calculation of a model having a higher execution priority among the plurality of mixed-precision deep learning models is executed first.

7. 6. The apparatus of claim 5, wherein the processor tracks and monitors execution time and resource usage for precision operations of each mixed-precision deep learning model to dynamically adjust the execution plan.

8. 1. A method for providing an execution plan for a plurality of multi-precision NPU-based mixed-precision deep learning models in a computing device, the method comprising: configuring a multi-precision Neural Processing Unit (NPU) including a processing element (PE) configured with a plurality of minimum units (Micro-PEs) via the processor; generating, via the processor, a plurality of mixed-precision deep learning models that perform a plurality of precision operations, each of which requires a different precision during model execution; and generating, via the processor, an execution plan for executing the plurality of mixed-precision deep learning models on the multi-precision NPU.

9. 9. The method of claim 8, wherein generating the execution plan includes generating the execution plan based on the structure and characteristics of each of the multiple mixed-precision deep learning models in a manner that ensures efficient distribution of precision operations for each model.

10. 10. The method of claim 8, wherein generating the execution plan comprises using dynamic programming to generate the execution plan to minimize execution time for each of the plurality of mixed-precision deep learning models while ensuring efficient resource utilization.

11. 11. The method of claim 10, wherein the step of generating the execution plan includes generating the execution plan as a result of applying the dynamic programming based on results of measuring execution times of all precision operations for each mixed-precision deep learning model through a pre-simulation method or an actual execution method.

12. 10. The method of claim 8, wherein generating the execution plan comprises analyzing data dependencies that occur during execution of each mixed-precision deep learning model, and optimizing the execution plan based on the analyzed data dependencies.

13. 10. The method of claim 8, wherein generating the execution plan comprises detecting errors and exceptional situations that occur during execution of the mixed-precision deep learning model, and adding handling plans for the detected errors and exceptional situations to the execution plan.

14. 9. The method of claim 8, further comprising dynamically allocating at least one or more atomic units that execute each of the multiple precision operations during the execution of the multiple mixed-precision deep learning models according to the execution plan.

15. A computer-readable recording medium storing a computer program, The computer program, when executed by a processor, A step of configuring a multi-precision NPU (Neural Processing Unit) including a processing element (PE) configured with a plurality of minimum units (Micro-PEs); generating a plurality of mixed-precision deep learning models that perform a plurality of precision operations each requiring different precision during the model execution process; and generating an execution plan for executing the plurality of mixed-precision deep learning models on the multi-precision NPU.

Images