Method for evaluating performance of artificial neural network model and system using same

Abstract

A method for evaluating processing performance of an artificial neural network (ANN) model, the method comprising selecting a type and number of at least one neural processing unit (NPU) for processing performance evaluation of a user, selecting at least one of a plurality of compilation options of an artificial neural network (ANN) model to be processed by the selected at least one NPU, uploading the ANN model and the at least one evaluation data set to be processed by the selected at least one NPU, compiling the ANN model according to at least one of the selected plurality of compiling options, and reporting processing performance by processing the ANN model on the selected at least one NPU.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the priority of the following applications filed in the Korean Intellectual Property Office: Korean Patent Application No. 10-2023-0086192 filed on July 4, 2023, Korean Patent Application No. 10-2023-0170668 filed on November 30, 2023, and Korean Patent Application No. 10-2024-0003712 filed on January 9, 2024, the disclosures of which are incorporated herein by reference. Technical Field

[0002] The present invention relates to an artificial neural network model performance evaluation method and a system using the method. Background Art

[0003] Humans possess the intelligence to identify, classify, reason, predict, and make decisions. Artificial intelligence (AI) attempts to mimic this cognitive ability of humans. The human brain is a complex network of many nerve cells, called neurons. Each of these neurons forms hundreds or even thousands of connections with other neurons through synapses. In order to replicate human intelligence, the concept of artificial neural networks (ANNs) has been proposed. This involves modeling the functional principles of biological neurons and their interconnections using nodes connected in a layer structure. Summary of the invention

[0004] The embodiment relates to an artificial neural network (ANN) system. The ANN system includes multiple neural processors, a memory, and one or more operational processors. The neural processors include a first neural processor of a first configuration and a second neural processor of a second configuration different from the first configuration. The one or more operational processors receive an ANN model, a first selection of one or more neural processors including at least one of the first neural processor or the second neural processor for instantiating the ANN model, and a compilation option. The one or more operational processors instantiate at least one layer of the ANN model on the one or more neural processors of the first selection by compiling the ANN model according to the compilation option. The one or more operational processors instantiate at least one layer of the ANN model through the one or more neural processors of the first selection, perform processing on one or more evaluation data sets, and generate one or more first performance parameters, which are associated with the processing of the one or more evaluation data sets by the one or more neural processors of the first selection to instantiate at least one layer of the ANN model.

[0005] In one or more embodiments, the ANN system further includes a computing device. The computing device includes one or more processors, and a memory storing instructions thereon. The instructions cause the one or more processors to receive a first selection of one or more neural processors, one or more evaluation data sets, and compilation options from a user device via a network. The one or more processors send a first selection of one or more neural processors, one or more evaluation data sets, and compilation options to one or more operational processors. The one or more processors receive one or more first performance parameters from the one or more operational processors, and send the received one or more first performance parameters to the user device via the network.

[0006] In one or more embodiments, the instructions cause the one or more processors to protect the one or more evaluation data sets by at least one of data encryption, differential privacy, and data masking.

[0007] In one or more embodiments, the compilation options include selecting to improve the performance of the ANN model using at least one of a quantization algorithm, a pruning algorithm, a retraining algorithm, a model compression algorithm, an artificial intelligence (AI)-based model optimization algorithm, or a knowledge extraction algorithm.

[0008] In one or more embodiments, at least the first neural processor includes internal memory and a multiply-accumulator, and wherein the instructions further cause the one or more operational processors to automatically set at least one of the compilation options based on the first configuration.

[0009] In one or more embodiments, the instructions further cause the one or more processors to determine whether at least another layer in the ANN model is operational using the first selected one or more neural processors.

[0010] In one or more embodiments, the instructions further cause the one or more processors to generate an error report in response to determining that at least another layer in the ANN model is not operational using the first selected one or more neural processors.

[0011] In one or more embodiments, the ANN system further includes a graphics processor for processing at least another layer in the ANN model that was determined to be inoperable using the selected one or more neural processors.

[0012] In one or more embodiments, the graphics processor further performs retraining of the ANN model for instantiation on the first selected one or more neural processors.

[0013] In one or more embodiments, the one or more first performance parameters include at least one of temperature distribution, power consumption, operations per second per watt, frames per second (FPS), inferences per second (IPS), and accuracy of inference or prediction of the first selected one or more neural processors.

[0014] In one or more embodiments, the one or more operational processors receive: a second selection of one or more neural processors including at least one of a first neural processor or a second neural processor for instantiating an ANN model. The one or more operational processors instantiate at least one layer of the ANN model on the second selected one or more neural processors by compiling the ANN model; instantiate at least one layer of the ANN model through the second selected one or more neural processors, perform processing on one or more evaluation data sets, and generate one or more second performance parameters, the second performance parameters being associated with the processing of the one or more evaluation data sets by instantiating at least one layer of the ANN model through the second selected one or more neural processors.

[0015] In one or more embodiments, one or more operational processors generate a recommendation regarding a first selection of one or more neural processors or a second selection of one or more neural processors by comparing one or more first performance parameters and one or more second performance parameters, and send the recommendation to a user terminal.

[0016] In one or more embodiments, the received compilation option represents one of a plurality of preset options, wherein the plurality of preset options represent a combination of applying (i) post-training quantization (PTQ), (ii) layer-by-layer retraining of the ANN model, and (iii) quantization-aware retraining (QAT).

[0017] Embodiments also involve displaying an option for selecting one or more neural processors, the neural processors comprising a first neural processor in a first configuration and a second neural processor in a second configuration different from the first configuration. Receive a first selection from a user of one or more neural processors for instantiating at least one layer of an artificial neural network (ANN) model. A compilation option is associated with compilation of the ANN model for instantiating at least one layer. Receive a first selection of the compilation option from a user. Send the first selection, the selected compilation option, and one or more evaluation data sets to a computing device coupled to the one or more neural processors. Receive one or more first performance parameters associated with processing of the one or more evaluation data sets by the first selected one or more neural processors to instantiate at least one layer of the ANN model using the first selected compilation option. Display the one or more first performance parameters.

[0018] In one or more embodiments, a second selection of one or more neural processors and a second selection of compilation options are received from a user. The second selection of one or more neural processors and the selected compilation options are sent to a computing device coupled to the one or more neural processors. One or more second performance parameters are displayed, the second performance parameters associated with processing the one or more evaluation data sets by instantiating at least one layer of the ANN model with the second selected one or more neural processors using the second selected compilation options.

[0019] In one or more embodiments, a recommendation regarding using a first selection of one or more neural processors or a second selection of one or more neural processors is received and displayed. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a block diagram showing an artificial neural network model performance evaluation system according to an example of the present disclosure.

[0021] Figure 2 is a block diagram showing an artificial neural network model processing device according to an example of the present disclosure.

[0022] Figure 3 is a block diagram showing a compiler of an artificial neural network model processing device according to an example of the present disclosure.

[0023] Figure 4 is a block diagram showing an optimization module of an artificial neural network model processing device according to an example of the present disclosure.

[0024] Figure 5A is a diagram of a user interface for selecting one or more neural processors and selecting compilation options according to an example of the present disclosure.

[0025] Figure 5B is a diagram of a user interface for displaying performance reports and recommendations on one or more neural processing units according to an example of the present disclosure.

[0026] FIG. 6A to FIG. 6D is a block diagram showing various configurations of a neural processing unit of an artificial neural network model processing device according to an example of the present disclosure.

[0027] Figure 7 is a block diagram illustrating multiple neural processing units according to examples of the present disclosure.

[0028] Figure 8 is a flowchart illustrating a method of evaluating the performance of an artificial neural network model instantiated on one or more neural processing units according to an example of the present disclosure.

[0029] Fig. 9 is a flowchart illustrating a method of evaluating performance according to another example of the present disclosure.

[0030] Fig.10 is a flowchart illustrating a method of evaluating performance according to another example of the present disclosure. DETAILED DESCRIPTION

[0031] The advantages and features of the present disclosure will become apparent by referring to the examples described in detail in the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed herein and may be implemented in many different forms, and these embodiments are provided only to complete the present disclosure and fully inform the ordinary technicians in the field to which the present disclosure belongs. With respect to the description in the accompanying drawings, similar reference numerals may be used for similar elements.

[0032] In the present disclosure, expressions such as “having”, “may have”, “including” or “may include” refer to the presence of features (eg, values, functions, behaviors, or components such as parts) and do not preclude the presence of additional features.

[0033] In the present disclosure, expressions such as "A or B", "at least one of A or / and B", or "one or more of A or / and B" may include all possible combinations thereof. For example, "A or B", "at least one of A and B", or "at least one of A or B" may mean (1) including at least one A, (2) including at least one B, or (3) including both at least one A and at least one B.

[0034] As used herein, expressions such as "first", "second", "first or second" and the like can modify various elements, regardless of order and / or importance. The expression is only used to distinguish an element from other elements, without limiting these elements. For example, a first user device and a second user device can represent different user devices, regardless of order or importance. For example, without departing from the scope of rights described in this disclosure, a first element can be named as a second element, and similarly, a second element can also be renamed as a first element.

[0035] When an element (e.g., a first element) is referred to as being "operably or communicatively coupled / connected" or "connected to" another element (e.g., a second element), it should be understood that the element may be directly connected to the other element or may be connected through another element (e.g., a third element). On the other hand, when an element (e.g., a first element) is referred to as being "directly connected" or "directly connected to" another element (e.g., a second element), it should be understood that there are no other elements (e.g., a third element) between the element and the other element.

[0036] As used in the present disclosure, the expression "configured to" may be used interchangeably with, for example, "suitable for," "capable of," "designed to," "adapted to," "manufactured to," or "capable of," depending on the context. The term "configured (or manufactured for)" may not necessarily mean "specially designed" hardware. Instead, in some cases, the phrase "a device configured to perform" may mean that the device is "capable of" performing certain operations in conjunction with other devices or elements. For example, the phrase "a processor configured (or set) to perform A, B, and C" may refer to a processor dedicated to performing these actions (e.g., an embedded processor), or a general-purpose processor (e.g., a CPU or application processor) that can perform these actions by executing one or more software programs stored on a memory device.

[0037] The terms used in this disclosure are only used to describe specific examples and are not intended to limit the scope of other examples. Unless the context clearly stipulates otherwise, singular expressions may include plural expressions. The terms used herein, including technical or scientific terms, may have the same meaning as the meanings generally understood by those of ordinary skill in the art described herein. Among the terms used in this disclosure, the terms defined in the general dictionary may be interpreted as having the same or similar meanings as those in the context of the relevant technology. Unless clearly defined in this article, it should not be interpreted in an ideal or overly formal sense. In some cases, even the terms defined in this disclosure cannot be interpreted as excluding examples of the present disclosure. The terms used herein are only used to describe specific examples and are not intended to limit the present disclosure.

[0038] Each feature of the various examples of the present disclosure may be partially or completely combined or combined with each other. The various examples of the present disclosure are technically capable of implementing various interlocks and drives, which can be fully understood by those skilled in the art. Each example of the present disclosure may be implemented independently of each other, or may be implemented together in an associated relationship.

[0039] The present disclosure is intended to address, among other things, problems in the commercialization and deployment of neural processing units (NPUs, also referred to herein as "neural processors") for processing artificial neural network (ANN) models. First, there is a lack of information for selecting a suitable processor to process a user-developed artificial neural network model. Second, the commercialization of NPUs is still in its infancy, and determining whether a GPU-based artificial neural network model is suitable for a particular NPU involves reviewing various questionnaires and data sheets and technical support from engineers. In particular, the number of layers, parameter sizes, and special functions of ANN models are different, and therefore, there is no guarantee that certain ANN models can be instantiated using a particular NPU. Third, it is difficult to know in advance whether a user-developed artificial neural network model will run on a particular NPU. In other words, the purchased NPU may prove to not support certain types of calculations or operations for executing the user-developed ANN model. Fourth, when a user-developed neural network model is executed on a particular NPU, it is difficult to know in advance how it will perform in terms of performance (e.g., power consumption and frames per second (FPS)). In particular, due to differences in the size of weights, the size of feature maps, the number of layers, and the characteristics of activation functions, it is difficult to know in advance the expected performance.

[0040] The present disclosure facilitates the selection and deployment of NPU products by allowing users to test NPU products before purchasing and providing recommendations on appropriate selection of NPU products. Specifically, embodiments enable users to TM , PyTorch TM 、ONNX TM The ANN model can be compiled, then instantiated on the selected NPU product and executed on the evaluation dataset to determine the compatibility of the ANN model with the NPU product and also evaluate its performance.

[0041] Figure 1 1 is a block diagram of an ANN model performance evaluation system 1000 according to an example of the present disclosure. The ANN performance evaluation system 1000 may include a user device 100, an ANN model processing device 200, and a server 300 between the user device 100 and the ANN model processing device 200. Figure 1 The ANN model performance evaluation system 1000 can process a specific ANN model on the ANN model processing device 200 and provide the processing performance evaluation result of the ANN model processing device 200 to the user via the user device 100.

[0042] The user device 100 may be a device used by a user to obtain processing performance evaluation result information of the ANN model processed on the ANN model processing device 200. The user device 100 may include a smartphone, a tablet PC, a PC, a laptop computer, etc. that can be connected to the server 300, and may provide a user interface for viewing information related to the ANN model. The user device 100 may access the server 300, for example, via a network service, an FTP server, a cloud server, or an application software executable on the user device 100. These are merely examples, and various other known communication technologies or technologies to be developed may be used instead to connect to the server 300. The user may send the ANN model to the server 300 using various communication technologies. Specifically, the user may upload the ANN model and a specific evaluation data set to the server 300 via the user device 100 for evaluating the processing performance of the NPU that is a candidate for purchase by the user.

[0043] The evaluation data set refers to an input fed to the ANN model processing device 200 for the ANN model processing device 200 to perform performance evaluation.

[0044] The user device 100 may receive the performance evaluation result of the ANN model by the ANN model processing device 200 from the ANN model processing device 200, and may display the result. The user device 100 may be any type of computing device that can perform one or more of the following: (i) uploading the ANN model to be evaluated by the ANN model performance evaluation system 1000 to the server 300, (ii) uploading an evaluation data set for evaluating the ANN model to the ANN model performance evaluation system 1000, and (iii) uploading a training data set for retraining the ANN model to the ANN model performance evaluation system 1000. In other words, the user device 100 may be used as a data transmitter for evaluating the performance of the ANN model and / or a receiver for receiving and displaying the performance evaluation result of the ANN model.

[0045] To this end, the user device 100 may include components such as a processor 112, a display device 114, a user interface 116, a network interface 118, and a memory 120. The display device 114 may present options for selecting one or more NPUs for instantiating the ANN model, and also present options for compiling the ANN model, as described below with reference to Figure 5A and Figure 5BDetailed description. The memory 120 may store software modules (e.g., a web browser) that can be executed by the processor 112 to access the server 300, and also store ANN models and performance evaluation data sets for sending to the ANN model processing device 200 via the server 300. The user interface 116 may include a keyboard and a mouse, and enables the user to provide: user input associated with selecting on one or more NPUs to instantiate the ANN model; and compilation options associated with the compilation of the ANN model. The network interface 318 is a hardware component (e.g., a network interface card) that enables the user device 100 to communicate with the server 300 via a network.

[0046] The ANN model processing device 200 includes an NPU farm 218 for instantiating an ANN model received by the user device 100 via the server 300. The ANN model processing device 200 may also compile the ANN model for instantiation on one or more NPUs in the NPU farm 218, evaluate the performance of the instantiated ANN model, and report the performance results to the user device 100 via the server 300, as described below with reference to Figure 2 Described in detail.

[0047] The server 300 is a computing device that communicates with the user device 100 to manage access to the ANN model processing device 200 to test and evaluate one or more NPUs in the NPU farm 218. The server 300 may include components such as a processor 312, a network interface 316, and a memory 318. The network interface 316 enables the server 300 to communicate with the user device 100 and the ANN model processing device 200 via a network. The memory 318 stores instructions that can be executed by the processor 312 to perform one or more of the following operations: (i) managing the user's account, (ii) authenticating and allowing the user to access the ANN model processing device 100 to evaluate one or more NPUs, (iii) receiving an ANN model, an evaluation data set, a user's selection of an NPU to be evaluated, and a user's selection of compilation options, (iv) encrypting and storing data received from the user, (v) sending the ANN model and the user's selection information to the ANN model processing device 100 via the network, and (vi) forwarding a performance report on the selected NPU and a recommendation on the NPU to the user device 100 via the network. The server 300 may perform various other services, such as providing a marketplace to purchase user-evaluated NPUs.

[0048] To enhance the security of data received from users (e.g., user-developed ANN models, training datasets, evaluation datasets), the server 300 may enable users to securely log into their accounts and perform data encryption, differential privacy, and data masking.

[0049] Data encryption protects the confidentiality of data by encrypting user data. Differential privacy uses statistical techniques to desensitize user data to remove personal information. Data masking protects user data by masking part of the data to hide sensitive information.

[0050] In addition, the access control of the server 300 limits which accounts can access user data, audits the log records of accounts that have accessed user data, and maintains logs of system and user data access to track who and when accessed the model and detect abnormal activities. In addition, the uploading of training datasets and / or evaluation datasets can also involve signing a separate user data protection agreement to provide legal protection for the user's ANN model, training dataset, and / or evaluation dataset.

[0051] Figure 2 is a block diagram of an ANN model processing device 200 according to an example of the present disclosure. The ANN model processing device 200 may include components such as a central processing unit (CPU) 214, an NPU field 218 (including multiple NPUs 220), a graphics processing unit (GPU) 230, and a memory 250. These components may communicate with each other via one or more communication buses or signal lines (not shown).

[0052] CPU 214 may include one or more operating processors for executing instructions stored in memory 250. Memory 250 may store various software modules, including but not limited to compiler 210, storage module 240, and reporting program 260. Memory 250 may include a volatile or non-volatile recording medium that can store various data, instructions, and information. For example, memory 250 may include a storage medium of at least one of the following types: flash memory type, hard disk type, multimedia card micro, card type memory (such as SD or XD memory), RAM, SRAM, ROM, EEPROM, PROM, network storage, cloud, and blockchain database.

[0053] The compiler 210 may translate a particular ANN model into machine code or instructions that can be executed by multiple NPUs 220. In doing so, the compiler 210 may take into account the different configurations and characteristics of the NPUs 220 selected to instantiate and execute the ANN model. Because each type of NPU may have a different number of processing elements (or cores), different internal memory sizes, and channel bandwidths, the compiler 210 generates machine code or instructions that are compatible with one or more NPUs 220 selected to instantiate and execute the ANN model. To this end, the compiler 210 may store the configuration or capabilities of each type of NPU that can be used for evaluation and testing.

[0054] The compiler 210 may perform compilation based on various compilation options selected by the user. The compilation options may be provided as user interface (UI) elements on the screen of the user device 100, as described below with reference to Figure 5A and Figure 5B The compiler 210 may set multiple compilation options differently for each NPU selected for performance evaluation to generate compatible machine code or instructions. The multiple compilation options may vary for different types of NPUs 220, so that even for the same ANN model, the compiled machine code or instructions may vary for different types of NPUs 220 with different configurations.

[0055] The storage module 240 may store various data used by the ANN model processing device 200. That is, the storage device 240 may store an ANN model compiled into a machine code or instruction form for configuring a selected NPU 220; one or more training data sets; one or more evaluation data sets; performance evaluation results; and output data from multiple neural processing units 220.

[0056] The reporter 260 may determine whether the compiled ANN model is operable by multiple NPUs 220. If the compiled ANN model is not operable by multiple NPUs 220, the reporter 260 may report that one or more layers of the ANN model are not operable by the selected NPU 220, or that a specific operation associated with the ANN model is not operable. If the compiled ANN model is executable by a specific NPU, the reporter 260 may report the processing performance of the specific NPU.

[0057] Performance can be indicated by performance parameters such as temperature distribution, power consumption (watts), trillion operations per second per watt (TOPS / W), frames per second (FPS), inferences per second (IPS), and inference accuracy. Temperature distribution refers to the temperature change data of the NPU measured over time when the NPU is running. Power consumption refers to the power data measured when the NPU is running. Since power consumption depends on the computational load of the ANN model developed by the user, the user's ANN model can be provided and deployed for accurate power measurement. Trillion operations per second per watt (TOPS / W) is an indicator of the efficiency of AI accelerators, meaning the number of operations that can be executed per watt per second. TOPS / W is an indicator of the energy efficiency of multiple NPUs 220 because it indicates how many operations can be performed per unit of power consumption of the hardware. Inference per second (IPS) is an indicator of the number of inference operations that can be executed by multiple NPUs 220 in one second, thereby indicating the computational processing speed of multiple NPUs 220. IPS can also be called frames per second (FPS). Accuracy refers to the inference accuracy of multiple NPUs 220, as an indicator of the percentage of correctly predicted samples in the total number. As further explained, the accuracy of the multiple NPUs 220 and the inference accuracy of the graphics processing unit 230 may be different. This is because the parameters of the ANN model inferred by the graphics processing unit 230 may be in the form of floating points, while the parameters of the ANN model inferred by the multiple NPUs 220 may be in the form of integers. In addition, various optimization algorithms may be optionally applied. Therefore, the parameters of the ANN model inferred by the multiple NPUs 220 may have differences in values ​​calculated by various operations, and therefore may have different inference accuracy from the ANN model inferred by the graphics processing unit 230. The difference in inference accuracy may depend on the structure and parameter size characteristics of the ANN model, in particular, the shorter the length of the bit width of the quantization parameter, the greater the decrease in inference accuracy due to over-quantization. For example, the quantization bit width may be 2 bits to 16 bits. The decrease in inference accuracy due to over-pruning is also often greater.

[0058] The reporter 260 may analyze the processing performance of the ANN model compiled according to each compilation option and recommend one of the multiple compilation options. The reporter 260 may also recommend a specific type of NPU for instantiating the ANN model based on the performance parameters of the different NPUs. The evaluation data set may be used to evaluate different types or combinations of NPUs to determine the performance parameters associated with each type of NPU or NPU combination. Based on the comparison of the performance parameters, the reporter 260 may recommend a type of NPU or a combination of NPUs suitable for instantiating the ANN model.

[0059] The memory 250 may also store Figure 22. The memory 250 may store instructions for combining outputs from multiple selected NPUs. When multiple NPUs are selected to generate their own outputs, which are then combined or processed to generate the outputs of the corresponding ANN model, the combination or processing of the outputs from the NPUs may be performed by the CPU 214. Alternatively, such operations may be performed by the GPU 230 or one of the selected NPUs.

[0060] The NPU farm 218 may include various NPU families with different performance and price points sold by a specific company. The NPU farm 218 can be accessed online via the server 300 to perform performance evaluation of the ANN model developed by the user. The NPU farm 218 can be provided in the form of a cloud NPU. Multiple NPUs 220 can receive evaluation data sets as input and receive compiled ANN models for instantiation and performance evaluation. Multiple NPUs 220 may include various types of NPUs. In one or more embodiments, the NPU 220 may include different types of NPUs available from manufacturers.

[0061] More specifically, multiple NPUs 220 may be classified based on processing power. For example, the first NPU may be an NPU for smart CCTV. The first NPU may have the characteristics of ultra-low power, low-level reasoning processing power (e.g., 5TOPS processing power), very small semiconductor package size, and very low price. Due to performance limitations, the first NPU may not support certain ANN models that include certain operations and require high memory bandwidth. For example, the first NPU may have a model name "DX-V1" and may calculate ANN models such as ResNet, Mobilenet v1 / v2, SSD, YOLOv5, YOLOv7, etc. On the other hand, the second NPU may be an NPU for image recognition, object detection, and object tracking of robots. The second NPU may have the characteristics of low power, medium reasoning processing power (e.g., 16TOPS processing power), small semiconductor package size, and low price. The second NPU may not support certain ANN models that require high memory bandwidth. For example, the second NPU may have a model name "DX-V2" and may calculate ANN models such as ResNet, Mobilenet v1 / v2, SSD, YOLOv5, YOLOv7, etc. The third NPU may be an NPU for image recognition, object detection, object tracking, and generation of AI services for self-driving cars. The third NPU may have low power, high-level reasoning processing capabilities (e.g., 25TOPS of processing power), medium semiconductor package size, and medium price. For example, the third NPU may have a model name "DX-M1", which may calculate ANN models such as ResNet, MobileNet v1 / v2 / v3, SSD, EfficientNet, Efficient Det, YOLOv5, YOLOv7, YOLOv8, DeepLabv3, PIDNet, ViT, Generative Adversarial Networks, Stable Diffusion, etc. The fourth NPU may be an NPU for CCTV control rooms, control centers, large language models, and generation of AI services. The fourth NPU may have low power, high-level reasoning processing capabilities (e.g., 400TOPS of processing power), a large semiconductor package size, and high price characteristics. For example, the fourth NPU may have a model name "DX-H1" and may calculate ANN models such as ResNet, Mobilenet v1 / v2, SSD, YOLOv5, YOLOv7, YOLOv8, DeepLabv3, PIDNet, ViT, Generative Adversarial Networks, Stable Diffusion, and Large LLM. In other words, each NPU may have different computing processing capabilities, different semiconductor chip die sizes, different power consumption characteristics, and the like. However, the types of the multiple NPUs 220 are not limited thereto and may be classified by various classification criteria.

[0062] GPU 230 is hardware that performs complex computing tasks in parallel. GPU is widely used for graphics and image processing, but its application has been extended to processing various machine learning operations. Although GPU 230 is illustrated as a single device, it can be embodied as multiple graphics processing units connected by cloud GPU, NVLink, NVSwitch, etc. Graphics processing unit 230 includes multiple cores that process multiple tasks in parallel. Therefore, graphics processing unit 230 can perform large-scale data processing tasks such as scientific computing and deep learning.

[0063] Specifically, GPU 230 can be used to train deep learning and machine learning models on large data sets. Deep learning models have a large number of parameters, making training very time-consuming. GPU 230 can perform operations in parallel to generate or update parameters, thereby speeding up training. When a user selects a specific NPU from multiple NPUs 220 and performs retraining of an ANN model through various compilation options, GPU 230 can be used to retrain the ANN model according to each compilation option. In addition, when a layer of the ANN model is incompatible with instantiation on an NPU, GPU 230 can be used to instantiate the layer and perform processing on the instantiated layer.

[0064] In one or more embodiments, multiple NPUs 220 and one or more GPUs 230 may be implemented in the form of an integrated chip (IC), such as a system on a chip (SoC) combining various computing devices, or a printed circuit board on which an integrated chip is mounted.

[0065] Figure 3 2 is a block diagram showing a compiler 210 of an ANN model processing device 200 according to an example of the present disclosure. The compiler 210 can compile the ANN model into machine code or instructions based on multiple compilation options. The compiler 210 may be provided with hardware data of an NPU selected from a plurality of NPUs 220. The hardware data of the NPU may include the size of the NPU internal memory, the hierarchical structure of the NPU internal storage, information about the number of processing elements (or cores), information about special function units, etc. The compiler 210 may determine the processing order of each layer based on the hardware data of the NPU and the graphic information of the ANN model to be compiled. The machine code or instructions may be fed to one or more selected NPUs 220 to configure them to instantiate the ANN model. The compiler 210 may include components such as an optimization module 211, a verification module 212, and a code generator module 213.

[0066] The optimization module 211 may perform a task of modifying an ANN model represented by a directed acyclic graph (DAG) to improve one or more of efficiency, accuracy, and speed. The user may select at least one of the various optimization options provided by the optimization module 211 online via the user device 100. For example, the optimization module 211 may provide an option to convert a parameter of a specific bit width to a parameter of another bit width. The specific bit width may be between 2 bits and 16 bits. For example, when one or more selected NPUs 220 are designed to process integer parameters, the optimization module 211 may convert an ANN model based on floating point parameters to an ANN model based on integer parameters. When one or more selected NPUs 220 are designed to process piecewise linear function approximation operations, the optimization module 211 may also convert an ANN model based on nonlinear trigonometric operations to an ANN model based on piecewise linear function approximation. The optimization module 211 may also apply various optimization algorithms to reduce the size of parameters such as weights, feature maps, etc. of the ANN model. For example, the optimization module 211 may improve the accuracy degradation problem of the optimized neural network model by using various retraining algorithms.

[0067] The verification module 212 may perform verification to determine whether the user's ANN model is operable on one or more selected NPUs 220. The verification module 212 determines whether the ANN model is executable by analyzing the structure of the modified ANN model and determining whether the operation of each layer is supported by the hardware of the one or more selected NPUs 220. If the operation is not executable, a separate error report file may be generated and reported to the user.

[0068] The code generator module 213 may generate machine code or instructions for instantiating and executing the ANN model modified by the optimization module 211 on each selected NPU 220. In one embodiment, such generation of machine code or instructions may be performed only on ANN models determined by the verification module 212 to be operable on one or more selected NPUs 220. The generated machine code may be provided to program one or more selected NPUs 220 to instantiate the modified ANN model.

[0069] For example, first to fourth machine codes or instruction sets corresponding to the modified ANN model may be generated and fed to the first to fourth NPUs, respectively.

[0070] Figure 4is a block diagram showing an optimization module 211 according to an example of the present disclosure. The optimization module 211 may modify the ANN model based on a plurality of compilation options to enhance the ANN model in at least one of efficiency, speed, and accuracy. The compilation options may be set based on the hardware information of the NPU 220 used to instantiate the ANN model. Additionally or alternatively, the optimization module 211 may automatically set a plurality of compilation options taking into account the characteristics or parameters of the ANN model (e.g., the size of the weights and the size of the feature maps) and the characteristics of decreased inference accuracy. The plurality of compilation options set using the optimization module 211 may be at least one of a quantization option, a pruning option, a retraining option, a model compression option, an AI-based model optimization option, and a knowledge extraction option.

[0071] Activation of the pruning option may provide a technique for reducing ANN model calculations. The pruning algorithm may replace small near-zero values ​​with zeros in the weights of all layers of the ANN model, thereby making the weights sparse. Multiple NPUs 220 may skip multiplication operations associated with zero weights to speed up the calculation of convolutions, reduce power consumption, and reduce the size of parameters in the ANN model machine code through pruning options. Zeroing a specific weight parameter through pruning is equivalent to disconnecting the neuron corresponding to the weight data in the neural network. The pruning options may include a first value-based pruning option that removes smaller weights, or a second percentage-based pruning option that removes a certain percentage of the minimum weights.

[0072] Activation of the quantization option can provide techniques for reducing the parameter size of the ANN model. The quantization algorithm can selectively reduce the number of bits in the weights and feature maps of each layer of the ANN model. When the quantization option reduces the number of bits in a specific feature map and a specific weight, it can reduce the overall parameter size of the machine code of the ANN model. For example, a 32-bit parameter of a floating point can be converted to a parameter of a 2-bit to 16-bit integer when the quantization option is active.

[0073] Activation of the model compression option applies techniques for compressing weight parameters, feature map parameters, etc. of the ANN model. The model compression technique can be implemented using compression techniques known in the art. This can reduce the parameter size of the machine code of the ANN model with the model compression option. The model compression option can be provided to an NPU that includes a decompression decoder.

[0074] Activation of the knowledge extraction option applies a technique for transferring knowledge acquired from a complex model (also referred to as a teacher model) to a smaller, simpler model (also referred to as a student model). In the knowledge extraction algorithm, the teacher model typically has a larger parameter size and higher accuracy than the student model. For example, in the retraining option described later, the knowledge extraction option can be used to improve the accuracy of the student model, where an ANN model trained with floating point 32-bit parameters can be set as the teacher model, and an ANN model with various optimization options can be set as the student training model. The student model can be a model with at least one of the following options selected: a pruning option, a quantization option, a model compression option, and a retraining option.

[0075] Activation of the retraining option applies a technique that can compensate for reduced inference accuracy when various optimization options are applied. For example, when a quantization option, a pruning option, or a model compression option is applied, the accuracy of the ANN model inferred by the plurality of NPUs 220 may be reduced. In this case, an option to retrain the pruned, quantized, and / or model compressed neural network model online may be provided to restore inference accuracy. Specifically, the retraining options may include a quantization-aware retraining option, a pruning-aware retraining option, and a transfer learning option.

[0076] Activation of the Quantization Aware Retraining (QAT) option incorporates quantization into the retraining phase of the neural network model, where the model fine-tunes the weights to reflect the quantization error. The quantization aware retraining algorithm can include loss function, gradient calculation, and optimization algorithm modifications. The Quantization Aware Retraining option can compensate for quantization errors by quantizing the trained neural network model and then performing fine-tuning to retrain it in a way that minimizes the quantization loss.

[0077] The activation of the Pruning-Aware Retraining (PAT) option identifies and removes less important weights from the trained neural network model, and then fine-tunes the active weights. Pruning criteria can include weight values, activation values, and sensitivity analysis. The Pruning-Aware Retraining option can reduce the size of the neural network model, increase the inference speed, and compensate for overfitting during the retraining process.

[0078] Activation of the transfer learning option allows the ANN model to learn by transferring knowledge from one task to another related task. The transfer learning algorithm is effective when there is not enough data to start with, or when training a neural network model from scratch requires a lot of computing resources.

[0079] The optimization module 211 can apply artificial intelligence-based optimization to the ANN model without limitation. The artificial intelligence-based optimization algorithm can be a method of generating a reduced-size ANN model by applying various algorithms from the compilation options. This includes exploring the structure of the ANN model using an AI-based reinforcement learning method or a method not based on a reduction method (such as a quantization algorithm, a pruning algorithm, a retraining algorithm, a model compression algorithm, and a model compression algorithm), the latter being a method in which the artificial intelligence integrated in the optimization module 211 performs the reduction process by itself to obtain an improved reduction result.

[0080] Figure 5A 1 is a user interface diagram for selecting one or more neural processors and selecting compilation options according to an example of the present disclosure. The user interface may be presented on the display device 114 of the user device 100 after the user uses the user device 100 to access the server 300 .

[0081] The user interface diagram shows two sections, an NPU selection section 510 and a compilation options section 520. The user can select one or more NPUs in the NPU selection section 510 to run simulations on the ANN model using one or more evaluation data sets. In this example, four types of NPUs are displayed for selection, namely DX-M1, DX-H1, DX-V1, and DX-V2. The user can identify the number of NPUs to be used in the online simulation for evaluating performance. Figure 5A In the example of , a DX-M1 is selected for testing and evaluation. By providing non-zero numbers for multiple types of NPUs in the NPU selection section 510, combinations of different types of NPUs can be used in online simulation and evaluation.

[0082] The compilation option section 520 displays preset options to facilitate the user to select compilation options. Figure 5A In the example of , the compilation option part 520 displays the first preset option, the second preset option, and the third preset option. In one embodiment, each preset option can be the most effective quantitative preset option from a specific perspective. The user can select at least one preset option by considering the characteristics of each preset option.

[0083] For example, the first preset option is an option to execute only a quantization algorithm to convert the 32-bit floating-point data of the trained ANN model into 8-bit integer data. In other examples, the converted bit data can be determined by the hardware configuration of the selected NPU. The first preset option can be called post-training quantization (PTQ) because the quantization algorithm is executed after the training of the ANN model. The first preset option has the advantage of quickly performing quantization, usually within a few minutes. Therefore, it is advantageous to quickly check the results of power consumption, computing processing speed, etc. of the ANN model provided by the user on the NPU selected by the user. The first preset option including the first quantization option can be provided to the user as an option called "DXNN Lite". Retraining of the ANN model can be omitted in the first preset option.

[0084] The second preset option can execute a quantization algorithm that converts the 32-bit floating point data of the ANN model into 8-bit integer data, and then executes an algorithm for retraining the ANN model layer by layer. As in the first preset option, the converted bit data may depend on the hardware configuration of the selected NPU. Selecting the second preset option may cause the retraining algorithm to be performed layer by layer using the ANN model that executes the first preset option as an input model. Therefore, the second preset option may be a combination of a quantization algorithm and an algorithm from one of the various retraining options provided by the optimization module 211. In the second preset option, data corresponding to a portion of a layer in the ANN model is quantized, and its quantization loss function is calculated. Then, data corresponding to another portion of multiple layers of the ANN model is quantized, and its quantization loss function is calculated. Such operations are repeated to enhance quantization by reducing the quantization loss of some layers. The advantage of the second preset option is that retraining can be performed in a manner that reduces the difference between floating point data (e.g., floating point 32) and integer data (e.g., integer 8) in the feature map of each layer, so retraining can be performed even without a training data set. The advantage of the second preset option is that quantization can be performed in a reasonable amount of time and is usually completed within a few hours. The accuracy of the user-provided ANN model on the user-selected NPU of the plurality of NPUs 220 tends to be better than the accuracy obtained using the first preset option. The second preset option including the second quantization option may be provided to the user under the service name "DXNN pro." The second quantization option may involve a retraining step of the ANN model because it performs a layer-by-layer retraining of the ANN model.

[0085] The third preset option performs a quantization algorithm to convert the 32-bit data representing the floating point of the ANN model into 8-bit data representing an integer, and then performs a quantization-aware training (QAT) algorithm. In other words, the third preset option can further use the ANN model that executes the first preset option as an input model to perform a quantization-aware retraining algorithm. Therefore, the third preset option can be a combination of a quantization algorithm and an algorithm from one of the various retraining options provided by the optimization module 211. In the third preset option, the quantization-aware retraining algorithm performs fine-tuning by quantizing the trained ANN model and then retraining it in a manner that reduces the decrease in inference accuracy due to quantization. However, in order to retrain in a manner that reduces the decrease in inference accuracy due to quantization, the user can provide a training data set for the neural network model.

[0086] In addition, the evaluation dataset can be used to suppress overfitting during retraining. Specifically, the quantization-aware retraining algorithm inputs the training dataset of the machine code and the quantized ANN model into the corresponding NPU to retrain it and compensate for the decrease in inference accuracy due to quantization error.

[0087] The third preset option has the advantage of ensuring relatively higher inference accuracy than the first and second preset options, but generally takes several days to complete and is suitable when accuracy has a higher priority. The third preset option including the third quantization option can be provided to the user under the service name "DXNN master." The third quantization option may involve a retraining step of the ANN model because the retraining algorithm is performed based on the ANN model inference accuracy. For the quantization-aware retraining algorithm of the third quantization option, a training data set and / or an evaluation data set of the ANN model may be received from the user during the retraining process in the direction of reducing the loss caused by quantization. The training data set is used for quantization-aware retraining. The evaluation data set is optional data that can be used to improve overfitting problems during retraining.

[0088] Figure 5B is a diagram of a user interface for displaying performance reports and recommendations for selecting one or more neural processing units according to an example of the present disclosure. Figure 5B In the example shown, the results of simulation / evaluation performed using two different types of NPUs are shown. The top left box shows the results using the DX-M1 NPU, while the top right box shows the results using the DX-H1 NPU. The bottom box shows the recommended choice of NPU based on the performance parameters of the two different NPUs.

[0089] FIG. 6A to FIG. 6C 2 is a block diagram showing the configuration of various NPUs in the NPU field 218 according to an example of the present disclosure. Specifically, Fig. 6A shows the internal configuration of the first NPU 220, Figure 6B shows the internal configuration of the second NPU 220-1, and Figure 6C The internal configuration of the third NPU 220 - 2 is shown.

[0090] Fig. 6A The first NPU 220 includes a processing element array 221 (also referred to as a “processor core array 221”), an NPU internal memory 222, and an NPU controller 223. The first NPU 220 may include a processing element array 221, an NPU internal memory 222, and an NPU controller 223 that controls the processing element array 211 and the NPU internal memory 222.

[0091] The NPU internal memory 222 may store information such as parameters for instantiating a portion of the ANN model or the entire ANN model on the processing element array 221, intermediate outputs generated by each processing element, and at least a subset of the ANN model data. The ANN model to which various optimization options are applied may be compiled into machine code or instructions for execution by various components of the first NPU 220 in a coordinated manner.

[0092] The NPU controller 223 controls the operation of the processing element array 221 for the reasoning operation of the first NPU 220 and the read and write sequence of the NPU internal memory 222. If these components support multiple modes, the NPU controller 223 can also configure the processing elements and the NPU internal memory according to the programming mode. The NPU controller 223 also allocates task processing elements in the processing element array 221, instructs the processing elements to read data from the NPU internal memory 222 or write data to the NPU internal memory, and also coordinates receiving data from the storage module 240 or writing data to the storage module 240 according to the machine code or instructions generated as a result of the compilation. Therefore, the NPU can process the operation of each layer sequentially according to the ANN model structure. The NPU controller 223 can obtain the memory address where the ANN model feature map and weights are stored, or determine the memory address to be stored.

[0093] Processing element array 221 includes a plurality of processing elements (or cores) PE1 to PE12 arranged in an array. Each processing element may include a multiplication and accumulation (MAC) circuit and / or an arithmetic logic unit (ALU) circuit. However, as an addition or alternative to the MAC circuit and the ALU circuit in the processing element, other circuits may also be included. For example, a processing element may have a plurality of circuits that are implemented as multiplier circuits and / or adder tree circuits that operate in parallel, thereby replacing the MAC circuit within a single processing element. In this case, processing element array 221 may be referred to as at least one processing element including a plurality of circuits.

[0094] Processing element array 221 includes a plurality of processing elements PE1 to PE12. Fig. 6A The plurality of processing elements PE1 to PE12 shown in FIG. 1 are for illustrative purposes, and the number of the plurality of processing elements PE1 to PE12 is not limited to Fig. 6A The number of the plurality of processing elements PE1 to PE12 may determine the size or number of processing element array 221. Processing element array 221 may be in the form of an N x M matrix, where N and M are integers greater than zero.

[0095] The arrangement and number of processing element array 221 can be designed to take into account the characteristics of the ANN model. In particular, the number of processing elements can be determined by considering the data size of the ANN model to be operated, the required inference speed, the required power consumption, etc. The data size of the ANN model can correspond to the number of layers of the ANN model and the size of the weight parameters of each layer. As the number of processing elements in the processing element array 221 increases, the parallel computing capability of operating the ANN model also increases, but the manufacturing cost and physical size may also increase. For example, if Figure 6B As shown, the second NPU 220-1 may include two processing element arrays 221-1 and 221-2. The two processing element arrays 221-1 and 221-2 may be grouped, and each array may include a plurality of processing elements PE1 to PE12.

[0096] In another example, if Figure 6C As shown, the third NPU 220-2 may include four processing element arrays 221-1, 221-2, 221-3, and 221-4. The four processing element arrays 221-1, 221-2, 221-3, and 221-4 may be grouped, and each array may include a plurality of processing elements PE1 to PE12.

[0097] In another example, if Fig.6D As shown, the fourth NPU 220-8 may include eight smaller first NPUs 220, such as Fig. 6A As shown. Each of the eight first NPUs 220 is assigned to process part of the operation of the ANN model to further improve the ANN model speed. In addition, some of the first NPUs 220 can be deactivated during operation to save power consumption of the fourth NPU 220-8. For these purposes, in addition to the NPU controller 223 in each first NPU 220, the fourth NPU 220-8 may also include a more advanced NPU controller (not shown) to distribute the operations of each of the eight neural processing units and coordinate their operations.

[0098] The characteristics and processing models of the first to fourth neural processing units are described above.

[0099] Figure 7 2 is a block diagram showing the configuration of multiple NPUs in the NPU field 218 according to an example of the present disclosure. Multiple NPUs 220 may include NPUs of different types. At least one NPU of the same type may also be included in the NPU field 218. For example, multiple "DX-M1" NPUs may be arranged to form a first group G1, multiple "DX-H1" NPUs may be arranged to form a second group G2, multiple "DX-V1" NPUs may be arranged to form a third group G3, and multiple "DX-V2" NPUs may be arranged to form a fourth group G4. The NPU field 218 may be a cloud-based NPU system that is configured to respond in real time to performance evaluation requests from multiple users received via online communication. Depending on the user's selection, the multiple NPUs 220 included in the first group G1 to the fourth group G4 may all be used for performance evaluation, or a subset of these NPUs 220 may be used for performance evaluation.

[0100] Security-sensitive user data may be stored in the server 300 , in the storage device 240 of the ANN model processing device 200 , or in both the server 300 and the storage device 240 of the ANN model processing device 200 .

[0101] At least one NPU 220 for computing can communicate with the server 300 to receive at least one specific ANN model for NPU performance evaluation and at least one specific evaluation data set fed to the ANN model. In other words, the NPU 220 can process user data for performance evaluation.

[0102] Figure 8 is a flow chart illustrating a method for evaluating the performance of an artificial neural network model instantiated on one or more NPUs according to an example of the present disclosure. Figure 8 The ANN model performance evaluation method S100 may include a step S110 of receiving a selection of one or more NPUs for evaluation, a step S120 of receiving a selection of a compilation option, a step S130 of receiving an ANN model at a server 300, a step S140 of compiling the ANN model instantiated on the one or more selected NPUs according to the compilation option; and a step S150 of reporting a processing result of the one or more selected NPUs.

[0103] In the NPU type selection step S110, the user can select the type of NPU to be used for performance evaluation. The type of NPU may vary depending on the NPU product line sold by a specific company. Figure 7In the example of , multiple "DX-M1" NPUs can be arranged to form a first group G1, multiple "DX-H1" NPUs can be arranged to form a second group G2, multiple "DX-V1" NPUs can be arranged to form a third group G3, and multiple "DX-V2" NPUs can be arranged to form a fourth group G4. In this case, the user selects one or more NPUs from the "DX-M1" NPU, the "DX-H1" NPU, the "DX-V1" NPU, and the "DX-V2" NPU for evaluation. The user can select only one or more NPUs of a single type for evaluation, or select a combination of NPUs of different types for evaluation.

[0104] Then, in the compilation option selection step S120, for at least one selected NPU, at least one of the multiple compilation options of the ANN model to be processed is selected. More specifically, in the compilation option selection step S120, the compilation options can be set based on the hardware information of the NPU220. In addition, in the compilation option selection step, multiple compilation options can be set based on the user's selection. In one or more embodiments, a description of the advantages and disadvantages of each compilation option can be displayed on the user device 100. Therefore, the user can customize various compilation options to meet the user's needs. In other words, the performance evaluation system 1000 can provide user-customized compilation options instead of preset options to meet the specific needs of the user. As described above, the compilation option can be at least one of a pruning algorithm, a quantization algorithm, a model compression algorithm, a knowledge extraction algorithm, a retraining algorithm, and an AI-based model optimization algorithm. Alternatively, the compilation option can be configured to select one of the predefined preset options.

[0105] Then, in the ANN model receiving step S130, at least one specific ANN model for evaluating the performance of the selected NPU is received from the user device 100 at the server 300. This may also be referred to as a user data uploading step.

[0106] Then, in the ANN model compilation step S140, the received ANN model is compiled according to the selected compilation options for instantiation on one or more selected NPUs. Machine code or instructions are generated as a result of the compilation and fed to one or more NPUs to run the simulation.

[0107] In the step S150 of reporting the results, it is first determined whether the compiled ANN model can be processed by multiple neural processing units 220. If the compiled ANN model cannot be processed by multiple neural processing units 220, the ANN model processing result reporting step S150 may report the layers that cannot be processed by multiple neural processing units 22 among the multiple layers of the ANN model. Then, the layers that cannot be processed by multiple neural processing units 220 may be processed by the graphics processing unit 230. If the compiled ANN model can be processed by multiple neural processing units 220, the ANN model processing result reporting step S150 may report the processing performance of multiple neural processing units 220.

[0108] Parameters of processing performance may be temperature distribution of the neural processing unit, power consumption (watts), trillion operations per second per watt (TOPS / W), frames per second (FPS), inferences per second (IPS), accuracy, etc.

[0109] If the user does not provide an evaluation data set, the ANN model performance evaluation system 1000 can analyze the input data size of the ANN model to generate corresponding pseudo data, and can use the generated pseudo data to perform performance evaluation. For example, the pseudo data size can be (224x 224x 3), (288x288x 3), (380x 380x 3), (515x 512x 3), (640x640x 3), etc., but is not limited to these sizes. In other words, even if a data set for evaluating reasoning performance is not provided from the user, performance evaluation results such as power consumption, TOPS / W, FPS, IPS, etc. of the neural processing unit can be generated. However, in this case, since the pseudo data may not be accompanied by an accurate reasoning answer, the reasoning accuracy evaluation result may not be provided.

[0110] According to examples of the present disclosure, a user may quickly determine whether the user's ANN model is operable on a specific NPU before purchasing the specific NPU.

[0111] According to examples of the present disclosure, a user may quickly determine how the user's ANN model will perform when instantiated and executed on a particular NPU before purchasing the particular NPU.

[0112] According to an example of the present disclosure, if each NPU is connected via a server for each type of NPU, the user can evaluate the user's ANN model online and receive the results of each NPU available for purchase. Therefore, the performance evaluation system 1000 can provide the user with information about the performance and price of the neural processing unit required to implement the AI ​​service developed by the user, which can help the user make a quick purchase decision.

[0113] Fig. 9is a flow chart illustrating a method for evaluating the performance of an ANN model instantiated on one or more NPUs according to an example of the present disclosure. Fig. 9 The ANN model performance evaluation method S200 may include: a step S110 of receiving a selection of one or more NPUs for evaluation, a step S120 of receiving a selection of a compilation option, a step S230 of receiving an ANN model and an evaluation dataset at a server 300, a step S140 of compiling the ANN model for instantiation on one or more selected NPUs according to the compilation option, and a step S150 of reporting a result of processing the evaluation dataset using the one or more selected NPUs.

[0114] In the NPU type selection step S110, the user can select the type of NPU to be used for performance evaluation. The type of NPU may vary depending on the NPU product line sold by a particular company. Figure 7 In the example of , multiple "DX-M1" NPUs can be arranged to form a first group G1, multiple "DX-H1" NPUs can be arranged to form a second group G2, multiple "DX-V1" NPUs can be arranged to form a third group G3, and multiple "DX-V2" NPUs can be arranged to form a fourth group G4. In this case, the user selects one or more NPUs from the "DX-M1" NPU, the "DX-H1" NPU, the "DX-V1" NPU, and the "DX-V2" NPU for evaluation. The user can select only one or more NPUs of a single type for evaluation, or select a combination of NPUs of different types for evaluation.

[0115] Then, in the compilation option selection step S120, for at least one selected NPU, at least one of the multiple compilation options of the ANN model to be processed is selected. More specifically, in the compilation option selection step S120, the compilation options can be set based on the hardware information of the NPU220. In addition, in the compilation option selection step, multiple compilation options can be set based on the user's selection. In one or more embodiments, a description of the advantages and disadvantages of each compilation option can be displayed on the user device 100. Therefore, the user can customize various compilation options to meet the user's needs. In other words, the performance evaluation system 1000 can provide user-customized compilation options instead of preset options to meet the specific needs of the user. As described above, the compilation option can be at least one of a pruning algorithm, a quantization algorithm, a model compression algorithm, a knowledge extraction algorithm, a retraining algorithm, and an AI-based model optimization algorithm. Alternatively, the compilation option can be configured to select one of the predefined preset options.

[0116] Then, in step S230, at least one specific ANN model and at least one specific evaluation data set for evaluating the performance of the selected NPU are received from the user device 100 at the server 300. This may also be referred to as a user data upload step. The specific evaluation data set described refers to an evaluation data set that is fed to at least one specific ANN model instantiated by the ANN model processing device 200 for performance evaluation of the ANN model processing device 200.

[0117] Then, in the ANN model compilation step S140, the received ANN model is compiled according to the selected compilation options for instantiation on one or more selected NPUs. Machine code or instructions are generated as a result of the compilation and fed to one or more NPUs to run the simulation.

[0118] In the ANN model processing result reporting step S150, the performance evaluation results of the neural processing unit that processed the compiled ANN model can be reported. The performance evaluation result report can be stored in the user's account or sent to the user's email address. However, the performance evaluation results can be provided to the user in various other ways. The performance evaluation results are also considered user data and can be subject to the security policy applied to user data.

[0119] In the ANN model processing result reporting step S150, it is first determined whether the compiled ANN model can be processed by multiple neural processing units 220. If the compiled ANN model cannot be processed by multiple neural processing units 220, the ANN model processing result reporting step S150 may report the layers that cannot be processed by multiple neural processing units 220 among the multiple layers of the ANN model. Then, the layers that cannot be processed by multiple neural processing units 220 may be processed by the graphics processing unit 230. If the compiled ANN model can be processed by multiple neural processing units 220, the ANN model processing result reporting step S150 may report the processing performance of multiple neural processing units 220.

[0120] Parameters of processing performance may be temperature distribution of the neural processing unit, power consumption (watts), trillion operations per second per watt (TOPS / W), frames per second (FPS), inferences per second (IPS), accuracy, etc.

[0121] According to examples of the present disclosure, a user may quickly determine how the user's ANN model will perform when instantiated and executed on a particular NPU before purchasing the particular NPU.

[0122] According to an example of the present disclosure, if each NPU is connected via a server for each type of NPU, the user can evaluate the user's ANN model online and receive the results of each NPU available for purchase. Therefore, the performance evaluation system 1000 can provide the user with information about the performance and price of the neural processing unit required to implement the AI ​​service developed by the user, which can help the user make a quick purchase decision.

[0123] Reference Fig.10 , a method for evaluating the performance of an ANN model with a retraining step according to another example of the present disclosure will be described. Fig.10 is a flow chart illustrating a method for evaluating the performance of an ANN model instantiated on one or more NPUs according to an example of the present disclosure. Fig.10 The ANN model performance evaluation method S300 may include a step S110 of receiving a selection of one or more NPUs for evaluation, a step S120 of receiving a selection of a compilation option, a step S230 of receiving an ANN model and an evaluation dataset at the server 300, a step S140 of compiling the ANN model for instantiation on the one or more selected NPUs according to the compilation option, a step S345 of performing retraining on the ANN model, and a step S150 of reporting a result of processing the evaluation dataset using the one or more selected NPUs.

[0124] In the NPU type selection step S110, the user can select the type of NPU to be used for performance evaluation. The type of NPU may vary depending on the NPU product line sold by a particular company. Figure 7 In the example of , multiple "DX-M1" NPUs can be arranged to form a first group G1, multiple "DX-H1" NPUs can be arranged to form a second group G2, multiple "DX-V1" NPUs can be arranged to form a third group G3, and multiple "DX-V2" NPUs can be arranged to form a fourth group G4. In this case, the user selects one or more NPUs from the "DX-M1" NPU, the "DX-H1" NPU, the "DX-V1" NPU, and the "DX-V2" NPU for evaluation. The user can select only one or more NPUs of a single type for evaluation, or select a combination of NPUs of different types for evaluation.

[0125] Then, in the compilation option selection step S120, for at least one selected NPU, at least one of the multiple compilation options of the ANN model to be processed is selected. More specifically, in the compilation option selection step S120, the compilation options can be set based on the hardware information of the NPU220. In addition, in the compilation option selection step, multiple compilation options can be set based on the user's selection. In one or more embodiments, a description of the advantages and disadvantages of each compilation option can be displayed on the user device 100. Therefore, the user can customize various compilation options to meet the user's needs. In other words, the performance evaluation system 1000 can provide user-customized compilation options instead of preset options to meet the specific needs of the user. As described above, the compilation option can be at least one of a pruning algorithm, a quantization algorithm, a model compression algorithm, a knowledge extraction algorithm, a retraining algorithm, and an AI-based model optimization algorithm. Alternatively, the compilation option can be configured to select one of the predefined preset options.

[0126] Then, in step S230, at least one specific ANN model and at least one specific evaluation data set for evaluating the performance of the selected NPU are received from the user device 100 at the server 300. This may also be referred to as a user data upload step. The specific evaluation data set described refers to an evaluation data set that is fed to at least one specific ANN model instantiated by the ANN model processing device 200 for performance evaluation of the ANN model processing device 200.

[0127] Then, in the ANN model compilation and processing step S140, the input ANN model is compiled according to the selected compilation options, and the compiled machine code and evaluation data set are input to the selected neural processing unit within the NPU field for processing.

[0128] If the retraining option is selected in the compilation options, retraining of the ANN model may be performed in the retraining step S345. During retraining, the performance evaluation system 1000 may assign the graphics processing unit 230 to perform retraining on the ANN model processing unit 200. For example, in the retraining step S345 of the ANN model, the graphics processing unit 230 may receive an ANN model and a training data set to which a pruning algorithm and / or a quantization algorithm are applied as input to perform retraining. Retraining may be performed epoch by epoch, and may be performed on the graphics processing unit 230 for several to hundreds of epochs. The retraining options may include a quantization-aware retraining option, a pruning-aware retraining scheme, and a transfer learning scheme.

[0129] In the ANN model processing result reporting step S150, the performance evaluation result of the neural processing unit that processed the compiled ANN model may be reported. The performance evaluation result report may be stored in the user's account or sent to the user's email address. However, the performance evaluation result may be provided to the user in a variety of ways, including but not limited to Figure 5B Performance evaluation results are also considered user data and may be subject to the security policies that apply to user data.

[0130] In the ANN model processing result reporting step S150, it is first determined whether the compiled ANN model can be processed by multiple neural processing units 220. If the compiled ANN model cannot be processed by multiple neural processing units 220, the ANN model processing result reporting step S150 may report the layers that cannot be processed by multiple neural processing units 220 among the multiple layers of the ANN model. Then, the layers that cannot be processed by multiple neural processing units 220 may be processed by the graphics processing unit 230. If the compiled ANN model can be processed by multiple neural processing units 220, the ANN model processing result reporting step S150 may report the processing performance of multiple neural processing units 220.

[0131] Parameters of processing performance may be temperature distribution of the neural processing unit, power consumption (watts), trillion operations per second per watt (TOPS / W), frames per second (FPS), inferences per second (IPS), accuracy, etc.

[0132] According to another example of the present disclosure, a user may quickly determine whether the user's ANN model is operable on a specific NPU before purchasing the specific NPU.

[0133] According to another example of the present disclosure, a user may quickly determine how the user's ANN model will perform when run on a particular NPU before purchasing the particular NPU.

[0134] According to another example of the present disclosure, if each NPU is connected via a server for each type of NPU, the user may evaluate the user's ANN model online and receive results for each NPU available for purchase.

[0135] According to another example of the present disclosure, an ANN model retraining algorithm optimized for a specific neural processing unit may be performed online via the performance evaluation system 1000. In this case, user data may be separated and protected from the operator of the performance evaluation system 1000 by the above-mentioned security policy.

[0136] Therefore, the performance evaluation system 1000 can provide users with information about the performance and price of the neural processing unit required to implement the AI ​​service developed by the user, which can help users make quick purchasing decisions.

[0137] The present disclosure has been described in more detail with reference to the accompanying drawings, but the present disclosure is not necessarily limited to these examples and can be practiced in various modifications. Therefore, the embodiments disclosed herein are intended to illustrate rather than limit the technical concepts of the present disclosure, and the scope of the technical concepts of the present disclosure is not limited by these embodiments. Therefore, the embodiments described above should be understood to be exemplary and non-restrictive in all aspects. The scope of protection of the present disclosure should be interpreted according to the following claims, and all technical ideas within the same scope should be interpreted as included within the scope of the present disclosure. [National R&D project supporting this invention] [Task Number] 1711195792 [Task Number] 00228938 [Department Name] Ministry of Science and ICT [Project Management (Professional) Organization Name] Korea Information and Communication Planning and Evaluation Institute [Research Project Name] Development of AI Semiconductor SW Integration Platform Technology [Research Project Name] Commercial Edge AI SoC Semiconductor SW Development Platform Technology Development [Contribution rate] 1 / 1 [Project implementation unit name] DeepX [Study period] 2023.04.01~2023.12.31.

Claims

1. An artificial neural network (ANN) system, comprising: a plurality of neural processors including a first neural processor of a first configuration and a second neural processor of a second configuration different from the first configuration; one or more operational processors; as well as a memory having stored thereon instructions which, when executed by the one or more operational processors, cause the one or more operational processors to: receiving an ANN model, a first selection of one or more neural processors including at least one of the first neural processor or the second neural processor for instantiating the ANN model, and a compilation option, instantiating at least one layer of the ANN model on the first selected one or more neural processors by compiling the ANN model according to the compilation options, Instantiating, by the one or more neural processors selected by the first selection, the at least one layer of the ANN model, performing processing on one or more evaluation data sets, and One or more first performance parameters are generated, the first performance parameters being associated with processing of the one or more evaluation data sets by instantiating at least one layer of the ANN model by the one or more neural processors selected by the first selection.

2. The ANN system according to claim 1, further comprising a computing device, the computing device comprising: one or more processors, and a memory having stored thereon instructions that cause the one or more processors to: receiving, via a network, from a user device, the first selection of the one or more neural processors, the one or more evaluation data sets, and the compilation options, sending the first selection of the one or more neural processors, the one or more evaluation data sets, and the compilation options to the one or more operational processors, receiving the one or more first performance parameters from the one or more operational processors, and The one or more received first performance parameters are sent to the user equipment via the network.

3. The ANN system of claim 2, wherein the instructions cause the one or more processors to protect the one or more evaluation data sets by at least one of data encryption, differential privacy, and data masking.

4. The ANN system according to claim 1, wherein the compilation options include selecting to improve the performance of the ANN model using at least one of a quantization algorithm, a pruning algorithm, a retraining algorithm, a model compression algorithm, an artificial intelligence (AI)-based model optimization algorithm, or a knowledge extraction algorithm.

5. The ANN system of claim 1 , wherein at least the first neural processor comprises an internal memory and a multiply-accumulator, and wherein the instructions further cause the one or more operational processors to automatically set the at least one of the compilation options based on the first configuration.

6. The ANN system of claim 1 , wherein the instructions further cause the one or more processors to: The one or more neural processors of the first selection are used to determine whether at least another layer in the ANN model is operational.

7. The ANN system of claim 6, wherein the instructions further cause the one or more processors to: In response to determining, using the one or more neural processors of the first selection, that at least the other layer in the ANN model is not operational, generating an error report.

8. The ANN system of claim 6, further comprising a graphics processor configured to process at least the other layer in the ANN model in which the one or more neural processors selected are determined to be inoperable.

9. The ANN system of claim 8, wherein the graphics processor is further configured to perform retraining of the ANN model for instantiation on the first selected one or more neural processors.

10. The ANN system of claim 1, wherein the one or more first performance parameters include at least one of the following of the one or more neural processors of the first selection: temperature distribution, power consumption, operations per second per watt, frames per second (FPS), inferences per second (IPS), and inference or prediction accuracy.

11. The ANN system of claim 1 , wherein the instructions further cause the one or more operating processors to: receiving a second selection of one or more neural processors including at least one of the first neural processor or the second neural processor for instantiating the ANN model, instantiating the at least one layer of the ANN model on the second selected one or more neural processors by compiling the ANN model; Instantiating, by the one or more neural processors selected by the second selection, the at least one layer of the ANN model, performing processing on the one or more evaluation data sets, and One or more second performance parameters are generated, the second performance parameters being associated with processing of the one or more evaluation data sets by instantiating the at least one layer of the ANN model by the one or more neural processors selected by the second selection.

12. The ANN system of claim 11, wherein the instructions further cause the one or more operating processors to: generating a recommendation regarding the first selection of one or more neural processors or the second selection of one or more neural processors by comparing the one or more first performance parameters and the one or more second performance parameters, and The recommendation is sent to the user terminal.

13. The ANN system according to claim 1, wherein: The received compilation option represents one of a plurality of preset options, wherein the plurality of preset options represent a combination of applying (i) post-training quantization (PTQ), (ii) layer-by-layer retraining of the ANN model, and (iii) quantization-aware retraining (QAT).

14. A method comprising: receiving, via a network, by one or more operational processors, an artificial neural network (ANN) model, a selection of one or more neural processors including at least one of a first neural processor or a second neural processor for instantiating the ANN model, and a compilation option, the first neural network processor being a first configuration and the second neural processor being a second configuration different from the first configuration; instantiating at least one layer of the ANN model on the first selected one or more neural processors by compiling the ANN model according to the compiling options; Instantiating, by the one or more first neural processors of the first selection, the at least one layer of the ANN model to perform processing on one or more evaluation data sets; generating one or more first performance parameters associated with processing of the one or more evaluation data sets by instantiating at least one layer of the ANN model by the one or more neural processors selected by the first selection; as well as The generated one or more first performance parameters are transmitted via the network.

15. The method according to claim 14, further comprising: receiving, by a computing device, from a user device, the first selection of the one or more neural processors, the one or more evaluation data sets, and the compilation options; sending the first selection of the one or more neural processors, the one or more evaluation data sets, and the compilation options to the one or more operational processors; receiving the one or more first performance parameters sent from the one or more operational processors, and The one or more received first performance parameters are sent to the user equipment via the network.

16. The method according to claim 15, further comprising: At least one of data encryption, differential privacy, and data masking is performed by the computing device on the one or more evaluation data sets.

17. The method of claim 14, wherein the compilation options include selecting to improve the performance of the ANN model using at least one of a quantization algorithm, a pruning algorithm, a retraining algorithm, a model compression algorithm, an artificial intelligence (AI)-based model optimization algorithm, or a knowledge extraction algorithm.

18. The method of claim 14, further comprising automatically setting at least one of the compilation options based on the first configuration or the second configuration.

19. The method of claim 14, further comprising: In response to determining, using the one or more neural processors of the first selection, that at least another layer in the ANN model is inoperable, generating an error report.

20. The method of claim 14, further comprising: In response to determining that the other layer is not operable using the selected one or more neural processors, at least another layer in the ANN model is processed by a graphics processor.

21. The method of claim 14, further comprising: The ANN model is retrained by a graphics processor for instantiation on the one or more neural processors of the first selection.

22. The method of claim 14, wherein the one or more first performance parameters include at least one of the following of the one or more neural processors of the first selection: temperature distribution, power consumption, operations per second per watt, frames per second (FPS), inferences per second (IPS), and inference or prediction accuracy.

23. The method of claim 14, further comprising: receiving, by the one or more operational processors, a second selection of one or more neural processors including at least one of the first neural processor or the second neural processor for instantiating the ANN model, instantiating the at least one layer of the ANN model on the second selected one or more neural processors by compiling the ANN model; Instantiating, by the one or more neural processors selected by the second selection, the at least one layer of the ANN model, performing processing on the one or more evaluation data sets, and One or more second performance parameters are generated, the second performance parameters being associated with the processing of the one or more evaluation data sets by the one or more neural processors instantiating the at least one layer of the ANN model by the second selection.

24. The method of claim 23, further comprising: generating a recommendation regarding the first selection of one or more neural processors or the second selection of one or more neural processors by comparing the one or more first performance parameters and the one or more second performance parameters, and The recommendation is sent to the user terminal.

25. The method of claim 14, wherein the compilation option represents one of a plurality of preset options, the plurality of preset options representing a combination of applying (i) post-training quantization (PTQ), (ii) layer-by-layer retraining of the ANN model, and (iii) quantization-aware retraining (QAT).

26. A method comprising: displaying an option for selecting one or more neural processors, the neural processors comprising a first neural processor of a first configuration and a second neural processor of a second configuration different from the first configuration; receiving a first selection from a user of the one or more neural processors for instantiating at least one layer of an artificial neural network (ANN) model; displaying compilation options associated with compiling the ANN model for instantiating the at least one layer; receiving a first selection of the compilation option from the user; sending the first selection, the selected compilation options, and one or more evaluation data sets to a computing device coupled to the one or more neural processors; receiving one or more first performance parameters associated with processing the one or more evaluation data sets by instantiating at least one layer of the ANN model using a first selection of the compilation options and a first selection of the one or more neural processors; as well as The one or more first performance parameters are displayed.

27. The method of claim 26, further comprising: receiving a second selection of the one or more neural processors from the user; receiving a second selection of the compilation option from the user; sending the second selection and the selected compilation option to the computing device coupled to the one or more neural processors; as well as One or more second performance parameters are received, the second performance parameters associated with processing the one or more evaluation data sets by instantiating at least one layer of the ANN model using a second selection of the compilation options and a second selection of the one or more neural processors.

28. The method of claim 27, further comprising: receiving a recommendation to use the first selection of the one or more neural processors or the second selection of the one or more neural processors; as well as The recommendation is displayed.

Images