Method for evaluating an artifitial neural network model performance and system using the same

Abstract

An artificial neural network model performance evaluation system is disclosed. The ekFMS artificial neural network model performance evaluation system disclosed herein includes a plurality of neural processors, each comprising a first neural processor of a first configuration and a second neural processor of a second configuration different from the first configuration, at least one operating processor, and a memory for storing instructions used by the at least one operating processor. The at least one operating processor receives a first processor selection information for selecting at least one of an artificial neural network model, a first neural processor, and a second neural processor, and compile option information. By compiling the artificial neural network model according to the compile option, the system processes the artificial neural network model used by the neural processor selected by the first processor selection information, applies at least one evaluation data set to the artificial neural network model by the neural processor selected by the first processor selection information, generates at least one first performance parameter related to the performance of the neural processor selected by the first processor selection information, and can report a layer or an operation among a plurality of layers of the artificial neural network model that cannot be processed by the neural processor selected by the first processor selection information.

Description

Technology Field

[0001] The present disclosure relates to a method for evaluating the performance of an artificial neural network model, a system using the same, and a system using the same. Background Technology

[0002] Humans possess intelligence capable of recognition, classification, inference, prediction, and control / decision-making. Artificial Intelligence (AI) refers to the artificial imitation of human intelligence.

[0003] The human brain is composed of countless nerve cells called neurons. Each neuron is connected to hundreds to thousands of other neurons through connection points called synapses. An Artificial Neural Network (ANN) model is a system that models the operating principles of biological neurons and the relationships between them in order to mimic human intelligence. In other words, an artificial neural network model is a system that connects nodes, which mimic neurons, in a layered structure. Prior art literature

[0004] U.S. Patent Application Publication US2022 / 0147813 (May 12, 2022) Published Patent Application No. 10-2020-0110092 (September 23, 2020) Published Patent Application No. 10-2023-0080305 (June 7, 2023) The problem to be solved

[0005] The inventor of the present disclosure recognized that there are various problems to be solved for the commercialization of a neural processing unit (NPU) that drives an artificial neural network model.

[0006] First, there is a lack of information in selecting a processor to run the artificial neural network model developed by the user.

[0007] Second, NPUs are a technology that has only just begun commercialization, and determining whether a GPU-based artificial neural network model will run on a specific NPU requires reviewing various questionnaires and data sheets, as well as technical support from engineers. In particular, because the number of layers and parameter sizes can change and special functions can be added depending on user needs, the range of variation is so wide that it is difficult to generalize the artificial neural network model.

[0008] Third, therefore, it is difficult to know in advance whether an artificial neural network model developed by a user will run on a specific NPU. In other words, problems may arise where the model cannot run after purchasing and evaluating it, as it may be found that certain computations or operations are not supported.

[0009] Fourth, it is difficult to know in advance what performance a user-developed artificial neural network model will achieve when running on a specific NPU. In other words, it is difficult to know in advance whether the desired power consumption and FPS are satisfied.

[0010] In particular, it is difficult to predict the desired performance in advance because the size of weights, the size of feature maps, the number of layers, and the characteristics of activation functions vary for each artificial neural network model.

[0011] Accordingly, the inventors of the present disclosure have AI code (e.g., TensorFlow TM , PyTorch TM , ONNX TMA method and apparatus are configured to enable faster determination of the optimal NPU product selection and model optimization conditions on the selected NPU by providing a solution or service that provides the best convenience and value to the user by performing all necessary series of tasks online in batches when a model file, etc., is dropped (uploaded) to a specific online simulation service.

[0012] The problems of the present disclosure are not limited to those mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below. means of solving the problem

[0013] The present disclosure relates to an artificial neural network model performance evaluation system, comprising a plurality of neural processors including a first neural processor of a first configuration and a second neural processor of a second configuration different from the first configuration, each having different computational processing performance, at least one operating processor, and a memory for storing instructions used in at least one operating processor. The at least one operating processor receives first processor selection information for selecting at least one of an artificial neural network model, a first neural processor, and a second neural processor, and compile option information, and compiles the artificial neural network model according to the compile option, thereby processing the artificial neural network model used in the neural processor selected by the first processor selection information, applying at least one evaluation data set to the artificial neural network model by the neural processor selected by the first processor selection information, and generating at least one first performance parameter related to the performance of the neural processor selected by the first processor selection information.

[0014] According to another feature of the present disclosure, an artificial neural network model performance evaluation system may further include a server that receives the first processor selection information, the compilation option information, and the at least one evaluation data set from a user device via a network, sends the first processor selection information, the compilation option information, and the at least one evaluation data set to the at least one operating processor, receives at least one first performance parameter from the at least one operating processor, and transmits the received at least one first performance parameter to the user device via a network.

[0015] According to another feature of the present disclosure, at least one of data encryption, differential privacy, and data masking may be applied to the at least one evaluation data set.

[0016] According to another feature of the present disclosure, the compile option information may include at least one of a quantization algorithm, a pruning algorithm, a retraining algorithm, a model compression algorithm, an AI-based model optimization algorithm, and a knowledge distillation algorithm.

[0017] According to another feature of the present disclosure, the first neural processor includes at least one internal memory and at least one multiplication and accumulation unit, and at least one operating processor can set the compilation option based on the first configuration.

[0018] According to another feature of the present disclosure, at least one operating processor can determine a layer among a plurality of layers of the artificial neural network model that can be processed by a neural processor selected by the first processor selection information.

[0019] According to another feature of the present disclosure, at least one operating processor may report a layer that cannot be processed by a neural processor selected by the first processor selection information among a plurality of layers of the artificial neural network model.

[0020] According to another feature of the present disclosure, an artificial neural network model performance evaluation system may further include a graphics processor that processes a layer that cannot be processed by a neural processor selected by the first processor selection information among a plurality of layers of the artificial neural network model.

[0021] According to another feature of the present disclosure, the graphics processor can perform retraining of an artificial neural network model.

[0022] According to another feature of the present disclosure, at least one first performance parameter may include at least one of a temperature profile, power consumption, Trillion Operations Per Second per Watt (TOPS / W), Frames Per Second (FPS), Inference Per Second (IPS), and inference and prediction accuracy.

[0023] According to another feature of the present disclosure, the at least one operating processor further receives second processor selection information for selecting at least one other of a first neural processor and a second neural processor, and by compiling the artificial neural network model, processes the artificial neural network model used in the neural processor selected by the second processor selection information, applies the at least one evaluation data set to the artificial neural network model by the neural processor selected by the second processor selection information, and generates at least one second performance parameter related to the performance of the neural processor selected by the second processor selection information.

[0024] According to another feature of the present disclosure, the at least one operating processor may recommend either the first processor selection information or the second processor selection information by comparing at least one first performance parameter and at least one second performance parameter.

[0025] According to another feature of the present disclosure, the compilation option may include at least one of a Post Training Quantization (PTQ) algorithm, a layer-wise retraining algorithm for an artificial neural network model, and a quantization-aware training (QAT) algorithm.

[0026] The present disclosure relates to a method for evaluating the performance of an artificial neural network model, comprising the steps of: receiving, by at least one operating processor, first processor selection information for selecting at least one of an artificial neural network model, a first neural processor, and a second neural processor, and compilation option information; by the at least one operating processor compiling the artificial neural network model according to the compilation option, processing the artificial neural network model used in the neural processor selected by the first processor selection information; by the at least one operating processor applying at least one evaluation data set to the artificial neural network model by the neural processor selected by the first processor selection information; and by the at least one operating processor generating at least one first performance parameter related to the performance of the neural processor selected by the first processor selection information and transmitting it through a network.

[0027] According to another feature of the present disclosure, a method for evaluating the performance of an artificial neural network model may further include the steps of: a server receiving the first processor selection information, the compilation option information, and the at least one evaluation data set from a user device via a network, and sending the first processor selection information, the compilation option information, and the at least one evaluation data set to the at least one operating processor; and the server receiving at least one first performance parameter from the at least one operating processor and transmitting the received at least one first performance parameter to the user device via a network.

[0028] According to another feature of the present disclosure, the method for evaluating the performance of an artificial neural network model may further include the step of setting the compilation option based on the hardware configuration of at least one of a first neural processor and a second neural processor.

[0029] According to another feature of the present disclosure, the method for evaluating the performance of an artificial neural network model may further include the step of reporting a layer that cannot be processed by a neural processor selected by the first processor selection information among a plurality of layers of the artificial neural network model.

[0030] According to another feature of the present disclosure, the method for evaluating the performance of an artificial neural network model may further include the step of a graphics processor processing a layer that cannot be processed by a neural processor selected by the first processor selection information among a plurality of layers of the artificial neural network model.

[0031] According to another feature of the present disclosure, the method for evaluating the performance of an artificial neural network model may further include the step of a graphics processor performing retraining of the artificial neural network model.

[0032] According to another feature of the present disclosure, a method for evaluating the performance of an artificial neural network model may further include: receiving second processor selection information in which at least one operating processor selects at least one other of a first neural processor and a second neural processor; processing the artificial neural network model used in the neural processor selected by the second processor selection information by compiling the artificial neural network model according to the compilation option; applying at least one evaluation data set to the artificial neural network model by the neural processor selected by the second processor selection information by the at least one operating processor; and generating at least one second performance parameter related to the performance of the neural processor selected by the second processor selection information and transmitting it over a network.

[0033] According to another feature of the present disclosure, the method for evaluating the performance of an artificial neural network model may further include the step of comparing at least one first performance parameter with at least one second performance parameter to recommend either the first processor selection information or the second processor selection information.

[0034] The present disclosure relates to a method for evaluating the performance of an artificial neural network model, comprising: a step of displaying an option for selecting a plurality of neural processors including a first neural processor of a first configuration and a second neural processor of a second configuration different from the first configuration; a step of receiving first processor selection information from a user for selecting at least one of the plurality of neural processors; a step of displaying a plurality of compilation options associated with the compilation of an artificial neural network model; a step of receiving first compilation option selection information among the plurality of compilation options from a user; a step of sending the first processor selection information, the first compilation option selection information, and at least one evaluation data set to a computing device connected to the plurality of neural processors; a step of receiving at least one first performance parameter associated with the processing performance of the neural processor selected by the first processor selection information according to the compilation option selected by the first compilation option selection information; and a step of displaying at least one first performance parameter.

[0035] According to another feature of the present disclosure, a method for evaluating the performance of an artificial neural network model may further include: receiving second processor selection information from a user for selecting at least one other of the plurality of neural processors; receiving second compilation option selection information from a user for selecting a plurality of compilation options; sending the second processor selection information and the second compilation option selection information to a computing device connected to the plurality of neural processors; and receiving at least one second performance parameter related to the processing performance of the neural processor selected by the second processor selection information according to the compilation option selected by the second compilation option selection information.

[0036] According to another feature of the present disclosure, the method for evaluating the performance of an artificial neural network model may further include the step of recommending either of the first processor selection information and the second processor selection information and displaying the same.

[0037] Specific details of other embodiments are included in the detailed description and drawings. Effects of the invention

[0038] According to one example of the present disclosure, a user can quickly determine in advance whether their artificial neural network model can operate on a specific NPU before purchasing the NPU.

[0039] According to one example of the present disclosure, a user can quickly know in advance, before purchasing a specific NPU, what performance their artificial neural network model will have when run on a specific NPU.

[0040] According to one example of the present disclosure, when each NPU is connected via a server by type, the user can evaluate the user's artificial neural network model online, evaluate whether to run each NPU online, and receive the results.

[0041] According to another example of the present disclosure, since the user's artificial neural network data can be protected in various ways, security issues that may arise during the performance evaluation of the NPU can be resolved.

[0042] The effects according to the present disclosure are not limited to those exemplified above, and a wider variety of effects are included within the present disclosure. Brief explanation of the drawing

[0043] FIG. 1 is a block diagram showing the configuration of an artificial neural network model performance evaluation system according to one example of the present disclosure. FIG. 2 is a block diagram showing the configuration of an artificial neural network model processing device according to one example of the present disclosure. FIG. 3 is a block diagram showing the configuration of a compiler of an artificial neural network model processing device according to one example of the present disclosure. FIG. 4 is a block diagram showing the configuration of an optimization module of an artificial neural network model processing device according to one example of the present disclosure. FIG. 5a is a block diagram showing a plurality of neural processing units and an interface for selecting compilation options of an artificial neural network model processing device according to one example of the present disclosure. FIG. 5b is a block diagram showing an interface for performance evaluation and proposal for a plurality of neural processing units of an artificial neural network model processing device according to one example of the present disclosure. FIGS. 6a to 6d are block diagrams showing the configuration of one neural processing unit of an artificial neural network model processing device according to one example of the present disclosure. FIG. 7 is a block diagram showing the configuration of a plurality of neural processing devices according to one example of the present disclosure. FIG. 8 is a flowchart illustrating a method for evaluating the performance of an artificial neural network model according to an example of the present disclosure. FIG. 9 is a flowchart illustrating a method for evaluating the performance of an artificial neural network model according to another example of the present disclosure. FIG. 10 is a flowchart illustrating a method for evaluating the performance of an artificial neural network model according to another example of the present disclosure. Specific details for implementing the invention

[0044] The advantages and features of the present disclosure and the methods for achieving them will become clear by referring to the examples described below in detail together with the accompanying drawings. However, the present disclosure is not limited to the examples disclosed below but may be implemented in various different forms; these examples are provided merely to make the present disclosure complete and to fully inform those skilled in the art of the scope of the invention, and the present disclosure is defined only by the scope of the claims. In connection with the description of the drawings, similar reference numerals may be used for similar components.

[0045] In the present disclosure, expressions such as “have,” “may have,” “include,” or “may include” indicate the presence of such features (e.g., numerical values, functions, actions, or components such as parts) and do not exclude the presence of additional features.

[0046] In the present disclosure, expressions such as “A or B,” “at least one of A or / and B,” or “one or more of A or / and B” may include all possible combinations of items listed together. For example, “A or B,” “at least one of A and B,” or “at least one of A or B” may refer to cases including (1) at least one A, (2) at least one B, or (3) both at least one A and at least one B.

[0047] Expressions such as "first," "second," "first," or "second" used in this disclosure may modify various components regardless of order and / or importance, and are used merely to distinguish one component from another and do not limit said components. For example, a first user device and a second user device may represent different user devices regardless of order or importance. For example, without departing from the scope of rights described in this disclosure, a first component may be named a second component, and similarly, a second component may be renamed a first component.

[0048] Where it is stated that a certain component (e.g., a first component) is "(operatively or communicatively) coupled with" or "connected to" another component (e.g., a second component), it should be understood that the said certain component may be directly connected to the said other component or connected through another component (e.g., a third component). On the other hand, where it is stated that a certain component (e.g., a first component) is "directly connected" or "directly connected" to another component (e.g., a second component), it may be understood that no other component (e.g., a third component) exists between the said certain component and the said other component.

[0049] The expression “configured to” as used in this disclosure may be replaced, depending on the context, with, for example, “suitable for,” “having the capacity to,” “designed to,” “adapted to,” “made to,” or “capable of.” The term “configured to” may not necessarily mean only that which is “specifically designed to” in hardware. Instead, in some situations, the expression “device configured to” may mean that the device is “capable of” in conjunction with other devices or components. For example, the phrase “processor configured to perform A, B, and C” may mean a dedicated processor for performing the said operations (e.g., an embedded processor), or a generic-purpose processor (e.g., a CPU or an application processor) capable of performing said operations by executing one or more software programs stored in a memory device.

[0050] The terms used in this disclosure are used merely to describe specific examples and are not intended to limit the scope of other examples. A singular expression may include a plural expression unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as generally understood by those skilled in the art described in this disclosure. Terms used in this disclosure that are defined in a general dictionary may be interpreted as having the same or similar meaning as they have in the context of the relevant technology, and are not to be interpreted in an ideal or overly formal sense unless explicitly defined in this disclosure. In some cases, even terms defined in this disclosure are not to be interpreted to exclude the examples of this disclosure.

[0051] The features of each of the various examples of the present disclosure may be combined or combined with one another, either partially or wholly, and as will be fully understood by those skilled in the art, various technical interlocking and operation are possible, and each example may be implemented independently of one another or together in an interlocking relationship.

[0052] Hereinafter, the present disclosure will be described in detail by describing preferred examples of the present disclosure with reference to the attached drawings.

[0053] FIG. 1 is a block diagram showing the configuration of an artificial neural network model performance evaluation system according to one example of the present disclosure.

[0054] Referring to FIG. 1, an artificial neural network model performance evaluation system (1000) according to one example of the present disclosure may include a user device (100), an artificial neural network model processing device (200), and a server (300).

[0055] An artificial neural network model performance evaluation system (1000) according to one example of the present disclosure of FIG. 1 may be configured to process a specific artificial neural network model in an artificial neural network model processing device (200) and provide the performance evaluation result of the artificial neural network model processing device (200) to a user online.

[0056] The user device (100) may be a device possessed by a user who wishes to obtain performance evaluation result information of an artificial neural network model processing device (200) that processes an artificial neural network model. The user device (100) may include a smartphone, tablet PC, PC, laptop, etc., which can connect to a server (300) and provide a user interface for checking information related to the artificial neural network model.

[0057] Methods by which a user device (100) connects to a server (300) may include connecting via a web service, connecting via an FTP server, connecting via a cloud server, or connecting via an application within the user device (100). However, the method by which the user device (100) connects to the server (300) is not limited to these and may utilize various known communication technologies.

[0058] The user can transmit information about an artificial neural network model to a server (300) using various communication technologies. Specifically, the user uploads at least one specific artificial neural network model and at least one specific evaluation dataset of said artificial neural network model to the server (300) via a user device (100) for evaluating the performance of a neural processing device that the user intends to purchase.

[0059] The aforementioned specific evaluation dataset refers to a dataset input to the artificial neural network model processing device (200) for evaluating the performance of the artificial neural network model processing device (200).

[0060] The user device (100) can receive the performance evaluation result of the artificial neural network model processing device (200) for the artificial neural network model from the artificial neural network model processing device (200) and output it.

[0061] For example, the user device (100) may be any type of terminal capable of uploading information about an artificial neural network model to be evaluated to the server (300) in the artificial neural network model performance evaluation system (1000).

[0062] For example, the user device (100) may be any type of terminal capable of uploading an evaluation dataset for evaluating the artificial neural network model to the artificial neural network model performance evaluation system (1000).

[0063] For example, the user device (100) may be any type of terminal capable of uploading a training dataset to the artificial neural network model performance evaluation system (1000) to retrain the artificial neural network model.

[0064] That is, the user device (100) may be referred to as a data transmission unit for evaluating the performance of an artificial neural network model or a unit for receiving the results of the performance evaluation of an artificial neural network model.

[0065] To this end, the user device (100) may include a processor (112), a display device (114), a user interface (116), a network interface (118), and memory (120). The display device (114) may display options for selecting one or more NPUs. Additionally, the display device (114) may also display options for compiling an artificial neural network model. The memory (120) may store executable software modules to allow the processor (112) to access the server (300), and may also store an artificial neural network model and performance evaluation data set to be transmitted to the artificial neural network model processing unit (200) via the server (300). The user interface (116) includes a keyboard and a mouse and may provide user input related to selecting one or more neural processing units to process the artificial neural network model and selecting compilation options related to compiling the artificial neural network model. The network interface (318) is a hardware component (e.g., a network interface card) that enables the user device (100) to communicate with the server (300) through a network.

[0066] The artificial neural network model processing device (200) includes a neural processing device for processing an artificial neural network model received from a user device (100) via a server (300). The artificial neural network model processing device (200) can also compile and evaluate the artificial neural network model. It determines the performance of the processed artificial neural network model and reports the performance results to the user device (100) via the server (300).

[0067] The artificial neural network model processing unit (200) may include a system composed of a general-purpose computer, laptop, cloud computer, cloud server, etc., which executes various programs to determine information about the neural processing unit. The artificial neural network model processing unit (200) may obtain at least one specific artificial neural network model for evaluating the performance of the neural processing unit and at least one specific evaluation dataset input to the artificial neural network model from the server (300), compile and process the artificial neural network model, and then provide a performance evaluation result.

[0068] A server (300) is a computing device that communicates with a user device (100) to manage access to an artificial neural network model processing unit (200). The server (300) may include a processor (312), a network interface (316), and memory (318). The network interface (316) enables the server (300) to communicate with the user device (100) and the artificial neural network model processing unit (200) over a network. The memory (318) stores instructions executable by the processor (312) to perform one or more of the following tasks: (i) managing accounts for users, and (ii) authenticating and allowing users to access the artificial neural network model processing unit (100) to evaluate one or more neural processing units. (iii) receive an artificial neural network model, an evaluation data set, a user selection for a neural processing unit to be evaluated, and a user selection for a compilation selection; (iv) encrypt and store the data received from the user; (v) transmit the artificial neural network model and the user's processor selection information to an artificial neural network model processing device (100) via a network; and (vi) transmit a performance report for the selected neural processing unit and a recommendation for the neural processing unit to a user device (100) via a network. The server (300) may also perform various other services.

[0069] To elaborate, the artificial neural network model, training data set, evaluation data set, etc. developed by the user constitute the user's intellectual property, so thorough security is required. Hereinafter, the artificial neural network model, training data set, evaluation data set, etc. developed by the user may be referred to as user data. Accordingly, in order to secure user data uploaded to the performance evaluation system (1000), the performance evaluation system (1000) may be configured to perform user account login, data encryption to protect the data itself, differential privacy, and data masking, as well as access control and audit logging for access control and auditing of the model.

[0070] Data encryption protects the confidentiality of data by encrypting user data. Differential Privacy uses statistical techniques to ensure that user data does not react sensitively to data containing personal information. Data masking protects user data by obscuring parts of it to hide sensitive information.

[0071] In addition, access control restricts the accounts that can access user data, audit logs record which accounts accessed the data, and audit logging maintains logs of system and user data access to track who accessed the model and when, thereby enabling the detection of abnormal activity.

[0072] Additionally, when uploading training data sets and / or evaluation data sets, an additional step of signing separate terms regarding user data protection may be performed. Thus, the user's artificial neural network model, training data sets, and / or evaluation data sets can be protected.

[0073] Below, with reference to FIG. 2, an artificial neural network model processing device (200) will be described.

[0074] FIG. 2 is a block diagram showing the configuration of an artificial neural network model processing device according to one example of the present disclosure.

[0075] Referring to FIG. 2, the artificial neural network model processing unit (200) may include a central processing unit (CPU, 214), an NPU farm (NPU FARM, 218) comprising a plurality of neural processing units (Neural Processing Unit Farm; NPU, 220), a graphic processing unit (GPU, 230), and memory (MEMORY, 250), and each component may communicate with one or more communication buses or signal lines. The artificial neural network model processing unit (200) may be operated by a specific operating system (OS). For example, the OS may be Microsoft Windows, MacOS, Linux (e.g., Ubuntu, Fedora, Debian, CentOS, Arch Linux), Unix, iOS, Android, etc.

[0076] The CPU (214) may include one or more operating processors for executing instructions stored in memory (250). Memory (250) may store various software modules including, but not limited to, a compiler (210), a storage module (240), and a reporting program (260).

[0077] Meanwhile, the memory (250) may include a volatile or non-volatile recording medium capable of storing various data, commands, and information.

[0078] For example, the memory (250) may include at least one type of storage medium among flash memory type, hard disk type, multimedia card micro type, card type memory (e.g., SD or XD memory, etc.), RAM, SRAM, ROM, EEPROM, PROM, network storage, cloud, and blockchain database.

[0079] A compiler (210) can convert a specific artificial neural network model into machine code that can be executed by a plurality of neural processing units (220). That is, the compiler (210) can generate machine code that can be executed by a plurality of neural processing units (220) having different characteristics. Accordingly, the compiler (210) can generate machine code to be executed by a selected neural processing unit (220) among the plurality of neural processing units (220). The machine code may also be referred to as binary code.

[0080] The compiler (210) can generate machine code for an artificial neural network model to evaluate the performance of at least one neural processing unit (220) selected for performance evaluation.

[0081] The compiler (210) may be configured to provide various compilation options. Compilation options may be provided as a UI on the screen of the user device (100) for the selection of various compilation options. The compiler (210) may generate machine code for an optimized artificial neural network model by setting multiple compilation options differently for each selected neural processing unit for performance evaluation.

[0082] Since multiple compilation options may vary depending on the type of multiple neural processing units (220), even for the same artificial neural network model, the compiled machine code may differ depending on the type of multiple neural processing units (220). That is, each machine code can be generated for each selected compilation option.

[0083] The storage module (240) can store various data used in the artificial neural network model processing unit (200). That is, the storage module (240) can store an artificial neural network model compiled in the form of machine code, one or more training data sets, one or more evaluation data sets, performance evaluation results, and output data from a plurality of neural processing units (220).

[0084] The reporting program (260) can process the compiled artificial neural network model and report the performance evaluation results.

[0085] That is, the reporting program (260) first determines whether the compiled artificial neural network model can be processed by a plurality of neural processing devices (220).

[0086] If the compiled artificial neural network model cannot be processed by a plurality of neural processing devices (220), the reporting program (260) may report a specific layer or a specific operation that cannot be processed by a plurality of neural processing devices (220) among the plurality of layers of the artificial neural network model.

[0087] If a compiled artificial neural network model can be executed by a specific neural processing unit among a plurality of neural processing units (220), the reporting program (260) can report the processing performance of the plurality of neural processing units (220).

[0088] Indicators of processing performance may include the temperature profile of the neural processing unit, power consumption (Watt), Trillion Operations Per Second per Watt (TOPS / W), Frames Per Second (FPS), Inferences Per Second (IPS), and Accuracy.

[0089] A temperature profile refers to data on the temperature changes of a neural processing unit measured over time while the neural processing unit is operating.

[0090] Power consumption refers to the power data measured when the neural processing unit is operating. Since power consumption varies depending on the computational characteristics of the artificial neural network model developed by the user, the user's artificial neural network model must be provided for accurate power measurement.

[0091] Trillion Operations Per Second per Watt (TOPS / W) is an indicator for measuring the efficiency of AI hardware, representing the number of operations that can be performed per second per watt.

[0092] TOPS / W indicates how much computation can be performed per unit of power consumed by the hardware, and is an indicator of the energy efficiency of multiple neural processing units (220).

[0093] Inference Per Second (IPS) is an indicator representing the number of inference operations that multiple neural processing units (220) can perform in one second, and signifies the processing speed of operations of multiple neural processing units (220). IPS may also be referred to as Frame Per Second (FPS).

[0094] Accuracy is an indicator representing the proportion of correctly predicted samples out of the total, and signifies the inference accuracy of multiple neural processing units (220). To elaborate, the accuracy of multiple neural processing units (220) and the inference accuracy of the graphics processing unit (230) may differ from each other. This is because the parameters of the artificial neural network model inferred by the graphics processing unit (230) may be floating-point, while the parameters of the artificial neural network model inferred by multiple neural processing units (220) may be integers. Additionally, various optimization algorithms may be selectively applied. Therefore, the parameters of the artificial neural network model inferred by multiple neural processing units (220) may have differences in values ​​calculated from various operations, and thus may have different inference accuracy from the artificial neural network model inferred by the graphics processing unit (230). Differences in inference accuracy can vary depending on the structure and parameter size characteristics of the artificial neural network model, and in particular, as the bitwidth of the quantized parameters becomes shorter, the degradation of inference accuracy due to excessive quantization tends to increase. For example, the quantized bitwidth can be 2 bits to 16 bits. There is also a tendency for the degradation of inference accuracy due to excessive pruning to increase.

[0095] Meanwhile, the reporting program (260) can analyze the processing performance of an artificial neural network model compiled according to each of the multiple compilation options and suggest one of the multiple compilation options in reverse.

[0096] Additionally, the reporting program (260) may also recommend a specific type of neural processing device (220) based on performance parameters of different neural processing devices (220).

[0097] Multiple neural processing units (220) may be in the form of an NPU farm (218) consisting of various NPU product families with different performance and price ranges sold by a specific company. The NPU farm (218) may be provided to perform performance evaluations of artificial neural network models developed by users online. The NPU farm may be provided in the form of a cloud NPU.

[0098] A plurality of neural processing units (220) can receive an evaluation dataset and input it into a compiled artificial neural network model to perform performance evaluation.

[0099] A plurality of neural processing devices (220) may include various types of neural processing devices.

[0100] More specifically, multiple neural processing units (220) can be classified based on computational capacity.

[0101] For example, the first neural processing unit may be a neural processing unit for smart CCTV. The first neural processing unit may have ultra-low power consumption, low level of inference processing performance (e.g., processing performance of 5 TOPS), a very small semiconductor package size, and very low cost characteristics. Due to performance limitations, the first neural processing unit may not support specific artificial neural network models that include specific operations and require high memory bandwidth. For example, the first neural processing unit may have a model name of “”, and the “” neural processing unit may compute artificial neural network models such as ResNet, Mobilenet v1 / v2, SSD, YOLOv5, YOLOv7, etc.

[0102] For example, the second neural processing unit may be a neural processing unit for image recognition, object detection, and object tracking of a robot. The second neural processing unit may have low power consumption, medium level inference processing performance (e.g., processing performance of 16 TOPS), a small semiconductor package size, and low cost characteristics. The second neural processing unit may not support certain artificial neural network models that require high memory bandwidth. For example, the second neural processing unit may have a model name of “”, and the “” neural processing unit may compute artificial neural network models such as ResNet, Mobilenet v1 / v2, SSD, YOLOv5, YOLOv7, etc.

[0103] For example, the third neural processing unit may be a neural processing unit for image recognition, object detection, object tracking, and generative AI services of an autonomous vehicle. The third neural processing unit may have low power consumption, high-level inference processing performance (e.g., processing performance of 25 TOPS), a medium semiconductor package size, and medium price characteristics. For example, the third neural processing unit may have a model name of “”, and the “” neural processing unit may compute artificial neural network models such as ResNet, MobileNet v1 / v2 / v3, SSD, EfficientNet, EfficientDet, YOLOv5, YOLOv7, YOLOv8, DeepLabv3, PIDNet, ViT, Generative adversarial network, and Stable diffusion.

[0104] For example, the fourth neural processing unit may be a neural processing unit for a CCTV control room, a control center, a large language model, or a generative AI service. The fourth neural processing unit may have low power consumption, high-level inference processing performance (e.g., processing performance of 400 TOPS), a large semiconductor package size, and high cost characteristics. For example, the fourth neural processing unit may have a model name of “”, and the “” neural processing unit may compute artificial neural network models such as ResNet, Mobilenet v1 / v2, SSD, YOLOv5, YOLOv7, YOLOv8, DeepLabv3, PIDNet, ViT, Generative adversarial network, Stable diffusion, and large LLM.

[0105] In other words, each neural processing unit can have different computational processing capabilities, different semiconductor chip sizes, different power consumption characteristics, etc.

[0106] However, the types of multiple neural processing devices (220) are not limited to this and can be classified according to various classification criteria.

[0107] A graphics processing unit (GPU; 230) is hardware that performs complex computational tasks as well as graphics and image processing. Although the graphics processing unit (230) is depicted as a single unit in the drawing, it is not limited thereto and may be multiple graphics processing units connected via a cloud GPU, NVLink, NVSwitch, etc.

[0108] The graphics processing unit (230) includes multiple cores and can process multiple tasks in parallel through the multiple cores. Accordingly, the graphics processing unit (230) can perform large-scale data processing tasks such as scientific computing and deep learning.

[0109] Specifically, the graphics processing unit (230) can be used to train deep learning and machine learning models on a large dataset. Deep learning models have a large number of parameters, so training takes a long time, but the graphics processing unit (230) can improve the training speed by processing these operations in parallel. When a user selects a specific neural processing unit from multiple neural processing units (220) and performs retraining of an artificial neural network model through various compilation options, the user selects an available graphics processing unit from the graphics processing unit (230), and the selected graphics processing unit can perform retraining of the artificial neural network model according to each compilation option.

[0110] Multiple neural processing units (220) and graphics processing units (230) can be implemented in the form of an integrated chip (IC), such as a System on Chip (SoC) with various computing units integrated, or a printed circuit board on which the integrated chip is mounted.

[0111] FIG. 3 is a block diagram showing the configuration of a compiler of an artificial neural network model processing device according to one example of the present disclosure.

[0112] Referring to FIG. 3, the compiler (210) of the artificial neural network model processing device can compile the artificial neural network model into machine code based on a plurality of compilation options.

[0113] Compiler (210) of an artificial neural network model processing device

[0114] It may include an optimization module (Optimizer, 211), a verification module (Verifier, 212), and a code generation module (Code Generator, 213).

[0115] The compiler (210) may receive structural data of a selected neural processing unit among a plurality of neural processing units (220). The structural data of the neural processing unit may include the memory size of the NPU internal memory, the hierarchical structure of the NPU internal memory, information on the number of processing elements, information on special operation circuit units, etc. Based on the structural data of the neural processing unit and the graph information of the artificial neural network model to be compiled, the compiler (210) may determine the processing order for each layer.

[0116] The optimization module (211) can perform the task of optimizing an artificial neural network model represented as a directed acyclic graph (DAG) to fit a selected neural processing unit among a plurality of neural processing units (220). The user can select at least one of the various optimization options provided by the optimization module (211) online through the user device (100).

[0117] For example, the optimization module (211) may provide an option to convert to integer parameters of a specific bit width. The specific bit width may be one bit width between 2 bits and 16 bits. Accordingly, the optimization module (2110) can convert an artificial neural network model based on floating-point parameters into an artificial neural network model based on integer parameters. In this case, a plurality of neural processing units (220) are designed to process integer parameters. For example, the optimization module (211) can convert an artificial neural network model based on non-linear trigonometric function operations into an artificial neural network model based on piecewise linear function approximation. In this case, a plurality of neural processing units (220) may be designed to process piecewise linear function approximation operations. For example, the optimization module (211) can reduce the size of parameters such as weights and feature maps of the artificial neural network model by applying various optimization algorithms. For example, the optimization module (211) can improve the problem of accuracy degradation of the optimized artificial neural network model through various retraining algorithms.

[0118] The verification module (212) can perform a verification function to determine whether a customer's artificial neural network model is executable on multiple neural processing units (220). The verification module (212) analyzes the structure of the optimized artificial neural network model and determines whether the artificial neural network model is executable by examining whether the operators for each layer are hardware-supported on multiple neural processing units (220). If execution is not possible, a separate error report file can be generated and reported to the user.

[0119] The code generation module (213) can optimize the artificial neural network model determined to be operable by the verification module (212) through the optimization module (211), and generate machine code for each optimized artificial neural network model to be executed on a selected neural processing unit among a plurality of neural processing units (220). The generated machine code is provided to the corresponding neural processing unit to enable performance evaluation of the plurality of neural processing units (220).

[0120] For example, a first machine code corresponding to a first artificial neural network model may be generated for a first neural processing unit among a plurality of neural processing units (220). A second machine code corresponding to a first artificial neural network model may be generated for a second neural processing unit among a plurality of neural processing units (220). A third machine code corresponding to a first artificial neural network model may be generated for a third neural processing unit among a plurality of neural processing units (220). A fourth machine code corresponding to a first artificial neural network model may be generated for a fourth neural processing unit among a plurality of neural processing units (220).

[0121] FIG. 4 is a block diagram showing the configuration of an optimization module of an artificial neural network model processing device according to one example of the present disclosure.

[0122] The optimization module (211) can optimize an artificial neural network model based on multiple compilation options.

[0123] More specifically, the optimization module (211) can set compilation options based on hardware information of the neural processing unit (220).

[0124] In addition, the optimization module (211) can set the plurality of compilation options by considering the characteristics of the parameters of the artificial neural network model (e.g., the size of the weights, the size of the feature maps, etc.) and the characteristics of the inference accuracy degradation.

[0125] A plurality of compilation options set by the optimization module (211) may be at least one of a quantization option, a pruning option, a retraining option, a model compression option, an AI-based model optimization option, and a knowledge distillation option.

[0126] A pruning option can provide a technique to reduce the computational load of an artificial neural network model. A pruning algorithm can be configured to replace small values ​​close to zero among the weights of all layers of an artificial neural network model with zero. Multiple neural processing units (220) can skip multiplication operations related to zero weights to improve the computational speed of convolution and reduce power consumption, and can reduce the parameter size of the machine code of an artificial neural network model to which the pruning option is applied. To elaborate, when a specific weight parameter is replaced with zero by the pruning option, the pruning algorithm can provide an effect substantially equivalent to disconnecting the network of the artificial neural network model containing the corresponding weight data. For example, a size-based first pruning option that removes small weights and a percentage-based second pruning option that removes a specific ratio of the smallest weights may be provided.

[0127] Quantization options can provide a technique to reduce the size of parameters in an artificial neural network model. Quantization algorithms can be configured to selectively reduce the number of bits of weights and feature maps in each layer of the artificial neural network model. When the number of bits of specific feature maps and specific weights is reduced by the quantization option, the size of the parameters in the machine code of the artificial neural network model to which the quantization option is applied can be reduced. For example, when the quantization option is applied, 32-bit parameters representing floating points can be converted into parameters with a bit width of 2 to 16 bits representing integers.

[0128] The model compression option can provide a technique for compressing weight parameters or feature map parameters of an artificial neural network model. The model compression technique can be implemented by utilizing conventional known compression techniques. Therefore, the parameter size of the machine code of an artificial neural network model to which the model compression option is applied can be reduced. The model compression option can be optionally provided to a neural processing unit including a decompression decoder among a plurality of neural processing units (220).

[0129] The knowledge distillation option can provide a technique for transferring knowledge obtained from a complex model (teacher model) to a smaller and simpler model (student model). In the knowledge distillation algorithm, the teacher model typically has a larger parameter size and higher accuracy compared to the student model. For example, the accuracy of the student model can be improved by using a knowledge distillation option in which an artificial neural network model trained with floating-point 32-bit parameters is set as the teacher model and an artificial neural network model with various optimization options applied is set as the student training model. Here, the student model may be a model in which at least one of the pruning option, quantization option, model compression option, and retraining option is selected.

[0130] The retraining option is a technique that can compensate for the reduced inference accuracy when various optimization options are applied. For example, when quantization options, pruning options, and model compression options are applied, the accuracy of the artificial neural network model inferred by multiple neural processing units (220) may be reduced. In such cases, an option may be provided to retrain the pruned, quantized, and / or model-compressed artificial neural network model online. The inference accuracy of the artificial neural network model after retraining can be increased again.

[0131] Specifically, retraining options may include quantization-aware retraining options, pruning-aware retraining options, and transfer learning options.

[0132] The Quantization-Aware Retraining (QAT) option integrates quantization into the training phase of an artificial neural network model, allowing the model to fine-tune weights by reflecting quantization errors. Quantization-aware retraining algorithms may include modifications to the loss function, gradient calculation, and optimization algorithm. The QAT option can compensate for quantization errors by quantizing a trained artificial neural network model and then performing fine-tuning to retrain it in a direction that minimizes the loss associated with quantization.

[0133] Pruning-aware retraining (PAT) is an option that identifies and removes less important weights from a trained artificial neural network model and then fine-tunes the remaining weights. Pruning criteria can utilize weight size, activation values, sensitivity analysis, etc. Pruning-aware retraining options can compensate for issues such as reducing the size of the artificial neural network model, improving inference speed, and mitigating the problem of overfitting during retraining.

[0134] Transfer learning refers to an option where an artificial neural network model learns by transferring knowledge acquired from one task to another related task. Transfer learning algorithms are effective when there is insufficient data initially or when training the artificial neural network model from scratch requires significant computational resources.

[0135] The optimization module (211) may provide an artificial intelligence-based optimization option, but is not limited thereto. The artificial intelligence-based optimization algorithm is a method of generating an optimally lightweight artificial neural network model by going through a process of exploring the structure of an artificial neural network model using an artificial intelligence reinforcement learning method, or a method of obtaining an optimal lightweight result by having the artificial intelligence within the optimization module (211) independently carry out the lightweighting process, rather than relying on a lightweighting method such as a quantization algorithm, a pruning algorithm, a retraining algorithm, a model compression algorithm, and a model compression algorithm.

[0136] FIG. 5a is a block diagram showing a plurality of neural processing units and an interface for selecting compilation options of an artificial neural network model processing device according to one example of the present disclosure.

[0137] The user interface can be displayed on the display device (114) of the user device (100) after the user accesses the server (300) using the user device (100).

[0138] The display device (114) displays two sections: an NPU selection section (510) and a compilation option section (520). The user can select one or more NPUs in the NPU selection section (510) to run a simulation for an artificial neural network model using one or more evaluation data sets. The NPU selection section (510) displays four types of NPUs: DX-M1, DX-H1, DX-V1, and DX-V2, and the user can select an NPU to use for online simulation for performance evaluation. FIG. 5A shows an example where one DX-M1 is selected for testing and evaluation.

[0139] The compilation option section (520) displays preset options to facilitate the user's compilation selection. Referring to FIG. 5a, the compilation option section (520) provides a first preset option, a second preset option, and a third preset option. However, the present disclosure is not limited to the preset options, and a wider variety of preset options may be provided.

[0140] For the convenience of explanation below, presets of a quantization algorithm are described as examples. Each preset option may be the most effective quantization preset option from a specific perspective, but the present disclosure is not limited thereto. The user may select at least one preset option by considering the characteristics of each preset option.

[0141] For example, the first preset option is an option that performs only a quantization algorithm to convert 32-bit data representing the floating point of a trained artificial neural network model into 8-bit data representing an integer. However, the present disclosure is not limited to 8 bits and can be set to have a bit width between 2 bits and 16 bits, and the bit width may be limited to a specific bit width depending on the hardware configuration of the selected neural processing device. That is, since the first preset option is a quantization algorithm executed after the training of the artificial neural network model, it can be referred to as Post Training Quantization (PTQ). The first preset option has the advantage of being able to perform quantization quickly because quantization is typically completed within a few minutes. Therefore, there is an advantage in being able to quickly check results regarding the power consumption, computation processing speed, etc. of the artificial neural network model provided by the user in the neural processing device selected by the user among the plurality of neural processing devices (220).

[0142] For example, a first preset option consisting of a first quantization option may be provided to the user as an option called DXNN Lite. Here, since the first quantization option does not require retraining, the retraining step of the artificial neural network model may not be performed.

[0143] For example, the second preset option is an option to perform a layer-wise retraining algorithm of the artificial neural network model after performing a quantization algorithm that converts 32-bit data representing the floating point of the artificial neural network model into 8-bit data representing the integer. However, the present disclosure is not limited to 8 bits and may be set to have a bit width between 2 bits and 16 bits, and the bit width may be limited to a specific bit width depending on the hardware configuration of the selected neural processing device.

[0144] That is, the second preset option may be an option configured to further perform a layer-by-layer retraining algorithm using the artificial neural network model that performed the first preset option as the input model. Accordingly, the second preset option may be an option that combines one of the various retraining options provided by the quantization algorithm and the optimization module (211).

[0145] That is, in the second preset option, data corresponding to some of the layers of the artificial neural network model is quantized, and the quantization loss function is calculated. Subsequently, data corresponding to other of the layers of the artificial neural network model is quantized, and the quantization loss function is calculated. By repeating the above-described operation, the quantization algorithm is optimized by performing quantization only on some of the layers of the artificial neural network model where the quantization loss function is minimized. In the case of the second preset option, since retraining can be performed in a direction that minimizes the difference between decimal data (e.g., floating point 32) and integer data (e.g., integer 8) of the feature map for each layer, there is an advantage that retraining is possible even without a training dataset. Since the second preset option typically completes quantization within a few hours, there is an advantage that quantization can be performed within an appropriate time. Therefore, there is an advantage that the accuracy of the artificial neural network model provided by the user can be improved relatively more than the first preset option in the neural processing device selected by the user among the multiple neural processing devices (220).

[0146] For example, a second preset option consisting of a second quantization option may be provided to the user under the service name DXNN pro. Here, since the second quantization option performs a layer-by-layer retraining algorithm of the artificial neural network model, a retraining step of the artificial neural network model may be required.

[0147] For example, the third preset option is an option to perform a quantization aware training (QAT) algorithm after performing a quantization algorithm that converts 32-bit data representing floating points of an artificial neural network model into 8-bit data representing integers. However, the present disclosure is not limited to 8 bits and may be set to have a bit width between 2 bits and 16 bits, and the bit width may be limited to a specific bit width depending on the hardware configuration of the selected neural processing device.

[0148] That is, the third preset option may be an option configured to further perform a quantization recognition retraining algorithm using the artificial neural network model that performed the first preset option as an input model. Therefore, the third preset option may be an option that combines the quantization algorithm with one of the various retraining options provided by the optimization module (211).

[0149] That is, in the third preset option, the quantization-aware retraining algorithm quantizes the learned artificial neural network model and then performs fine-tuning by retraining in a direction that minimizes the degradation of inference accuracy due to quantization. However, in order to retrain in a direction that minimizes the degradation of inference accuracy due to quantization, the user must provide a training data set of the artificial neural network model. Additionally, an evaluation data set may be required to prevent overfitting during retraining. Specifically, the quantization-aware retraining algorithm inputs the machine code of the quantized artificial neural network model and the training data set into a corresponding neural processing unit among a plurality of neural processing units (220) to retrain the model to compensate for the degradation of inference accuracy due to quantization error. The third preset option has the advantage of guaranteeing relatively higher inference accuracy than the first and second preset options, but since quantization can be completed within a few days, it can be performed when accuracy is important.

[0150] For example, a third preset option consisting of a third quantization option may be provided to the user under the service name DXNN master. In this case, since the third quantization option performs a retraining algorithm based on the inference accuracy of the artificial neural network model, a retraining step of the artificial neural network model may be required. For the quantization recognition retraining algorithm of the third quantization option, training datasets and / or evaluation datasets of the artificial neural network model are required from the user during the process of retraining in a direction that minimizes loss due to quantization. Here, the training dataset is essential data required for quantization recognition retraining. The evaluation dataset is data that can be optionally used to improve the problem of overfitting during retraining.

[0151] FIG. 5b is a block diagram showing an interface for performance evaluation and proposal for a plurality of neural processing units of an artificial neural network model processing device according to one example of the present disclosure.

[0152] Figure 5b shows the results of simulation and evaluation performed using two different types of NPUs. The top left box shows the results using the DX-M1 NPU, and the box above shows the results using the DX-H1 NPU. The bottom box may display a recommended NPU selection based on the performance parameters of the two different NPUs.

[0153] FIGS. 6a to 6d are block diagrams showing the configuration of one neural processing unit of an artificial neural network model processing device according to one example of the present disclosure.

[0154] Specifically, FIG. 6a illustrates the internal configuration of a first neural processing unit among a plurality of neural processing units. FIG. 6b illustrates the internal configuration of a second neural processing unit among a plurality of neural processing units. FIG. 6c illustrates the internal configuration of a third neural processing unit among a plurality of neural processing units. FIG. 6d illustrates the internal configuration of a third neural processing unit among a plurality of neural processing units.

[0155] The first neural processing unit (220) illustrated in FIG. 6a is configured to include a processing element array (221), an NPU internal memory (222), and an NPU controller (223).

[0156] For example, the first neural processing unit (220) may be configured to include a processing element array (221), an NPU internal memory (222) configured to store an artificial neural network model that can be inferred from the processing element array (221) or to store at least some data of the artificial neural network model, and an NPU controller (223) configured to control the processing element array (221) and the NPU internal memory (222). Here, the artificial neural network model may be machine code compiled with various optimization options applied.

[0157] The NPU controller (223) is configured to control the operation of the processing element array (221) for the inference operation of the first neural processing unit (220) and the read and write order of the NPU internal memory (222).

[0158] The NPU controller (223) can be configured to control the processing element array (221) and the NPU internal memory (222) by machine code. The NPU controller (223) is configured to control the processing element array (221) and the NPU internal memory (222) at each operation stage according to the operation scheduling defined in the machine code. Accordingly, the neural processing unit can sequentially process operations for each layer according to the structure of the artificial neural network model. Here, the NPU controller (223) can acquire a memory address where the feature map and weights of the artificial neural network model are stored, or determine a memory address to be stored.

[0159] A processing element array (221) refers to a configuration in which a plurality of processing elements (PE1 to PE12) are arranged in an array form. Each processing element may be configured to include a MAC (multiply and accumulate) operator and / or an ALU (Arithmetic Logic Unit) operator. However, examples according to the present disclosure are not limited thereto.

[0160] Although multiple processing elements are illustrated as examples in FIG. 6a, it is also possible to configure a processing element by replacing the MAC within a single processing element with a plurality of multipliers and an adder tree implemented in parallel. In this case, the processing element array (221) may be referred to as at least one processing element including a plurality of arithmetic units.

[0161] The processing element array (221) is configured to include a plurality of processing elements (PE1 to PE12). The plurality of processing elements (PE1 to PE12) shown in FIG. 6a are merely examples for convenience of explanation, and the number of the plurality of processing elements (PE1 to PE12) is not limited. The size or number of the processing element array (221) may be determined by the number of the plurality of processing elements (PE1 to PE12). The size of the processing element array (221) may be implemented in the form of an N x M matrix, where N and M are integers greater than 0. The processing element array (221) may include N x M processing elements. That is, there may be one or more processing elements.

[0162] The size of the processing element array (221) can be designed by taking into account the characteristics of the artificial neural network model. To elaborate, the number of processing elements can be determined by taking into account the data size of the artificial neural network model to be operated, the required operating speed, and the required power consumption. The data size of the artificial neural network model can be determined by corresponding to the number of layers of the artificial neural network model and the weight data size of each layer.

[0163] Accordingly, the size of the processing element array (221) of the first neural processing device (220) according to one example of the present disclosure is not limited. As the number of processing elements of the processing element array (221) increases, the parallel processing capability of the operating artificial neural network model increases, but the manufacturing cost and physical size may increase.

[0164] For example, as illustrated in FIG. 6b, the second neural processing unit (220-1) may include two processing element arrays (221-1, 221-2). Each of the two processing element arrays (221-1, 221-2) may be grouped to include a plurality of processing elements (PE1 to PE12).

[0165] As another example, as illustrated in FIG. 6c, the third neural processing unit (220-2) may include four processing element arrays (221-1, 221-2, 221-3, 221-4). Each of the four processing element arrays (221-1, 221-2, 221-3, 221-4) may be grouped to include a plurality of processing elements (PE1 to PE12).

[0166] As another example, as shown in FIG. 6d, the fourth neural processing unit (220-3) may include eight first neural processing units (220).

[0167] Each of the eight first neural processing units (220) divides and processes the operations of the artificial neural network model. Accordingly, the processing speed of the fourth neural processing unit (220-3) can be further improved.

[0168] Accordingly, the fourth neural processing unit (220-3) may further include a control unit that assigns the operation of each of the eight neural processing units.

[0169] The description of the characteristics and processing models of the first to fourth neural processing devices is as described above.

[0170] FIG. 7 is a block diagram showing the configuration of a plurality of neural processing devices according to one example of the present disclosure.

[0171] A plurality of neural processing devices (220) may include a plurality of types of neural processing devices. And, at least one neural processing device of the same type may be arranged.

[0172] For example, multiple neural processing units may be arranged to form a first group (G1), multiple neural processing units may be arranged to form a second group (G2), multiple neural processing units may be arranged to form a third group (G3), and multiple neural processing units may be arranged to form a fourth group (G4). A group of multiple types of neural processing units may be referred to as an NPU FARM. An NPU FARM may be a cloud-type NPU system configured to respond in real-time to performance evaluation requests from multiple users connected online.

[0173] And, the plurality of neural processing devices (220) included in the first to fourth groups (G1 to G4) may all be used for performance evaluation or only some may be used for performance evaluation, depending on the user's choice.

[0174] User data requiring security can be stored in the server (300) or in the storage module (240) of the artificial neural network model processing device (200).

[0175] At least one neural processing unit (220) used for computation can communicate with a server (300) to receive at least one specific artificial neural network model for evaluating the performance of the neural processing unit and at least one specific evaluation dataset to be input to the artificial neural network model. That is, the neural processing unit (220) can receive user data required for performance evaluation.

[0176] Below, methods for evaluating the performance of artificial neural network models using the artificial neural network model performance evaluation system described above are explained.

[0177] FIG. 8 is a flowchart illustrating a method for evaluating the performance of an artificial neural network model according to one example of the present disclosure.

[0178] Referring to FIG. 8, the artificial neural network model performance evaluation method (S100) includes a neural processing unit type selection step (S110); a compilation option selection step (S120); an artificial neural network model upload step (S130); an artificial neural network model compilation and processing step (S140); and an artificial neural network model processing result reporting step (S150).

[0179] In the neural processing unit type selection step (S110), the user selects the type of neural processing unit to be evaluated for performance. The type of neural processing unit may vary depending on the product family of neural processing units sold by a specific company.

[0180] As shown in FIG. 7, a plurality of neural processing devices may be arranged to form a first group (G1), a plurality of neural processing devices may be arranged to form a second group (G2), a plurality of neural processing devices may be arranged to form a third group (G3), and a plurality of neural processing devices may be arranged to form a fourth group (G4).

[0181] The user selects the neural processor to perform performance evaluation from among the “” neural processor, “” neural processor, “” neural processor, and “” neural processor.

[0182] Additionally, the server (300) can select at least one neural processing device available within the NPU FARM among the selected specific types of neural processing devices. The specific types of neural processing devices selected at this time are not limited to a single type but may be multiple types.

[0183] Subsequently, in the compilation option selection step (S120), at least one of the multiple compilation options for the artificial neural network model to be processed by the selected at least one neural processing unit is selected.

[0184] More specifically, in the compilation option selection step (S120), compilation options can be set based on hardware information of the neural processing unit (220).

[0185] In addition, during the compilation option selection step, multiple compilation options can be set by the user's selection. Here, explanations regarding the advantages and disadvantages of each compilation option can be displayed on the user device (100). Thus, the user can select various compilation options in a manner that is suitable for the user. That is, the performance evaluation system (1000) can provide compilation options in a user customization manner, rather than preset options, to meet the user's specific requirements.

[0186] The above-described compilation option may be at least one of a pruning algorithm, a quantization algorithm, a model compression algorithm, a knowledge distillation algorithm, a retraining algorithm, and an AI-based model optimization algorithm.

[0187] Alternatively, the compilation option can be configured to select one of the prepared preset options.

[0188] Subsequently, in the artificial neural network model upload step (S130), at least one specific artificial neural network model for evaluating the performance of the selected neural processing device is uploaded to the server (300). Here, the artificial neural network model upload step (S130) may be referred to as the user data upload step.

[0189] Subsequently, in the artificial neural network model compilation and processing step (S140), the input artificial neural network model is compiled according to the selected compilation option, and the compiled machine code is input into a selected neural processing unit within the NPU FARM for processing.

[0190] In the artificial neural network model processing result reporting step (S150), the performance evaluation results of the neural processing unit that processed the compiled artificial neural network model may be reported. The performance evaluation result report may be stored in the user's account or sent to the user's email address. However, it is not limited to these methods, and the performance evaluation results may be provided to the user in various ways. The performance evaluation results are also treated as user data, and security policies applied to user data may be applied.

[0191] To elaborate, in the artificial neural network model processing result reporting step (S150), it is first determined whether the compiled artificial neural network model can be processed by a plurality of neural processing devices (220).

[0192] If the compiled artificial neural network model cannot be processed by a plurality of neural processing devices (220), in the artificial neural network model processing result reporting step (S150), the layer that cannot be processed by a plurality of neural processing devices (220) among the plurality of layers of the artificial neural network model may be reported.

[0193] And, among the multiple layers of the artificial neural network model, the layer that cannot be processed by the multiple neural processing units (220) can be processed by the graphics processing unit (230).

[0194] If the compiled artificial neural network model can be processed by a plurality of neural processing devices (220), the processing performance of the plurality of neural processing devices (220) can be reported in the artificial neural network model processing result reporting step (S150).

[0195] Indicators of processing performance may include the temperature profile of the neural processing unit, power consumption (Watt), Trillion Operations Per Second per Watt (TOPS / W), Frames Per Second (FPS), Inferences Per Second (IPS), and Accuracy.

[0196] Here, if the user does not provide an evaluation data set, the artificial neural network model performance evaluation system (1000) can analyze the size of the input data of the artificial neural network model to generate corresponding dummy data and perform a performance evaluation using the generated dummy data. For example, the size of the dummy data may be (224 x 224 x 3), (288 x 288 x 3), (380 x 380 x 3), (515 x 512 x 3), (640 x 640 x 3), etc., but the present disclosure is not limited to the size of the dummy data. That is, even if a data set capable of evaluating inference performance is not provided, it may be possible to generate performance evaluation results such as power consumption, TOPS / W, FPS, IPS, etc. of the neural processing device. However, in such cases, the inference accuracy evaluation results may not be provided.

[0197] According to one example of the present disclosure, a user can quickly determine in advance whether their artificial neural network model can operate on a specific NPU before purchasing the NPU.

[0198] According to one example of the present disclosure, a user can quickly know in advance, before purchasing a specific NPU, what performance their artificial neural network model will have when run on a specific NPU.

[0199] According to one example of the present disclosure, when each NPU is connected via a server by type, the user can evaluate the user's artificial neural network model online, evaluate whether it runs online for each available NPU, and receive the results.

[0200] Accordingly, the performance evaluation system (1000) can help the user quickly decide on a purchase by providing the user with information on the performance and price of the neural processing device required for the implementation of the AI ​​service developed by the user.

[0201] FIG. 9 is a flowchart illustrating a method for evaluating the performance of an artificial neural network model according to another example of the present disclosure.

[0202] Referring to FIG. 9, the artificial neural network model performance evaluation method (S200) includes a neural processing unit type selection step (S110); a compilation option selection step (S120); an artificial neural network model and dataset upload step (S230); an artificial neural network model compilation and processing step (S140); and an artificial neural network model processing result reporting step (S150).

[0203] In the neural processing unit type selection step (S110), the user selects the type of neural processing unit to be evaluated for performance. The type of neural processing unit may vary depending on the product family of neural processing units sold by a specific company.

[0204] As shown in FIG. 7, a plurality of neural processing devices may be arranged to form a first group (G1), a plurality of neural processing devices may be arranged to form a second group (G2), a plurality of neural processing devices may be arranged to form a third group (G3), and a plurality of neural processing devices may be arranged to form a fourth group (G4).

[0205] The user selects the neural processor to perform performance evaluation from among the “” neural processor, “” neural processor, “” neural processor, and “” neural processor.

[0206] Additionally, the server (300) can select at least one neural processing device available within the NPU FARM among the selected specific types of neural processing devices. The specific types of neural processing devices selected at this time are not limited to a single type but may be multiple types.

[0207] Subsequently, in the compilation option selection step (S120), at least one of the multiple compilation options for the artificial neural network model to be processed by the selected at least one neural processing unit is selected.

[0208] More specifically, in the compilation option selection step (S120), compilation options can be set based on hardware information of the neural processing unit (220).

[0209] In addition, during the compilation option selection step, multiple compilation options can be set by the user's selection. Here, explanations regarding the advantages and disadvantages of each compilation option can be displayed on the user device (100). Thus, the user can select various compilation options in a manner that is suitable for the user. That is, the performance evaluation system (1000) can provide compilation options in a user customization manner, rather than preset options, to meet the user's specific requirements.

[0210] The above-described compilation option may be at least one of a pruning algorithm, a quantization algorithm, a model compression algorithm, a knowledge distillation algorithm, a retraining algorithm, and an AI-based model optimization algorithm.

[0211] Alternatively, the compilation option can be configured to select one of the prepared preset options.

[0212] Subsequently, in the artificial neural network model and dataset upload step (S230), at least one specific artificial neural network model for evaluating the performance of the selected neural processing device and at least one specific evaluation dataset input to the artificial neural network model are uploaded to the server (300). The aforementioned specific evaluation dataset refers to an evaluation dataset input to the artificial neural network model processing device (200) for evaluating the performance of the artificial neural network model processing device (200). Here, the artificial neural network model and dataset upload step (S230) may be referred to as the user data upload step.

[0213] Subsequently, in the artificial neural network model compilation and processing step (S140), the input artificial neural network model is compiled according to the selected compilation option, and the compiled machine code and evaluation dataset are input into a selected neural processing unit within the NPU FARM for processing.

[0214] In the artificial neural network model processing result reporting step (S150), the performance evaluation results of the neural processing unit that processed the compiled artificial neural network model may be reported. The performance evaluation result report may be stored in the user's account or sent to the user's email address. However, it is not limited to these methods, and the performance evaluation results may be provided to the user in various ways. The performance evaluation results are also treated as user data, and security policies applied to user data may be applied.

[0215] To elaborate, in the artificial neural network model processing result reporting step (S150), it is first determined whether the compiled artificial neural network model can be processed by a plurality of neural processing devices (220).

[0216] If the compiled artificial neural network model cannot be processed by a plurality of neural processing devices (220), in the artificial neural network model processing result reporting step (S150), the layer that cannot be processed by a plurality of neural processing devices (220) among the plurality of layers of the artificial neural network model may be reported.

[0217] And, among the multiple layers of the artificial neural network model, the layer that cannot be processed by the multiple neural processing units (220) can be processed by the graphics processing unit (230).

[0218] If the compiled artificial neural network model can be processed by a plurality of neural processing devices (220), the processing performance of the plurality of neural processing devices (220) can be reported in the artificial neural network model processing result reporting step (S150).

[0219] Indicators of processing performance may include the temperature profile of the neural processing unit, power consumption (Watt), Trillion Operations Per Second per Watt (TOPS / W), Frames Per Second (FPS), Inferences Per Second (IPS), and Accuracy.

[0220] According to another example of the present disclosure, a user can quickly determine in advance whether their artificial neural network model can operate on a specific NPU before purchasing the NPU.

[0221] According to another example of the present disclosure, a user can quickly know in advance, before purchasing a specific NPU, what performance their artificial neural network model will perform when run on a specific NPU.

[0222] According to another example of the present disclosure, when each NPU is connected via a server by type, the user can evaluate the user's artificial neural network model online, evaluate whether to run each NPU online, and receive the results.

[0223] Accordingly, the performance evaluation system (1000) can help the user quickly decide on a purchase by providing the user with information on the performance and price of the neural processing device required for the implementation of the AI ​​service developed by the user.

[0224] Hereinafter, with reference to FIG. 10, a method for evaluating the performance of an artificial neural network model according to another example of the present disclosure, in which a retraining step is added, will be described.

[0225] FIG. 10 is a flowchart illustrating a method for evaluating the performance of an artificial neural network model according to another example of the present disclosure.

[0226] Referring to FIG. 10, the artificial neural network model performance evaluation method (S300) includes a neural processing unit type selection step (S110); a compilation option selection step (S120); an artificial neural network model and dataset upload step (S230); an artificial neural network model compilation and processing step (S140); an artificial neural network model retraining step (S345); and an artificial neural network model processing result reporting step (S150).

[0227] In the neural processing unit type selection step (S110), the user selects the type of neural processing unit to be evaluated for performance. The type of neural processing unit may vary depending on the product family of neural processing units sold by a specific company.

[0228] As shown in FIG. 7, a plurality of neural processing devices may be arranged to form a first group (G1), a plurality of neural processing devices may be arranged to form a second group (G2), a plurality of neural processing devices may be arranged to form a third group (G3), and a plurality of neural processing devices may be arranged to form a fourth group (G4).

[0229] The user selects the neural processor to perform performance evaluation from among the “” neural processor, “” neural processor, “” neural processor, and “” neural processor.

[0230] Additionally, the server (300) can select at least one neural processing device available within the NPU FARM among the selected specific types of neural processing devices. The specific types of neural processing devices selected at this time are not limited to a single type but may be multiple types.

[0231] Subsequently, in the compilation option selection step (S120), at least one of the multiple compilation options for the artificial neural network model to be processed by the selected at least one neural processing unit is selected.

[0232] More specifically, in the compilation option selection step (S120), compilation options can be set based on hardware information of the neural processing unit (220).

[0233] In addition, during the compilation option selection step, multiple compilation options can be set by the user's selection. Here, explanations regarding the advantages and disadvantages of each compilation option can be displayed on the user device (100). Thus, the user can select various compilation options in a manner that is suitable for the user. That is, the performance evaluation system (1000) can provide compilation options in a user customization manner, rather than preset options, to meet the user's specific requirements.

[0234] The above-described compilation option may be at least one of a pruning algorithm, a quantization algorithm, a model compression algorithm, a knowledge distillation algorithm, a retraining algorithm, and an AI-based model optimization algorithm.

[0235] Alternatively, the compilation option can be configured to select one of the prepared preset options.

[0236] Subsequently, in the artificial neural network model and dataset upload step (S230), at least one specific artificial neural network model for evaluating the performance of the selected neural processing device and at least one specific evaluation dataset input to the artificial neural network model are uploaded to the server (300). The aforementioned specific evaluation dataset refers to an evaluation dataset input to the artificial neural network model processing device (200) for evaluating the performance of the artificial neural network model processing device (200). Here, the artificial neural network model and dataset upload step (S230) may be referred to as the user data upload step.

[0237] Subsequently, in the artificial neural network model compilation and processing step (S140), the input artificial neural network model is compiled according to the selected compilation option, and the compiled machine code and evaluation dataset are input into a selected neural processing unit within the NPU FARM for processing.

[0238] For example, the compilation option selected by the user may include a retraining option to perform retraining. If at least one retraining option is selected in the compilation option, a retraining step (S345) of the artificial neural network model may be performed after the artificial neural network model compilation and processing step (S140). In the retraining step (S345) of the artificial neural network model, the performance evaluation system (1000) assigns a graphics processing unit (230) to perform retraining in the artificial neural network model processing unit (200). For example, in the retraining step (S345) of the artificial neural network model, the graphics processing unit (230) may receive an artificial neural network model to which a pruning algorithm and / or a quantization algorithm have been applied and a training dataset as input to perform retraining. Retraining may be performed in units of epochs, and several to hundreds of epochs may be performed in the graphics processing unit (230).

[0239] The retraining option is a technique that can compensate for the reduced inference accuracy when various optimization options are applied. For example, when quantization options, pruning options, and model compression options are applied, the accuracy of the artificial neural network model inferred by multiple neural processing units (220) may be reduced. In such cases, an option may be provided to retrain the pruned, quantized, and / or model-compressed artificial neural network model online. The inference accuracy of the artificial neural network model after retraining can be increased again.

[0240] Specifically, retraining options may include quantization-aware retraining options, pruning-aware retraining options, and transfer learning options.

[0241] In the artificial neural network model processing result reporting step (S150), the performance evaluation results of the neural processing unit that processed the compiled artificial neural network model may be reported. The performance evaluation result report may be stored in the user's account or sent to the user's email address. However, it is not limited to these methods, and the performance evaluation results may be provided to the user in various ways. The performance evaluation results are also treated as user data, and security policies applied to user data may be applied.

[0242] To elaborate, in the artificial neural network model processing result reporting step (S150), it is first determined whether the compiled artificial neural network model can be processed by a plurality of neural processing devices (220).

[0243] If the compiled artificial neural network model cannot be processed by a plurality of neural processing devices (220), in the artificial neural network model processing result reporting step (S150), the layer that cannot be processed by a plurality of neural processing devices (220) among the plurality of layers of the artificial neural network model may be reported.

[0244] And, among the multiple layers of the artificial neural network model, the layer that cannot be processed by the multiple neural processing units (220) can be processed by the graphics processing unit (230).

[0245] If the compiled artificial neural network model can be processed by a plurality of neural processing devices (220), the processing performance of the plurality of neural processing devices (220) can be reported in the artificial neural network model processing result reporting step (S150).

[0246] Indicators of processing performance may include the temperature profile of the neural processing unit, power consumption (Watt), Trillion Operations Per Second per Watt (TOPS / W), Frames Per Second (FPS), Inferences Per Second (IPS), and Accuracy.

[0247] According to another example of the present disclosure, a user can quickly determine in advance whether their artificial neural network model can operate on a specific NPU before purchasing the NPU.

[0248] According to another example of the present disclosure, a user can quickly know in advance, before purchasing a specific NPU, what performance their artificial neural network model will perform when run on a specific NPU.

[0249] According to another example of the present disclosure, when each NPU is connected via a server by type, the user can evaluate the user's artificial neural network model online, evaluate whether to run each NPU online, and receive the results.

[0250] According to another example of the present disclosure, an artificial neural network model retraining algorithm optimized for a specific neural processing unit can be performed online through a performance evaluation system (1000). At this time, user data can be separated and protected from the operator of the performance evaluation system (1000) by the security policy described above.

[0251] Accordingly, the performance evaluation system (1000) can help the user quickly decide on a purchase by providing the user with information on the performance and price of the neural processing device required for the implementation of the AI ​​service developed by the user.

[0252] The present disclosure relates to an artificial neural network model performance evaluation system, which may be configured to include a plurality of neural processing units (NPUs) having different computational processing capabilities, a server on which at least one artificial neural network model (ANN) for performance evaluation of at least one of the plurality of NPUs and at least one specific evaluation dataset are uploaded, a compiler that selects at least one of the plurality of NPUs and compiles the ANN according to one of a plurality of compilation options, and a reporting program that processes the compiled ANN on the selected NPU and reports the performance evaluation results.

[0253] According to another feature of the present disclosure, a compiler can set the plurality of compilation options based on at least one of a quantization algorithm, a pruning algorithm, a retraining algorithm, a model compression algorithm, an AI-based model optimization algorithm, and a knowledge distillation algorithm.

[0254] According to another feature of the present disclosure, the NPU includes at least one internal memory and at least one multiplicator and accumulator, and the compiler can set the plurality of compilation options based on the hardware information of the NPU.

[0255] According to another feature of the present disclosure, a graphics processing unit that processes a retraining algorithm of the ANN among the plurality of compilation options may be further included.

[0256] According to another feature of the present disclosure, the reporting program can report whether an artificial neural network model compiled according to any one of a plurality of compilation options is operable on the NPU.

[0257] According to another feature of the present disclosure, if an artificial neural network model compiled according to any one of the plurality of compilation options cannot be operated on the NPU, the reporting program may report the layer that cannot be operated among the plurality of layers of the artificial neural network model.

[0258] According to another feature of the present disclosure, the reporting program reports the processing performance of an artificial neural network model compiled according to each of a plurality of compilation options, and the indicators of processing performance may include a temperature profile, power consumption, Trillion Operations Per Second per Watt (TOPS / W), Frames Per Second (FPS), Inferences Per Second (IPS), and Accuracy.

[0259] According to another feature of the present disclosure, the reporting program can analyze the processing performance of an artificial neural network model compiled according to each of a plurality of compilation options and suggest one of the plurality of compilation options.

[0260] According to another feature of the present disclosure, the plurality of compilation options may include first to third preset options, the first preset option may include a Post Training Quantization (PTQ) algorithm, the second preset option may include a Post Training Quantization (PTQ) algorithm and a layer-wise retraining algorithm for an artificial neural network model, and the third preset option may include a Post Training Quantization (PTQ) algorithm and a quantization-aware training (QAT) algorithm.

[0261] According to another feature of the present disclosure, at least one of data encryption, differential privacy, and data masking may be applied to at least one specific evaluation dataset uploaded to the server.

[0262] The present disclosure relates to a method for evaluating the performance of an artificial neural network model, comprising: a step of a user selecting the type and number of at least one Neural Processing Unit (NPU) to perform the performance evaluation; a step of selecting at least one of a plurality of compilation options for an artificial neural network (ANN) to be processed by the selected at least one NPU; a step of uploading the ANN to be processed by the selected at least one NPU and at least one specific evaluation dataset; a step of compiling and processing the ANN according to at least one of the selected plurality of compilation options; and a step of the selected NPU processing the ANN and reporting the performance evaluation.

[0263] According to another feature of the present disclosure, in the step of selecting at least one of the plurality of compilation options, the plurality of compilation options can be set based on at least one of a quantization algorithm, a pruning algorithm, a retraining algorithm, a model compression algorithm, an AI-based model optimization algorithm, and a knowledge distillation algorithm.

[0264] According to another feature of the present disclosure, the NPU includes at least one internal memory and at least one multiplying and accumulating unit, and in the step of selecting at least one of a plurality of compilation options, the plurality of compilation options can be set based on hardware information of the NPU.

[0265] According to another feature of the present disclosure, after the step of compiling and processing the ANN, the step of retraining the ANN may be further included.

[0266] According to another feature of the present disclosure, in the step of reporting the performance evaluation, whether an artificial neural network model compiled according to any one of a plurality of compilation options can be operated on the NPU may be reported.

[0267] According to another feature of the present disclosure, if an artificial neural network model compiled according to any one of the plurality of compilation options cannot be operated on the NPU, in the step of reporting a performance evaluation, a layer that cannot be operated among the plurality of layers of the artificial neural network model may be reported.

[0268] According to another feature of the present disclosure, in the step of reporting the performance evaluation, the processing performance of an artificial neural network model compiled according to each of a plurality of compilation options is reported, and the indicators of processing performance may include Trillion Operations Per Second per Watt (TOPS / W), Frames Per Second (FPS), Inferences Per Second (IPS), and Accuracy.

[0269] According to another feature of the present disclosure, in the step of reporting the performance evaluation, the processing performance of an artificial neural network model compiled according to each of a plurality of compilation options can be analyzed, and any one of the plurality of compilation options can be proposed.

[0270] According to another feature of the present disclosure, the plurality of compilation options may include first to third preset options, the first preset option may include a Post Training Quantization (PTQ) algorithm, the second preset option may include a Post Training Quantization (PTQ) algorithm and a layer-wise retraining algorithm for an artificial neural network model, and the third preset option may include a Post Training Quantization (PTQ) algorithm and a quantization-aware training (QAT) algorithm.

[0271] According to another feature of the present disclosure, at least one of data encryption, differential privacy, and data masking may be applied to the at least one specific evaluation dataset.

[0272] According to another feature of the present disclosure, in the step of uploading at least one specific evaluation dataset, one may sign terms regarding user data protection regarding the evaluation dataset.

[0273] The present disclosure relates to an artificial neural network model performance evaluation system, comprising a plurality of neural processors including a first neural processor of a first configuration and a second neural processor of a second configuration different from the first configuration, each having different computational processing performance, at least one operating processor, and a memory for storing instructions used in at least one operating processor. The at least one operating processor receives first processor selection information for selecting at least one of an artificial neural network model, a first neural processor, and a second neural processor, and compile option information, and compiles the artificial neural network model according to the compile option, thereby processing the artificial neural network model used in the neural processor selected by the first processor selection information, applying at least one evaluation data set to the artificial neural network model by the neural processor selected by the first processor selection information, and generating at least one first performance parameter related to the performance of the neural processor selected by the first processor selection information.

[0274] According to another feature of the present disclosure, an artificial neural network model performance evaluation system may further include a server that receives the first processor selection information, the compilation option information, and the at least one evaluation data set from a user device via a network, sends the first processor selection information, the compilation option information, and the at least one evaluation data set to the at least one operating processor, receives at least one first performance parameter from the at least one operating processor, and transmits the received at least one first performance parameter to the user device via a network.

[0275] According to another feature of the present disclosure, at least one of data encryption, differential privacy, and data masking may be applied to the at least one evaluation data set.

[0276] According to another feature of the present disclosure, the compile option information may include at least one of a quantization algorithm, a pruning algorithm, a retraining algorithm, a model compression algorithm, an AI-based model optimization algorithm, and a knowledge distillation algorithm.

[0277] According to another feature of the present disclosure, the first neural processor includes at least one internal memory and at least one multiplication and accumulation unit, and at least one operating processor can set the compilation option based on the first configuration.

[0278] According to another feature of the present disclosure, at least one operating processor can determine a layer among a plurality of layers of the artificial neural network model that can be processed by a neural processor selected by the first processor selection information.

[0279] According to another feature of the present disclosure, at least one operating processor may report a layer that cannot be processed by a neural processor selected by the first processor selection information among a plurality of layers of the artificial neural network model.

[0280] According to another feature of the present disclosure, an artificial neural network model performance evaluation system may further include a graphics processor that processes a layer that cannot be processed by a neural processor selected by the first processor selection information among a plurality of layers of the artificial neural network model.

[0281] According to another feature of the present disclosure, the graphics processor can perform retraining of an artificial neural network model.

[0282] According to another feature of the present disclosure, at least one first performance parameter may include at least one of a temperature profile, power consumption, Trillion Operations Per Second per Watt (TOPS / W), Frames Per Second (FPS), Inference Per Second (IPS), and inference and prediction accuracy.

[0283] According to another feature of the present disclosure, the at least one operating processor further receives second processor selection information for selecting at least one other of a first neural processor and a second neural processor, and by compiling the artificial neural network model, processes the artificial neural network model used in the neural processor selected by the second processor selection information, applies the at least one evaluation data set to the artificial neural network model by the neural processor selected by the second processor selection information, and generates at least one second performance parameter related to the performance of the neural processor selected by the second processor selection information.

[0284] According to another feature of the present disclosure, the at least one operating processor may recommend either the first processor selection information or the second processor selection information by comparing at least one first performance parameter and at least one second performance parameter.

[0285] According to another feature of the present disclosure, the compilation option may include at least one of a Post Training Quantization (PTQ) algorithm, a layer-wise retraining algorithm for an artificial neural network model, and a quantization-aware training (QAT) algorithm.

[0286] The present disclosure relates to a method for evaluating the performance of an artificial neural network model, comprising the steps of: receiving, by at least one operating processor, first processor selection information for selecting at least one of an artificial neural network model, a first neural processor, and a second neural processor, and compilation option information; by the at least one operating processor compiling the artificial neural network model according to the compilation option, processing the artificial neural network model used in the neural processor selected by the first processor selection information; by the at least one operating processor applying at least one evaluation data set to the artificial neural network model by the neural processor selected by the first processor selection information; and by the at least one operating processor generating at least one first performance parameter related to the performance of the neural processor selected by the first processor selection information and transmitting it through a network.

[0287] According to another feature of the present disclosure, a method for evaluating the performance of an artificial neural network model may further include the steps of: a server receiving the first processor selection information, the compilation option information, and the at least one evaluation data set from a user device via a network, and sending the first processor selection information, the compilation option information, and the at least one evaluation data set to the at least one operating processor; and the server receiving at least one first performance parameter from the at least one operating processor and transmitting the received at least one first performance parameter to the user device via a network.

[0288] According to another feature of the present disclosure, the method for evaluating the performance of an artificial neural network model may further include the step of setting the compilation option based on the hardware configuration of at least one of a first neural processor and a second neural processor.

[0289] According to another feature of the present disclosure, the method for evaluating the performance of an artificial neural network model may further include the step of reporting a layer that cannot be processed by a neural processor selected by the first processor selection information among a plurality of layers of the artificial neural network model.

[0290] According to another feature of the present disclosure, the method for evaluating the performance of an artificial neural network model may further include the step of a graphics processor processing a layer that cannot be processed by a neural processor selected by the first processor selection information among a plurality of layers of the artificial neural network model.

[0291] According to another feature of the present disclosure, the method for evaluating the performance of an artificial neural network model may further include the step of a graphics processor performing retraining of the artificial neural network model.

[0292] According to another feature of the present disclosure, a method for evaluating the performance of an artificial neural network model may further include: receiving second processor selection information in which at least one operating processor selects at least one other of a first neural processor and a second neural processor; processing the artificial neural network model used in the neural processor selected by the second processor selection information by compiling the artificial neural network model according to the compilation option; applying at least one evaluation data set to the artificial neural network model by the neural processor selected by the second processor selection information by the at least one operating processor; and generating at least one second performance parameter related to the performance of the neural processor selected by the second processor selection information and transmitting it through a network.

[0293] According to another feature of the present disclosure, the method for evaluating the performance of an artificial neural network model may further include the step of comparing at least one first performance parameter with at least one second performance parameter to recommend either the first processor selection information or the second processor selection information.

[0294] The present disclosure relates to a method for evaluating the performance of an artificial neural network model, comprising: a step of displaying an option for selecting a plurality of neural processors including a first neural processor of a first configuration and a second neural processor of a second configuration different from the first configuration; a step of receiving first processor selection information from a user for selecting at least one of the plurality of neural processors; a step of displaying a plurality of compilation options associated with the compilation of an artificial neural network model; a step of receiving first compilation option selection information among the plurality of compilation options from a user; a step of sending the first processor selection information, the first compilation option selection information, and at least one evaluation data set to a computing device connected to the plurality of neural processors; a step of receiving at least one first performance parameter associated with the processing performance of the neural processor selected by the first processor selection information according to the compilation option selected by the first compilation option selection information; and a step of displaying at least one first performance parameter.

[0295] According to another feature of the present disclosure, a method for evaluating the performance of an artificial neural network model may further include: receiving second processor selection information from a user for selecting at least one other of the plurality of neural processors; receiving second compilation option selection information from a user for selecting a plurality of compilation options; sending the second processor selection information and the second compilation option selection information to a computing device connected to the plurality of neural processors; and receiving at least one second performance parameter related to the processing performance of the neural processor selected by the second processor selection information according to the compilation option selected by the second compilation option selection information.

[0296] According to another feature of the present disclosure, the method for evaluating the performance of an artificial neural network model may further include the step of recommending either of the first processor selection information and the second processor selection information and displaying the same.

[0297] Although examples of the present disclosure have been described in more detail with reference to the attached drawings, the present disclosure is not necessarily limited to these examples and may be modified in various ways within the scope of the technical concept of the present disclosure. Accordingly, the examples disclosed in the present disclosure are intended to explain, not limit, the technical concept of the present disclosure, and the scope of the technical concept of the present disclosure is not limited by these examples. Therefore, the examples described above should be understood as illustrative in all respects and not restrictive. The scope of protection of the present invention shall be interpreted by the claims below, and all technical concepts within an equivalent scope shall be interpreted as being included within the scope of rights of the present invention. Explanation of the symbols

[0298] 100: User device 200: Artificial Neural Network Model Processing Unit 210: Compiler 211: Optimization Module 212: Verifier 213: Code Generator 220: Multiple neural processing units 221: Processing element array 222: NPU internal memory 223: NPU Controller 230: Graphics Processing Unit 240: Storage device 250: Reporting Program 300: Server 1000: Performance Evaluation System

Claims

Claim 1 An artificial neural network model performance evaluation system comprising: a plurality of neural processors including a first neural processor of a first configuration and a second neural processor of a second configuration different from the first configuration; at least one operating processor; and a memory for storing instructions used by the at least one operating processor, wherein the at least one operating processor receives first processor selection information for selecting at least one of an artificial neural network model, a first neural processor, and a second neural processor, and compile option information, and processes the artificial neural network model used in the neural processor selected by the first processor selection information by compiling the artificial neural network model according to the compile option, applies at least one evaluation data set to the artificial neural network model by the neural processor selected by the first processor selection information, generates at least one first performance parameter related to the performance of the neural processor selected by the first processor selection information, and reports a layer or operation that cannot be processed by the neural processor selected by the first processor selection information among a plurality of layers of the artificial neural network model. Claim 2 An artificial neural network model performance evaluation system according to claim 1, further comprising a server that receives the first processor selection information, the compilation option information, and the at least one evaluation data set from a user device via a network, sends the first processor selection information, the compilation option information, and the at least one evaluation data set to the at least one operating processor, receives at least one first performance parameter from the at least one operating processor, and transmits the received at least one first performance parameter to the user device via a network. Claim 3 An artificial neural network model performance evaluation system according to claim 1, wherein the first neural processor includes at least one internal memory and at least one multiplication and accumulation unit, and the at least one operating processor sets the compilation option based on the first configuration. Claim 4 An artificial neural network model performance evaluation system according to claim 1, wherein at least one operating processor analyzes the input data size of the artificial neural network model to generate corresponding dummy data when the at least one evaluation data set is not provided, and evaluates the performance of a neural processor selected by the first processor selection information using the generated dummy data. Claim 5 An artificial neural network model performance evaluation system according to claim 1, further comprising a graphics processor that processes a layer that cannot be processed by a neural processor selected by the first processor selection information among a plurality of layers of the artificial neural network model, or a graphics processor that processes an operation that cannot be processed. Claim 6 An artificial neural network model performance evaluation system according to claim 1, wherein the at least one first performance parameter comprises at least one of a temperature profile, power consumption, Trillion Operations Per Second per Watt (TOPS / W), Frame Per Second (FPS), Inference Per Second (IPS), and inference and prediction accuracy (Accuracy). Claim 7 An artificial neural network model performance evaluation system according to claim 1, wherein the at least one operating processor further receives second processor selection information for selecting at least one other of a first neural processor and a second neural processor, processes the artificial neural network model used in the neural processor selected by the second processor selection information by compiling the artificial neural network model, applies the at least one evaluation data set to the artificial neural network model by the neural processor selected by the second processor selection information, and generates at least one second performance parameter related to the performance of the neural processor selected by the second processor selection information. Claim 8 In claim 7, the artificial neural network model performance evaluation system, wherein the at least one operating processor compares the at least one first performance parameter and the at least one second performance parameter to recommend either the first processor selection information or the second processor selection information. Claim 9 An artificial neural network model performance evaluation system according to claim 1, wherein the at least one operating processor analyzes the structure of the artificial neural network model and determines whether the artificial neural network model is executable based on whether an operator included in each of the plurality of layers of the artificial neural network model is hardware-supported in a neural processor selected by the first processor selection information. Claim 10 An artificial neural network model performance evaluation system according to claim 1, wherein the at least one operating processor determines the processing order of each of a plurality of layers of the artificial neural network model based on structural data of a neural processor selected by the first processor selection information and graph information representing the connection relationship between the layers of the artificial neural network model. Claim 11 A method for evaluating the performance of an artificial neural network model, comprising: receiving, at least one operating processor, first processor selection information for selecting at least one of an artificial neural network model, a first neural processor, and a second neural processor, and compilation option information; processing the artificial neural network model used in the neural processor selected by the first processor selection information by compiling the artificial neural network model according to the compilation option; applying at least one evaluation data set to the artificial neural network model by the neural processor selected by the first processor selection information by the at least one operating processor; generating at least one first performance parameter related to the performance of the neural processor selected by the first processor selection information and transmitting it through a network by the at least one operating processor; and reporting a layer among a plurality of layers of the artificial neural network model that cannot be processed by the neural processor selected by the first processor selection information. Claim 12 A method for evaluating the performance of an artificial neural network model according to claim 11, further comprising: a step in which a server receives the first processor selection information, the compilation option information, and the at least one evaluation data set from a user device via a network, and sends the first processor selection information, the compilation option information, and the at least one evaluation data set to the at least one operating processor; and a step in which the server receives at least one first performance parameter from the at least one operating processor and transmits the received at least one first performance parameter to the user device via a network. Claim 13 An artificial neural network model performance evaluation method according to claim 11, further comprising the step of setting the compilation option based on at least one hardware configuration of the first neural processor and the second neural processor. Claim 14 A method for evaluating the performance of an artificial neural network model according to claim 11, further comprising the step of a graphics processor processing a layer that cannot be processed by a neural processor selected by the first processor selection information among a plurality of layers of the artificial neural network model. Claim 15 A method for evaluating the performance of an artificial neural network model according to claim 11, further comprising the step of a graphics processor performing retraining of the artificial neural network model. Claim 16 A method for evaluating the performance of an artificial neural network model according to claim 11, further comprising: receiving second processor selection information in which at least one operating processor selects at least one other of a first neural processor and a second neural processor; processing the artificial neural network model used in the neural processor selected by the second processor selection information by compiling the artificial neural network model according to the compilation option; applying at least one evaluation data set to the artificial neural network model by the neural processor selected by the second processor selection information by the at least one operating processor; and generating at least one second performance parameter related to the performance of the neural processor selected by the second processor selection information and transmitting it through a network. Claim 17 An artificial neural network model performance evaluation method according to claim 16, further comprising the step of comparing at least one first performance parameter and at least one second performance parameter to recommend either the first processor selection information or the second processor selection information. Claim 18 A method for evaluating the performance of an artificial neural network model according to claim 11, further comprising the step of the at least one operating processor analyzing the structure of the artificial neural network model and determining whether the artificial neural network model is executable based on whether an operator included in each of the plurality of layers of the artificial neural network model is hardware-supported in a neural processor selected by the first processor selection information. Claim 19 An artificial neural network model performance evaluation method according to claim 11, further comprising the step of determining the processing order of each of a plurality of layers of the artificial neural network model based on structural data of a neural processor selected by the first processor selection information and graph information representing the connection relationship between the layers of the artificial neural network model. Claim 20 A method for evaluating the performance of an artificial neural network model according to claim 11, wherein the at least one operating processor further comprises the steps of: analyzing the input data size of the artificial neural network model when the at least one evaluation data set is not provided; generating dummy data corresponding to the input data size; and evaluating the performance of a neural processor selected by the first processor selection information using the generated dummy data. Claim 21 A method for evaluating the performance of an artificial neural network model, comprising: a step of displaying an option for selecting a plurality of neural processors including a first neural processor of a first configuration and a second neural processor of a second configuration different from the first configuration; a step of receiving first processor selection information from a user for selecting at least one of the plurality of neural processors; a step of displaying a plurality of compilation options associated with the compilation of an artificial neural network model; a step of receiving first compilation option selection information among the plurality of compilation options from a user; a step of sending the first processor selection information, the first compilation option selection information, and at least one evaluation data set to a computing device connected to the plurality of neural processors; a step of receiving information regarding a layer among the plurality of layers of the artificial neural network model that cannot be processed by the neural processor selected by the first processor selection information; and a step of displaying information regarding the layer that cannot be processed. Claim 22 An artificial neural network model performance evaluation method according to claim 21, further comprising: receiving at least one first performance parameter related to the processing performance of a neural processor selected by the first processor selection information according to the compilation option selected by the first compilation option selection information; and displaying at least one first performance parameter.

Images