Method for training neural network model, system for generating neural network model, and non-transitory computer-readable storage medium

Abstract

A method for training a neural network model based on a plurality of heterogeneous processing units may include training a neural network model having a plurality of lightweight weights by using a training data set on a first processing unit to be executable on a second processing unit, and performing inference on the evaluation data set using the plurality of lightweight weights on the second processing unit.

Description

Technical Field

[0001] This invention relates to a method and system for training neural network models based on multiple heterogeneous processing devices. Background Technology

[0002] Humans possess intelligence such as identification, classification, inference, prediction, and control / decision-making. Artificial intelligence (AI) is an artificial imitation of human intelligence.

[0003] The human brain is composed of nerve cells called neurons. Each neuron connects to hundreds or thousands of other neurons through connection structures called synapses. To mimic human intelligence, neural network (NN) models simulate the operation of biological neurons and the connections between multiple neurons. In other words, neural networks mimic neurons through a layered system of interconnected nodes.

[0004] Neural network models can be categorized into "single-layer neural networks" and "multi-layer neural networks" based on the number of layers. A typical multi-layer neural network consists of an input layer, hidden layers, and an output layer. The input layer receives external data, and the number of neurons in the input layer is the same as the number of input variables. The hidden layer is located between the input and output layers, receiving multiple signals from the input layer, extracting multiple features, and passing them to the output layer. The output layer receives multiple signals from the hidden layers and outputs them to the outside. Multiple input signals between neurons are first multiplied by their respective weights (whose values ​​are between 0 and 1), and then summed. If the sum is greater than the neuron's threshold, the neuron is activated and outputs the value through an activation function.

[0005] On the other hand, increasing the number of hidden layers in a neural network to achieve a higher level of artificial intelligence is called a deep neural network (DNN).

[0006] Deep neural networks come in many types, but Convolutional Neural Networks (CNNs) are renowned for their ability to extract features from input data and recognize multiple feature patterns. A CNN functions similarly to the visual cortex of the human brain, which processes images. CNNs are known to be particularly well-suited for image processing. A CNN consists of a series of convolutional channels and multiple pooling channels.

[0007] Convolutional operations account for the majority of computation time in convolutional neural networks (CNNs). CNNs use matrix-like kernels to extract multiple image features from each channel to identify multiple objects, and pooling provides homomorphisms such as motion and deformation. In each channel, input data is convolved with the kernel to obtain a feature map, followed by the application of activation functions such as the rectified linear unit (ReLU) to generate activation maps for each channel, after which pooling can be performed. The neural network that actually performs pattern classification is located at the end of the feature extraction neural network, called a fully connected layer. In the computational processing of CNNs, most of the computation is done through convolution or matrix multiplication.

[0008] With the development of AI inference capabilities, various electronic devices such as AI speakers, smartphones, smart refrigerators, VR devices, AR devices, AI monitors, AI robot vacuum cleaners, tablet computers, laptops, self-driving cars, bipedal robots, quadrupedal robots, and industrial robots can now use AI to provide various inference services such as sound recognition, speech recognition, image recognition, object detection, driver fatigue detection, dangerous moment detection, and gesture detection.

[0009] With the recent development of deep learning technology, the performance of artificial neural network inference services is continuously improving through big data learning. These artificial neural network inference services repeatedly train on artificial neural networks using a large amount of training data and infer various complex data from the trained neural network models. Therefore, diversified services are being provided for the aforementioned electronic devices by utilizing artificial neural network technology. Summary of the Invention

[0010] Several embodiments relate to the execution of training a neural network model on a first processing circuit for use on a second processing circuit having a different configuration than the first processing circuit. Training or retraining of the neural network model is performed on the first processing circuit to obtain multiple first weights for the neural network model. The multiple first weights are transferred from the first processing circuit to the second processing circuit. After inference is performed by the second processing circuit, multiple evaluation results are generated on the performance of the inference in the second processing circuit using the transferred multiple first weights. The multiple evaluation results are sent to the first processing circuit to update the multiple first weights to multiple second weights. The multiple second weights are transferred to the second processing circuit to perform inference.

[0011] In one or more embodiments, the amount of data of the plurality of third weights operable by the first processing circuit is greater than the amount of data of the plurality of first weights or the plurality of second weights operable by the second processing circuit.

[0012] In one or more embodiments, the first processing circuit is a general-purpose graphics processing unit (GPGPU), and the second processing circuit is a neural processing unit (NPU).

[0013] In one or more embodiments, the second processing circuitry includes internal memory, multiple processing elements coupled to the internal memory and configured to perform multiple multiply-accumulate operations using the multiple first weights or the multiple second weights, and an activation function operation circuitry coupled to at least the internal memory or the multiple processing elements. The activation function operation circuitry applies an activation function to the outputs from the multiple processing elements.

[0014] In one or more embodiments, the power consumption of the second processing circuit is lower than that of the first processing circuit.

[0015] In one or more embodiments, one or more compilation options associated with the neural network model are configured for training or retraining of the first processing circuit and deployment of the second processing circuit. The neural network model is compiled according to the multiple compilation options to generate machine code for instantiating the neural network model in the first processing circuit. The machine code is then transmitted to the second processing circuit for execution.

[0016] In one or more embodiments, one or more compilation options include performing at least one of the following: pruning algorithm, quantization algorithm, parameter refinement algorithm, outlier mitigation algorithm, model compression algorithm, knowledge distillation algorithm, retraining algorithm, and artificial intelligence-based optimization algorithm.

[0017] In one or more embodiments, multiple evaluation results include at least one of the following: temperature distribution of the second processing circuit, power consumption of the second processing circuit, number of operations per unit power consumption, frames per second (FPS), inferences per second (IPS), and accuracy.

[0018] In one or more embodiments, multiple evaluation results are generated from the second processing circuit and sent to the first processing circuit on an epoch-by-epoch basis. Attached Figure Description

[0019] Figure 1 The diagram illustrates an example neural network model.

[0020] Figure 2 This is a schematic diagram illustrating a convolutional neural network related to the present invention.

[0021] Figure 3This is a schematic diagram illustrating the operation of a convolutional neural network related to the present invention.

[0022] Figure 4 This is a block diagram illustrating the configuration of a neural network model training system based on a heterogeneous processing device according to an example of the present invention.

[0023] Figure 5 This is a block diagram illustrating the configuration of a neural network model performance evaluation device according to an example of the present invention.

[0024] Figure 6 This is a block diagram illustrating the configuration of at least one of a plurality of neural processing units according to an example of the present invention.

[0025] Figure 7 The block diagram of a neural network model training system based on a heterogeneous processing device in a neural network model performance evaluation device is illustrated according to an example of the present invention.

[0026] Figure 8 This is a block diagram illustrating the configuration of a compiler for a neural network model performance evaluation device according to an example of the present invention.

[0027] Figure 9 This is a block diagram illustrating the configuration of an optimization module of a neural network model performance evaluation device according to an example of the present invention.

[0028] Figure 10 This is a schematic diagram illustrating one of a plurality of processing elements according to an example of the present invention.

[0029] Figure 11 This is an example flowchart illustrating a neural network model training method based on a heterogeneous processing device according to an example of the present invention.

[0030] [Explanation of Labels in the Attached Image]

[0031] 10: Controller

[0032] 20: Direct Memory Access Module

[0033] 30: Internal Memory

[0034] 40,40-1: Processing element

[0035] 41: Multiplier

[0036] 42: Adder

[0037] 43: Accumulator

[0038] 44: Bit quantization unit

[0039] 50: Special Function Unit

[0040] 100: General-purpose graphics processor

[0041] 110a: Neural Network Model

[0042] 110a-1: Input Layer

[0043] 110a-2: First Connection Network

[0044] 110a-3: First Hidden Layer

[0045] 110a-4: Second Connection Network

[0046] 110a-5: Second Hidden Layer

[0047] 110a-6: Third Connection Network

[0048] 110a-7: Third Hidden Layer

[0049] 200, 200-1, 200-N: Neural processing units

[0050] 300: Memory

[0051] 310: Compiler

[0052] 311: Optimizer

[0053] 312: Validator

[0054] 313: Code Generation Module

[0055] 320: Storage

[0056] 330: Reporting Procedure

[0057] 700: Central Processing Unit

[0058] 1000: User device

[0059] 1120: Processor

[0060] 1140: Display device

[0061] 1160: User Interface

[0062] 1180: Network Interface

[0063] 1200: Memory

[0064] 2000: Performance Evaluation Device

[0065] 3000: Server

[0066] 3120: Processor

[0067] 3160: Network Interface

[0068] 3180: Memory

[0069] 10000: Neural Network Model Training System

[0070] PE1, PE2, PE3, PE4, PE5, PE6: Processing components

[0071] S110, S120, S130: Steps Detailed Implementation

[0072] The specific structures or step-by-step descriptions used to illustrate the inventive concepts disclosed in this specification or application are merely examples for the purpose of illustrating examples of the inventive concepts.

[0073] Examples of the concepts according to the present invention may be implemented in various forms. Examples of the concepts according to the present invention should not be construed as limited to those described in this specification or application.

[0074] Examples of the concepts according to the present invention can be applied with various modifications. The present invention can be presented in many forms. Therefore, specific examples are illustrated in the accompanying drawings and described in detail in the present invention. However, this is not intended to limit the examples of the concepts according to the present invention to a particular inventive form. Therefore, it should be understood that all modifications, equivalents, or alternatives contained within the spirit and scope of the present invention are included in the present invention.

[0075] The terms "first" and / or "second" can be used to describe multiple elements. However, the invention should not be limited to the foregoing terms. These terms are used only to distinguish one element from another. For example, without departing from the scope of the claims according to the concept of the invention, a first element may be referred to as a second element, and similarly, a second element may also be referred to as a first element.

[0076] When an element is described as being "connected to" or "in contact with" another element, it should be understood that the other element may be directly connected to or in contact with the other element, but there may also be other elements positioned between them. On the other hand, when it is mentioned that an element is "directly connected to" or "directly in contact with" another element, it should be understood that there are no other elements between them. Other expressions describing the relationship between elements, such as "between" and "directly between" or "adjacent" and "directly adjacent," should be interpreted similarly.

[0077] In this invention, expressions such as “A or B”, “at least one of A and / or B”, or “one or more of A and / or B” may include all of their combinations. For example, “A or B”, “at least one of A and B”, or “at least one of A or B” may refer to: (1) containing at least one A, (2) containing at least one B, or (3) containing at least one A and at least one B.

[0078] As used herein, expressions such as "first," "second," or "first or second" may modify individual elements regardless of their order and / or importance. These expressions are used only to distinguish one element from others and do not limit the number of elements. For example, "first user equipment" and "second user device" may represent different user devices, regardless of their order or importance. For example, without departing from the scope of the invention, a first element may be renamed a second element, and similarly, a second element may be renamed a first element.

[0079] The terminology used in this invention is for describing specific examples only and is not intended to limit the scope of other examples. Singular expressions may include plural expressions unless the context clearly specifies otherwise. The terminology used herein, including technical or scientific terms, may have the same meaning as understood by one of ordinary skill in the art to which this document pertains.

[0080] In the terminology used in this invention, terms as defined in a general dictionary may be interpreted as having the same or similar meaning as in the context of the related art. Unless expressly defined herein, they should not be interpreted in an idealized or overly formalized manner. In some cases, even the terms defined in this invention should not be construed as excluding examples of this invention.

[0081] The terminology used herein is for the purpose of describing particular examples only and is not intended to limit the invention. Singular expressions include plural expressions unless the context clearly specifies otherwise. In this specification, terms such as “comprising” or “having” are intended to indicate the presence of the described features, quantities, steps, operations, components, parts, or combinations thereof. Therefore, it should be understood that the presence or addition of one or more other features, quantities, steps, operations, components, parts, or combinations thereof is not excluded.

[0082] Unless otherwise defined, all terms used herein (including technical or scientific terms) shall have the same meaning as understood by one of ordinary skill in the art. Terms as defined in a general dictionary shall be interpreted as having the meaning consistent with their meaning in the context of the relevant art. Unless expressly defined in this invention, they shall not be interpreted in an idealized or overly formalized manner.

[0083] The features of the various embodiments of the present invention can be combined or incorporated into each other, either partially or entirely. The various embodiments of the present invention are technically capable of achieving various linkages and drives that are fully understood by those skilled in the art. The various embodiments of the present invention can be implemented independently of each other or can be implemented together in a related manner.

[0084] In the descriptive examples, descriptions of technical content known in the technical field to which this invention pertains but not directly related to this invention may be omitted. This is done to more clearly convey the essence of the invention without obscuring it by omitting unnecessary descriptions.

[0085] To facilitate understanding of this invention, the following is a brief explanation of the terminology used herein:

[0086] GPGPU (or General Purpose GPU): refers to the use of multiple graphics processing units (GPUs) traditionally used for graphics rendering to perform computations typically handled by a central processing unit (CPU). With GPGPU, the ability of multiple GPUs to manage a large number of parallel computations is utilized to perform tasks beyond graphics rendering, such as multiple scientific simulations, machine learning, and other parallelizable computational tasks.

[0087] NPU: an abbreviation for Neural Processing Unit, which can refer to a processor dedicated to computing neural network models and independent of the central processing unit.

[0088] NN: an abbreviation for Neural Network, a network of nodes connected in a layered structure, mimicking the way multiple neurons in the human brain are connected through multiple synapses to simulate human intelligence.

[0089] Information of a neural network: Information may include the network structure, information about the number of layers, information about the connections between layers, information about multiple parameters of each layer, information about the computational processing methods, information about the activation functions, the data type (such as floating-point numbers or integers) of multiple parameters of each layer, and the bit width of each parameter.

[0090] DNN: an abbreviation for Deep Neural Network, which can refer to an increase in the number of hidden layers in a neural network to achieve a higher level of artificial intelligence.

[0091] CNN: an abbreviation for Convolutional Neural Network, a neural network that operates similarly to the visual cortex of the human brain in processing multiple images. Convolutional neural networks are known for being particularly well-suited for image processing and for their ability to extract features from input data and recognize feature patterns.

[0092] Transformer: A transformer neural network is a DNN based on multiple attention techniques. It utilizes multiple matrix multiplication operations. The transformer can receive input values ​​and parameters such as query (Q), key (K), and value (V) to obtain the output value, i.e., attention (Q,K,V). Based on the output value (i.e., attention (Q,K,V)), the transformer can handle various inference operations.

[0093] Kernel: refers to the multiple weights of an N×M matrix formed by multiple convolutions. Each layer of a neural network model has multiple kernels, and the number of kernels can be referred to as the number of channels, the number of filters, etc.

[0094] In recent years, various neural processing units have been developed to accelerate the instruction cycle of artificial intelligence applications. However, as the required functionality and accuracy of multiple inference services utilizing neural networks increase, the size of multiple neural network model parameters, computational demands, and the number of learning parameters have increased significantly. These trends have significantly raised the performance requirements for effectively executing inference operations across multiple processors and memory.

[0095] Meanwhile, multiple neural processing units can accelerate AI computation and reduce power consumption. However, various neural processing devices use multiple dedicated neural network models and corresponding dedicated acceleration circuits to accelerate AI computation and reduce power consumption. Therefore, when a neural network model trained on a graphics processing unit is executed on an NPU, the accuracy of the neural network model may differ from that when executed on a graphics processing unit.

[0096] Traditional neural network models are constrained by excessive power consumption, significant heat generation, processor bottlenecks caused by high memory bandwidth requirements, and memory latency. Therefore, improving the computational performance of neural network models presents various challenges, prompting research into lightweight neural network models to address these issues. Specifically, when the number of parameters in a neural network model is large, the processor may be unable to prepare the necessary data in advance, leading to frequent latency. Furthermore, in such cases, the processor may enter a starved or idle state due to insufficient data supply, preventing it from performing actual calculations and thus reducing computational performance.

[0097] These challenges are exacerbated by the diverse range of electronic devices used for on-device or edge AI computing. Edge AI computing refers to the surrounding environment in which AI computations occur, including the devices that directly generate data and various electronic devices in the vicinity of multiple data-generating devices. Such systems are often referred to as edge AI devices.

[0098] To clarify, an edge AI device can be defined as an AI computing system located at the periphery of a cloud computing system, at a certain distance from and communicating with multiple servers in a data center. Edge AI devices can also perform multiple tasks requiring real-time and reliable performance, such as autonomous robots or self-driving cars that need to process large amounts of data within a millisecond. Therefore, the range of applications for edge AI devices is rapidly expanding.

[0099] Therefore, various embodiments relate to techniques for lightweighting neural network models to make them suitable for standalone, low-power, low-cost neural processing devices. In other words, the number of parameters in a neural network model can be reduced so that the model can be embedded in multiple electronic devices, allowing for multiple independent operations.

[0100] Furthermore, several embodiments address several aspects of commercially available multiple neural processing units (NPUs) designed to handle multiple neural network models. First, there is a lack of sufficient information to select a suitable NPU to execute user-developed neural network models. Second, multiple NPUs are in the early stages of commercialization, and determining whether a GPGPU-based neural network model can operate on a specific NPU requires reviewing various questionnaires, multiple data sheets, and obtaining technical support from engineers. In particular, the number of layers, parameter scale, and special functions of the model can be modified based on user needs, making it difficult to generalize the neural network model. Third, predicting in advance whether a user-developed neural network model will be challenging to run on a specific NPU may lead to disputes after post-purchase evaluation that the NPU does not support specific operations or functions. Fourth, the development environments used by users are mostly GPGPU environments, but multiple general-purpose graphics processors and multiple neural processing units can have multiple different computational circuits processing essentially the same algorithms. Therefore, each time a neural network model is compiled for the NPU, the computation results can vary due to these differences in the multiple computational circuits. Furthermore, it is impossible to fully consider the multiple differences in the multiple computational circuits between multiple neural processing units and multiple general-purpose graphics processors when training neural network models on GPGPUs. Fifth, it is difficult to predict how a user-developed neural network model will perform when executed on a specific NPU. Specifically, it is challenging to determine in advance whether the expected power consumption and frames per second requirements will be met.

[0101] In particular, because the magnitude of multiple weights, the size of feature maps, the number of layers, the various characteristics of activation functions, and other factors vary from model to model, predicting ideal performance in advance is challenging. Therefore, the inventors of this invention aim to provide a method and apparatus for training neural network models, enabling users to more efficiently determine the optimal NPU product selection and the model optimization conditions of the selected NPU. This can be achieved by using AI program code (such as TensorFlow)... TM PyTorch TM ONNX TM When model files (such as model files) are uploaded to a specific online simulation service, all necessary operations are performed in batches online, thereby achieving this by providing users with solutions or services that offer the best convenience and value.

[0102] Furthermore, since the device-side neural processing unit can be designed for multiple low-cost, low-power embedded devices, it is typically used to perform only a few inference functions without supporting the training of multiple neural network models.

[0103] On the other hand, the results and evaluations from multiple general-purpose graphics processing units (GPUs) acting as general-purpose computing devices are not always the same as the results from multiple neural processing units (NNs). Specifically, because NNs have a pipelined architecture, unlike multiple GPUs, the accuracy of inference performed by a neural network model executed on multiple GPUs can differ from the accuracy of inference performed on multiple NNs. Therefore, multiple embodiments can measure the inference accuracy of a model on an NPU while simultaneously training a neural network model on a GPGPU to improve its accuracy when executed on a NN.

[0104] Furthermore, when evaluating the accuracy of training a neural network model on an evaluation dataset using a GPGPU, it can be disadvantageous in terms of both speed and power consumption compared to performing computations on a relatively simple NPU. Therefore, several embodiments can also improve instruction cycle and power efficiency by simultaneously training the neural network model on a GPGPU and executing the neural network model on the NPU to measure its inference accuracy. Since the scope of this invention is not limited to the above, other aspects not explicitly listed will be understood by those skilled in the art from the following description.

[0105] Figure 1 This diagram illustrates an example neural network model. Several operations of the example neural network model 110a, which can operate in an NPU, will then be described. Figure 1The example neural network model 110a can be a neural network trained to perform various inference functions such as object recognition and speech recognition. The neural network model 110a can be a deep neural network. However, the example neural network model 110a according to the present invention is not limited to a deep neural network. For example, neural network model 110a can be Siamese Network, Triplet Network, Contrastive Loss, FaceNet, DeepID, SphereFace, ArcFace, Florence-2, DaViT (Dual Attention Visual Transformer), MobileViT, ViT, Swin-Transformer, Transformer, YOLO, CNN, PIDNet (Parallel Interactive Decoding Network), BiseNet (Dual Path Segmentation Network), RCNN (Region Convolutional Neural Network), VGG, VGG16, DenseNet (Dense Connection Network), SegNet (Segmentation Network), DeconvNet (Deconvolutional Network), DeepLAB Models such as V3+, U-net, SqueezeNet (a compressed network), AlexNet, ResNet18, MobileNet-v2, GoogLeNet, ResNet-v2, ResNet50, ResNet101, and Inception-v3 are used. The scope of this invention is not limited to the models mentioned above. The neural network model 110a can also be an ensemble model based on at least two different models.

[0106] In the following description, the inference procedure performed by the example neural network model 110a will be described. The neural network model 110a is an example deep neural network model, comprising an input layer 110a-1, a first connection network 110a-2, a first hidden layer 110a-3, a second connection network 110a-4, a second hidden layer 110a-5, a third connection network 110a-6, and an output layer 110a-7. However, the present invention is not limited to this. Figure 1 The neural network model shown. The first hidden layer 110a-3 and the second hidden layer 110a-5 can also be referred to as multiple hidden layers.

[0107] Input layer 110a-1 may, for example, contain input node x1 and input node x2, that is, input layer 110a-1 may contain information about two input values.

[0108] The first connection network 110a-2 may contain information about six weight values ​​used to connect the nodes of the input layer 110a-1 to the nodes of the first hidden layer 110a-3. Each weight value is multiplied by an input node value, and the sum of the multiple multiplied values ​​is stored in the first hidden layer 110a-3. The multiple weight values ​​and the multiple input node values ​​may be referred to as multiple parameters of the neural network model.

[0109] The first hidden layer 110a-3 may contain nodes a1, a2 and a3, that is, the first hidden layer 110a-3 may contain information about the values ​​of the three nodes.

[0110] Figure 1 The first processing element PE1 can handle multiple operations on node a1.

[0111] Figure 1 The second processing element PE2 can handle multiple operations on node a2.

[0112] Figure 1 The third processing element PE3 can process multiple operations on node a3. The second connection network 110a-4 may, for example, contain information about nine weight values ​​for connecting each node of the first hidden layer 110a-3 to each node of the second hidden layer 110a-5. Each of the multiple weight values ​​of the second connection network 110a-4 is multiplied by multiple node values ​​from the first hidden layer 110a-3, and the sum of the multiple multiplied values ​​is stored in the second hidden layer 110a-5.

[0113] The second hidden layer 110a-5 may contain nodes b1, b2 and b3. For example, the second hidden layer 110a-5 may contain information about the values ​​of the three nodes.

[0114] Figure 1 The fourth processing element PE4 can handle multiple operations on the b1 node.

[0115] Figure 1 The fifth processing element PE5 can handle multiple operations on the b2 node.

[0116] Figure 1 The sixth processing element PE6 can handle multiple operations on the b3 node.

[0117] For example, the third connection network 110a-6 may contain information about six weight values ​​that connect each node of the second hidden layer 110a-5 to each node of the output layer 110a-7. Each of the weight values ​​of the third connection network 110a-6 is multiplied by a plurality of node values ​​from the second hidden layer 110a-5, and the sum of these multiplications is stored in the output layer 110a-7.

[0118] Output layer 110a-7 may contain node y1 and node y2. For example, output layer 110a-7 may contain information about the values ​​of the two nodes.

[0119] Figure 1 The seventh processing element PE7 can handle the operations on node y1.

[0120] Figure 1 The eighth processing element PE8 can handle operations on the y2 node.

[0121] Each node can correspond to a feature value, and each feature value can correspond to a feature map.

[0122] Figure 2 This diagram illustrates a convolutional neural network related to the present invention. The convolutional neural network can be a combination of one or more convolutional layers, pooling layers, and fully connected layers. Convolutional neural networks have a structure suitable for learning and inference from two-dimensional data and can be trained using the backpropagation algorithm.

[0123] In the examples of this invention, the convolutional neural network has kernels for each channel to extract multiple features from the channel's input image. The kernels can be organized as a two-dimensional matrix and perform multiple convolution operations while traversing the input data. The size of the kernels can be arbitrary, and the stride of the kernel traversing the input data can also be arbitrary. The result of each kernel performing convolution on the entire input data can be referred to as a feature map or activation map.

[0124] In the following description, a kernel may contain a single set of weights or multiple sets of weights. The number of kernels used in each layer may be referred to as the number of channels.

[0125] Since convolution operations involve multiple kernels and input data, activation functions can be applied to increase nonlinearity. When an activation function is applied to the feature map of the result of a convolution operation, the feature map is called an activation map.

[0126] Specifically, refer to Figure 2 A convolutional neural network may contain at least one convolutional layer, at least one pooling layer, and at least one fully connected layer. For example, convolution can be defined by two main parameters: the size of the input data (typically a 1×1, 3×3, or 5×5 matrix) and the depth of the output feature map (the number of kernels). These key parameters can be derived through convolution operations. These convolutions may start at a depth of 32, continue to a depth of 64, and end at a depth of 128 or 256. A convolution operation can be defined as sliding a 3×3 or 5×5 kernel across the input image matrix, multiplying the kernel weights by the overlapping elements of the input image matrix, and then summing all the results.

[0127] Activation functions can be applied to the output feature map generated in this way to finally output an activation map. Furthermore, the weights used in the current layer can be passed to subsequent layers via convolution. Pooling layers can perform pooling operations to reduce the size of the feature map by downsampling the output data (i.e., the activation map). For example, pooling operations may include, but are not limited to, max pooling and / or average pooling.

[0128] Max pooling uses a kernel and outputs the maximum value in the region where the feature map overlaps with the kernel by sliding the feature map and kernel. Average pooling outputs the average value in the region where the feature map overlaps with the kernel. Thus, because the size of the feature map is reduced by the pooling operation, the number of weights in the feature map is also reduced.

[0129] Fully connected layers can classify the data output from pooling layers into multiple categories (i.e., inference values) and output the classified categories and their scores. The data output from pooling layers forms a three-dimensional feature map, which can be converted into a one-dimensional vector and used as input to fully connected layers.

[0130] Figure 3 This is a diagram illustrating the operation of a convolutional neural network. (See reference...) Figure 3 The example input image is presented as a 6×6 two-dimensional matrix. Furthermore, in... Figure 3 In this process, three nodes are used, named Channel 1, Channel 2, and Channel 3.

[0131] First, the convolution behavior is described. Input image ( Figure 3 Presented as 6×6) and core 1 for channel 1 at the first node ( Figure 3 The features are presented as 3×3) convolutions, and the features Figure 1 ( Figure 3 The image (presented as a 4x4 image) is output as the result. Furthermore, the input image (...) Figure 3 (represented in 6×6 dimensions) and core 2 (used for channel 2 at the second node) Figure 3 Convolution is performed (represented by a 3×3 size), and the features... Figure 2 ( Figure 3 The image (represented in 4×4 dimensions) is output as the result. Furthermore, the input image is compared with kernel 3 (used for channel 3) at the third node. Figure 3 Convolution is performed (represented by a 3×3 size), and the features... Figure 3 ( Figure 3 The result is output as a 4×4 dimension.

[0132] In order to handle each convolution operation, the processing elements PE1 to PE12 of the neural processing unit are configured to perform multiplication and accumulation operations.

[0133] Next, the operation of activation functions will be described. Activation functions can be applied to the features output by the autoconvolution operation. Figure 1,feature Figure 2 and characteristics Figure 3 ( Figure 3 Each element is presented as a 4×4 sample size. The output after the activation function is applied can also be a 4×4 sample size.

[0134] Next, the pooling operation will be described. The features output from the activation functions above... Figure 1 ,feature Figure 2 and characteristics Figure 3 ( Figure 3 Each feature map (4×4) is input to three nodes. Pooling operations can be performed using multiple feature maps output from the activation function as input. Pooling can be performed to reduce the size of the matrix or to emphasize multiple specific values ​​in the matrix. Several pooling methods include max pooling, average pooling, and min pooling. Max pooling is used to collect the maximum number of values ​​within a specific region of the matrix, while average pooling can be used to average multiple values ​​within a specific region.

[0135] At Figure 3 In the example, a 4×4 feature map is rendered by reducing it to a 2×2 size through pooling. Specifically, the first node is used for the feature map of channel 1. Figure 1 As input, pooling is performed and the output is, for example, a 2×2 matrix. The second node uses features for channel 2. Figure 2 As input, pooling is performed and the output is, for example, a 2×2 matrix. The third node is used for features of channel 3. Figure 3 As input, perform pooling and output, for example, a 2×2 matrix.

[0136] The aforementioned convolution, activation function, and pooling are repeatedly performed, and ultimately, the output can be as follows: Figure 2 The diagram shown is fully connected.

[0137] Among various deep neural network models, convolutional neural networks (CNNs) are the most popular method in the field of computer vision. In particular, CNNs have demonstrated outstanding performance in research areas that perform various tasks such as image classification and object detection.

[0138] Figure 4 This is a block diagram illustrating the configuration of a neural network model training system based on a heterogeneous processing device according to an example of the present invention. (Refer to...) Figure 4 According to an example of the present invention, a neural network model training system based on a heterogeneous processing device (hereinafter referred to as a "neural network model training system") 10000 may include a user device 1000, a neural network model performance evaluation device 2000, and a server 3000.

[0139] Figure 4The neural network model training system 10000 shown according to an example of the present invention can process a specific neural network model in a performance evaluation device (hereinafter referred to as "performance evaluation device") 2000 for multiple neural network models, and provide multiple performance evaluation results of the neural network model performance evaluation device 2000 online to the user.

[0140] User device 1000 may refer to a user-accessible device and may be used to obtain multiple performance evaluation results from neural network model performance evaluation device 2000 that processes neural network models. User device 1000 may include (among other devices) smartphones, tablets, personal computers, laptops, or other devices capable of connecting to server 3000. User device 1000 provides a user interface for accessing information related to neural network models. In this case, user device 1000 may also be considered an edge device.

[0141] In addition, the user device 1000 may include a neural processing unit, and the neural network model performance evaluation device 2000 can provide a neural network model for execution on the neural processing unit of the user device 1000.

[0142] The method by which user device 1000 connects to server 3000 may include connecting via web services, FTP servers, cloud servers, or applications on user device 1000. However, such connection methods are not limited to these options and may employ various known communication technologies.

[0143] Users can use various communication technologies to transmit information about neural network models to server 3000. Specifically, users can upload at least one specific neural network model and at least one evaluation dataset of the model to server 3000 via user device 1000 to evaluate the performance of neural processing devices that users are interested in purchasing.

[0144] The aforementioned specific evaluation dataset may refer to the dataset input to the neural network model performance evaluation device 2000 used to evaluate the performance of the neural network model.

[0145] User device 1000 can receive the performance evaluation results of the neural network model from neural network model performance evaluation device 2000, and can output the performance evaluation results provided by neural network model performance evaluation device 2000.

[0146] User device 1000 can be any type of terminal capable of uploading information about the neural network model to be evaluated from neural network model training system 10000 to server 3000. Furthermore, user device 1000 can be any type of terminal capable of uploading test datasets used to evaluate the neural network model to neural network model training system 10000. Moreover, user device 1000 can be any type of terminal capable of uploading training datasets used to retrain the neural network model to neural network model training system 10000. In other words, user device 1000 can be a data transmission unit for performance evaluation of neural network models or a result receiving unit for performance evaluation of neural network models.

[0147] For this purpose, user device 1000 may include at least one of the following components: processor 1120, display device 1140, user interface 1160, network interface 1180, and memory 1200. Display device 1140 may present multiple options for selecting one or more neural processing units. Furthermore, display device 1140 may present multiple options for compiling neural network models. Memory 1200 may store executable software that allows processor 1120 to access server 3000. Memory 1200 may also store multiple neural network models and multiple performance evaluation datasets for transmission via server 3000 to performance evaluation device 2000. User interface 1160 may include multiple input devices such as a keyboard and mouse. User interface 1160 facilitates user input for selecting one or more neural processing units to process neural network models and for selecting compilation options related to compiling neural network models. Network interface 1180 may be a hardware component (such as a network adapter) enabling user device 1000 to communicate with server 3000 via a network.

[0148] The neural network model performance evaluation device 2000 may include at least one neural processing unit for processing neural network models received from the user device 1000 via the server 3000. The neural network model performance evaluation device 2000 may also compile and evaluate multiple neural network models. The performance of the processed neural network models can be determined, and multiple performance results can be reported to the user device 1000 via the server 3000.

[0149] The neural network model performance evaluation device 2000 may include a system containing one or more general-purpose computers, laptops, cloud computers, cloud servers, or similar programs for executing determinations of information related to neural processing devices. The neural network model performance evaluation device 2000 may obtain at least one specific neural network model for evaluating the performance of a neural processing device and a specific set of evaluation data input to the neural network model from a server 3000, compile and process the neural network model, and provide multiple performance evaluation results.

[0150] Server 3000 is a computing device that communicates with user device 1000 to manage access to neural network model performance evaluation device 2000. The server may include processor 3120, network interface 3160, and memory 3180. Network interface 3160 enables server 3000 to communicate with user device 1000 and neural network model performance evaluation device 2000 via a network. Memory 3180 may store instructions executable by processor 3120 to perform one or more of the following tasks: (i) managing user accounts; (ii) authorizing and enabling users to access neural network model processing devices for evaluating one or more neural processing devices; (iii) receiving multiple user inputs (including selected neural network models, evaluation datasets, selected neural processing devices for evaluation, and compilation options); (iv) encrypting and storing data received from users; (v) transmitting neural network models and user processor selection information to neural network model processing devices via a network; and (vi) transmitting multiple performance reports and suggestions for selected neural processing devices to user device 1000 via a network. Server 3000 can also be configured to provide various additional services as needed.

[0151] In detail, multiple neural network models, training datasets, evaluation datasets, and similar assets developed by multiple users constitute the intellectual property of these users and require stringent security measures. In many cases, training datasets used to develop multiple commercially viable AI services can be worth hundreds of thousands to hundreds of millions of dollars. Hereafter, such assets, including multiple neural network models, training datasets, and evaluation datasets developed by users, are collectively referred to as user data. To ensure the security of user data uploaded to the neural network model training system 10,000, the system can integrate multiple protection measures. These may include user account login verification, data encryption, multiple differential privacy technologies, and data masking to protect the data itself. Furthermore, mechanisms such as access control and audit logs can be implemented to regulate and monitor model access and usage.

[0152] Data encryption can be used to protect user data by converting information into an encoded format that can only be deciphered by authorized personnel to ensure confidentiality. Differential privacy technologies use statistical methods to reduce the sensitivity of user data, especially when the data contains personal information, thereby protecting individual privacy. Data masking technologies protect user information by obfuscating and hiding sensitive information, such as replacing parts of the data with multiple pseudonyms or symbols.

[0153] Access to user data can be restricted through access control mechanisms to ensure that only authorized accounts are allowed to enter. Audit logs can be implemented to record the accounts that access user data, maintaining detailed logs of system interactions with user data. These logs can track model accessors and timestamps, helping to detect anomalous activity and strengthen security oversight.

[0154] In addition, when uploading training and / or evaluation datasets, multiple users may be required to sign a separate user data protection agreement. This ensures that users' neural network models, training datasets, and evaluation datasets are protected.

[0155] The neural network model performance evaluation system according to an example of the present invention can be configured to utilize a neural network model training system based on multiple heterogeneous processing devices. A detailed description of the neural network model training system based on a heterogeneous processing device system will be provided in subsequent paragraphs.

[0156] Reference Figure 5 The neural network model performance evaluation device 2000 will be described. Figure 5 This is a block diagram illustrating a neural network model performance evaluation device according to an example of the present invention. The neural network model performance evaluation device 2000 may include a general-purpose graphics processor 100, at least one neural processing unit 200, and memory 300. The components of the neural network model performance evaluation device 2000 can communicate with each other via one or more communication buses or signal lines. The neural network model performance evaluation device 2000 may operate on a specific operating system (OS). For example, the operating system may be Microsoft Windows, MacOS, Linux (such as Ubuntu, Fedora, Debian, CentOS, Arch Linux), Unix, iOS, or Android, etc.

[0157] According to an example of the present invention, a neural network model performance evaluation device 2000 may include at least one general-purpose graphics processor 100 and at least one neural processing unit 200-1. The general-purpose graphics processor 100 may be configured to execute a plurality of instructions. The neural processing unit 200-1 may be configured to execute a plurality of instructions.

[0158] First, the general-purpose graphics processor 100 series is hardware that performs complex computational tasks and graphics and image processing. The number of general-purpose graphics processors 100 is shown as one, but it is not limited to this, and multiple general-purpose graphics processors connected by cloud GPUs, NVLink, or NVSwitch can be used.

[0159] The general-purpose graphics processing unit 100 may contain multiple cores, and these cores can process multiple tasks in parallel. Therefore, the general-purpose graphics processing unit 100 can perform large-scale data processing tasks such as scientific computing and deep learning.

[0160] Specifically, the general-purpose graphics processing unit 100 can be used to train deep learning and machine learning models on multiple large datasets. While deep learning models typically have a large number of parameters and require considerable training time, the general-purpose graphics processing unit 100 can execute these operations in parallel, thereby accelerating the learning process of the neural network model. The general-purpose graphics processing unit 100 can be effectively utilized when a user selects a specific neural processing unit from one or more neural processing units 200 and performs training or retraining of the neural network model using various compilation options. A suitable graphics processing device can be selected based on the user's needs, and the selected device can perform retraining of the neural network model according to multiple specific compilation options.

[0161] In other words, the general-purpose graphics processing unit 100 can receive training datasets uploaded by the user device 1000 via the server 3000 and use the datasets to train or retrain a neural network model. For example, during the retraining process, the general-purpose graphics processing unit 100 can simultaneously apply pruning and / or quantization algorithms to the neural network model to retrain it in conjunction with the neural network model training dataset. This retraining can cover multiple cycles, and the general-purpose graphics processing unit 100 can execute several to hundreds of cycles in the process.

[0162] The retraining option is a technique to compensate for the decrease in inference accuracy caused by the application of various optimization options. For example, when applying quantization, pruning, or model compression options, the accuracy of a neural network model with one or more neural processing units 200 inferences may decrease. In this case, the system can provide an online retraining option to retrain the pruned, quantized, and / or compressed neural network model, thereby restoring its inference accuracy.

[0163] Meanwhile, the general-purpose graphics processor 100 may include one or more arithmetic processors for executing instructions stored in memory 300. Furthermore, the central processing unit 700 in the performance evaluation device 2000 may load and execute the compiler 310 stored in memory 300. The machine code generated by the central processing unit 700 may be loaded and executed by the general-purpose graphics processor 100 for training or retraining neural network models.

[0164] One or more neural processing units 200 can be implemented as a neural processing unit farm, consisting of various families of neural processing units with different performance levels and price points provided by a specific company. This neural processing unit farm can be provided online for convenient evaluation of the performance of user-developed neural network models, or it can be provided in the form of cloud-based neural processing units. However, the examples described in this invention are not limited to neural processing unit farms, and there is no limitation on the number of individual NPUs (e.g., 200-1) contained within at least one neural processing unit 200.

[0165] One or more neural processing units 200 may contain various types of neural processing units.

[0166] More specifically, one or more neural processing units 200 can be classified based on computing power.

[0167] For example, the first NPU could be an NPU used in smart closed-circuit television. The first NPU could feature ultra-low power consumption, low-order inference processing capabilities (e.g., 5 TOPS processing power), extremely small semiconductor package size, and extremely low cost. Due to performance limitations, the first NPU may not support specific neural network (NN) models that include specific computations and require high memory bandwidth. For example, the first NPU could be the "DX-V1" provided by DEEPX Co., Ltd. in Seongnam, Gyeonggi Province, South Korea, which can compute NN models such as ResNet, MobileNet v1 / v2, SSD, YOLOv5, and YOLOv7.

[0168] For example, the second NPU could be an NPU used for image recognition, object detection, and object tracking in robots. The second NPU could feature low power consumption, moderate inference processing power (e.g., 16 TOPS), small semiconductor package size, and low cost. The second NPU may not support specific NN models involving high memory bandwidth. For example, the second NPU could be model "DX-V2," also provided by DEEPX Ltd., and capable of operating NN models such as ResNet, Mobilenet v1 / v2, SSD, YOLOv5, and YOLOv7.

[0169] For example, a third NPU could be used for image recognition, object detection, object tracking, and generative AI services in autonomous vehicles. A third NPU could feature low power consumption, high-order inference processing capabilities (e.g., 25 TOPS), a medium semiconductor package size, and a medium price. For instance, a third NPU could be model "DX-M1," also provided by DEEPX Ltd., and capable of running ResNet, MobileNet v1 / v2 / v3, SSD, EfficientNet, EfficientDet, YOLOv5, YOLOv7, YOLOv8, DeepLabv3, PIDNet, ViT, generative adversarial networks, and stable diffusion neural network models.

[0170] For example, the fourth NPU could be used in CCTV monitoring rooms, control centers, large language models, and generative AI services. The fourth NPU could feature low power consumption, high-order inference processing capabilities (e.g., 400 TOPS), large semiconductor package size, and high cost. For instance, the fourth NPU could be model "DX-H1," also provided by DEEPX Ltd., and capable of processing NN models such as ResNet, Mobilenet v1 / v2, SSD, YOLOv5, YOLOv7, YOLOv8, DeepLabv3, PIDNet, ViT, generative adversarial networks, stable diffusion, and large LLMs.

[0171] In other words, each NPU can have different computing power, different semiconductor chip die size, and different power consumption characteristics. However, the types of multiple neural processing units 200 are not limited to this, and can be classified according to various classification criteria.

[0172] Simultaneously, one or more neural processing units 200 may receive an evaluation dataset uploaded by the user device 1000 via the server 3000 and feed it into the compiled neural network model to perform performance evaluation. The evaluation dataset refers to the set of evaluation data input to the performance evaluation of the neural network model performance evaluation device 2000.

[0173] Based on the selected compilation options, the input neural network model can be compiled, and the resulting machine code, along with the evaluation dataset, can be transferred to the selected neural processing unit 200-1 within the NPU resource pool for processing. The compilation options can be configured according to user preferences and can include multiple variations, allowing users to specify different options to meet their specific needs. In other words, the neural network model training system 10000 based on heterogeneous processing devices can provide user-customized compilation options, rather than relying solely on predefined options, thereby satisfying the user's detailed requirements.

[0174] One or more compilation options may include at least one of the following: pruning algorithm, quantization algorithm, parameter refinement algorithm, outlier mitigation algorithm, model compression algorithm, knowledge distillation algorithm, retraining algorithm, and AI-based optimization algorithm.

[0175] Alternatively, the compilation options can be configured to select one of the available default options. Furthermore, performance evaluation results, i.e., processing performance, can be generated and reported for the neural processing unit 200-1 processing the compiled neural network model. Performance evaluation reports can be stored in the user's account or transmitted to the user's email address. However, performance evaluation results may also be provided to the user in various other ways, and are not limited to this. Such performance evaluation results are also classified as user data and are subject to user data security policies.

[0176] The parameters of processing performance can include the NPU's temperature profile, power consumption (watts), trillion operations per watt per second (TOPS / W), frames per second (FPS), inferences per second (IPS), accuracy, etc.

[0177] Meanwhile, the general-purpose graphics processor 100 and at least one neural processing unit 200 can be implemented in the form of integrated circuits (ICs), such as a system-on-a-chip (SoC) that integrates various computing circuits, or a printed circuit board on which integrated circuits are mounted.

[0178] Memory 300 can store various software, including but not limited to compiler 310, storage module 320 and reporting program 330.

[0179] In addition, memory 300 may include volatile or non-volatile recording media capable of storing various data, instructions and information.

[0180] For example, memory 300 may contain at least one of the following types of storage media: flash memory, hard disk, multimedia card micro, card-type memory (such as SD or XD memory), RAM, SRAM, ROM, EEPROM, PROM, network storage, cloud and blockchain database.

[0181] As previously described, the central processing unit 700 in the neural network model performance evaluation device 2000 can load and execute the compiler 310 stored in the memory 300. The compiler 310 can be implemented as a semiconductor circuit or as software stored in the memory 300 and executed by the CPU 700. However, the present invention is not limited to these embodiments. In some embodiments, the compiler 310 can be software executed by a general-purpose graphics processor 100.

[0182] The compiler 310 can translate a specific neural network model into machine code executable by one or more neural processing units 200. In other words, the compiler 310 can generate machine code suitable for each neural processing unit 200 with different characteristics. Accordingly, the compiler 310 can generate machine code adapted for execution by a selected neural processing unit 200-1 from one or more neural processing units 200. This machine code can also be called binary code. The compiler 310 can specifically generate machine code for the neural network model to evaluate the performance of the selected neural processing unit 200-1 from one or more neural processing units 200.

[0183] The compiler 310 provides a variety of compilation options, which can be displayed as a user interface (UI) on the screen of the user device 1000, allowing the user to select the desired compilation options. These compilation options can be differentiated for each NPU selected for performance evaluation, enabling the generation of machine code tailored to the selected neural network model.

[0184] Since multiple compilation options can vary depending on the category of one or more neural processing units 200, the machine code compiled from the same neural network model can also differ depending on the category of one or more neural processing units 200. In other words, independent machine code can be generated for each selected compilation option. The storage module 320 can store various types of data used by the neural network model performance evaluation device 2000. Specifically, the storage module 320 can store one or more of the following: the machine code of the compiled neural network model, one or more training datasets, one or more evaluation datasets, performance evaluation results, and output data generated by one or more neural processing units 200.

[0185] The reporting procedure 330 can process the compiled neural network model to report the results of the performance evaluation. That is, the reporting procedure 330 can first determine whether the compiled neural network model can be processed by one or more neural processing units 200.

[0186] If a compiled neural network model cannot be processed by one or more neural processing units 200, the reporting program 330 may report the specific layer in the multi-layer structure of the neural network model that cannot be processed, or the specific operation that cannot be performed.

[0187] If the compiled neural network model can be executed by a specific NPU in one or more neural processing units 200, the reporting program 330 can report the processing performance of at least one neural processing unit 200.

[0188] The processing performance parameters of an NPU may include one or more of the following: temperature profile, power consumption (watts), trillion operations per second (TOPS / W), frames per second (FPS), inferences per second (IPS), accuracy, etc.

[0189] Temperature profiles refer to the temperature change data measured over time during NPU operation.

[0190] Power consumption refers to the electrical data measured when the NPU is operating. Since power consumption depends on the computational load of the user-developed NN model, the user's NN model needs to be provided for deployment to obtain accurate power consumption measurements.

[0191] Trillions of operations per watt per second (TOPS / W) is a metric for measuring the efficiency of AI accelerators, meaning the number of operations that can be performed per second per watt of power. TOPS / W serves as an energy efficiency indicator for one or more neural processing units 200, reflecting the amount of computation the hardware can perform per unit of power consumption.

[0192] Inferences per second (IPS) refers to the number of inference operations that one or more neural processing units 200 can perform in one second, and is used to represent the processing speed of one or more neural processing units 200. Inferences per second can also be referred to as frames per second (FPS).

[0193] Accuracy refers to the inference accuracy of one or more neural processing units 200, representing the percentage of correctly inferred samples out of the total sample size. It should be noted that the accuracy of one or more neural processing units 200 may differ from the inference accuracy of the graphics processing unit 100. This is because the neural network model parameters inferred by the graphics processing unit 100 are in floating-point format, while the neural network model parameters inferred by one or more neural processing units 200 are at least partially in integer format. Furthermore, various optimization algorithms can be selectively applied. Therefore, the values ​​obtained after various calculations of the neural network model parameters inferred by one or more neural processing units 200 may differ, resulting in an accuracy rate different from that of the graphics processing unit 100. The degree of this difference in inference accuracy can depend on the structure and parameter size characteristics of the neural network model, especially when the bit width of the quantization parameters is shorter, the more significant the degradation in inference accuracy caused by overquantization. For example, the quantization bit width can be from 2 bits to 16 bits. The same trend of degradation in inference accuracy caused by overpruning also applies.

[0194] More specifically, certain NPUs may have built-in dedicated circuitry supporting only multiply-accumulate (MAC) operations on integer parameters, while other NPUs may be configured with dedicated circuitry to handle only floating-point MAC operations. Some NPUs may have MAC operation circuitry capable of handling 32-bit parameter inputs, while others support MAC operations on 16-bit or 8-bit parameter inputs. Furthermore, certain NPUs may be equipped with MAC operation circuitry capable of handling parameter inputs of different bit widths. These differences can lead to different results when inference is performed on the general-purpose graphics processor 100 and the specific neural processing unit 200-1, even using the same evaluation dataset.

[0195] To further explain, a particular NPU may include circuitry designed to process activation function operations in an approximate computational manner, while other NPUs may include circuitry designed to process activation function operations based on a lookup table. These differences can cause different results when the same evaluation dataset is inferred on the general-purpose graphics processor 100 and the specific neural processing unit 200-1, respectively.

[0196] Simultaneously, the reporting program 330 can analyze the processing performance of the compiled neural network model under various compilation options and can recommend one of several compilation options. Furthermore, the reporting program 330 can also recommend specific categories of neural processing units 200 based on the performance parameters of different NPUs.

[0197] Simultaneously, the reporting program 330 can analyze the processing performance of the compiled neural network model for each compilation option, and then recommend the best compilation option. In addition, the reporting program 330 can also suggest suitable neural processing unit 200 categories based on the performance parameters of various NPUs.

[0198] Figure 6 This is a block diagram illustrating the configuration of at least one of a plurality of neural processing units according to an example of the present invention. One or more neural processing units 200 can be controlled by a controller. Specifically, before each of the neural processing units 200-1 to 200-N directly accesses, reads, and / or writes to memory 300, the controller can pre-execute a schedule of memory access operations and computational operations.

[0199] Reference Figure 5 and Figure 6Each neural processing unit 200-1 to 200-N may include a controller 10, a direct memory access (DMA) module 20, internal memory 30, a plurality of processing elements 40, and special function units 50. For ease of explanation, neural processing unit 200-1 is described below as a representative example of at least one neural processing unit 200. This is for illustrative purposes only and is equally applicable to all NPUs within one or more neural processing units 200.

[0200] The components of the neural processing unit 200-1 are distinguished based on their computational functions, and each component can be implemented using at least one of a substrate, a resistive element, and a transistor. Therefore, each component can constitute a semiconductor circuit containing a large number of interconnected transistors, some of which are difficult to visually identify or distinguish and can only be identified by their functional characteristics. Therefore, the functional units of the neural processing device 200-1 can be collectively referred to as circuit units.

[0201] The controller 10 can be configured to manage the operations required for neural network model computation by coordinating the direct memory access module 20, internal memory 30, multiple processing elements 40, and special function units 50. The controller 10 can be directly or indirectly coupled to these elements to enable communication. For example, the controller 10 can dynamically adjust the parameter configuration in the internal memory 30 according to its capacity. The controller 10 can also control the neural processing unit 200-1 by executing machine code (such as binary code) generated from the compiled neural network model. For example, the compiler can generate machine code defining the following operations based on the hardware characteristics of a specific neural processing unit 200-1 (such as the number of processing elements, memory capacity, functionality of special function units, presence of post-processing units, etc.): the order of reading and writing neural network model data, the processing order of neural network layers, the processing order of convolution or matrix multiplication operations, and the sequence of read and write operations for direct memory access. Therefore, the controller 10 executes control of the neural processing unit 200-1 according to the machine code, ensuring that the performance conforms to the hardware capabilities.

[0202] The controller 10 can acquire scheduling information to manage the job order of the neural network model executed by the neural processing unit 200-1. This scheduling is based on the directed acyclic graph (DAG) of the neural network model compiled by the compiler. The compiler can generate the job schedule based on the number of processing elements in the neural processing unit 200-1, the size of the internal memory 30, the parameter sizes of each layer of the neural network model, and similar parameters.

[0203] Based on this job schedule, the controller 10 can control the number of processing elements in each computational step and manage the parameter read and write operations in the internal memory 30 for each step. The compiler can determine the job schedule based on its understanding of the hardware architecture and performance characteristics of the neural processing unit 200-1. Furthermore, the compiler can determine the processing order of the required data based on the computational order of the neural network model, covering operations such as layering, convolution, and matrix multiplication. This ensures data locality and generates efficient machine code for execution.

[0204] In some examples, the neural processing unit 200-1 may include an embedded compiler. Based on the above configuration, the neural processing unit 200-1 can directly generate machine code from input files of various artificial intelligence software frame formats. Examples of such artificial intelligence software frameworks include TensorFlow, PyTorch, Keras, XGBoost, MXNet, Darknet, and ONNX.

[0205] The direct memory access module 20 enables the neural processing unit 200-1 to directly access, read from, and write to memory 300. The neural processing unit 200-1 can retrieve various data related to the neural network model from memory 300 through the direct memory access module 20. Memory 300 can be embedded within a system-on-chip (SoC) or implemented as a standalone memory device. Internal memory 30 located within the internal region of the neural processing unit 200-1 chip can serve as cache memory, storing or caching data processed within the internal region of the chip.

[0206] Internal memory 30 can retrieve and store at least a portion of the parameters used to compute the neural network model from memory 300. It can store all or part of the neural network model depending on the memory capacity allocated to each parameter and the data size of each layer of the model. Representative parameters processed in the neural network model may include attention, key-value cache, activation map, input feature map, output feature map, weights, and similar elements.

[0207] Specifically, internal memory 30 can retrieve and store parameters related to input data from memory 300. Furthermore, it can read and store parameters related to output data generated by the multiple processing elements 40. As detailed later, the parameters of the neural network model may include input parameters and weights. Input or output parameters read from or written to internal memory 30 may include activation parameters, feature map parameters, key-value cache parameters, attention parameters, and related data types.

[0208] Internal memory 30 may include one or more memory types, such as read-only memory, static random access memory (SRAM), dynamic random access memory (DRAM), resistive RAM, magneto-resistive RAM, phase-change RAM, ferroelectric RAM, flash memory, high-bandwidth memory, or similar technologies. In one example, internal memory 30 may be implemented as static random access memory (SRAM) to improve computational processing speed. Furthermore, internal memory 30 may be organized into one or more memory cells, such as embankments or similar structures. It may also include homogeneous or heterogeneous memory configurations.

[0209] The data (e.g., parameters of a neural network model) stored in the memory units of internal memory 30 is not fixed to a specific type (e.g., attention, key-value cache, activation map, input feature map, weights, or output feature map). Instead, it can dynamically change to other types according to computational needs. In other words, its utilization efficiency can be improved by adjusting the memory allocation within internal memory 30. Therefore, the amount of data allocated to various parameters stored in internal memory 30 can vary according to specific computational steps.

[0210] Multiple processing elements 40 may include an array of processing elements and / or an addition tree to perform multiply-add operations.

[0211] Each processing element 40 can receive and calculate the input feature map corresponding to the neural network input data and / or the core corresponding to the weights.

[0212] These processing elements can perform addition, multiplication, accumulation, and other operations required to process neural network models. To achieve this functionality, each processing element may contain multiply-accumulate operands, arithmetic logic operands, or similar elements.

[0213] For example, the processing element can receive multiple input feature maps and multiple weights, perform convolution or matrix multiplication operations, and generate an output feature map. Furthermore, the multiple processing elements 40 can also be referred to as artificial intelligence computing units.

[0214] In another example, the processing element can perform a general matrix multiplication operation or matrix multiplication operation on the input feature map and multiple weights to output an output feature map. More specifically, the processing element can multiply the input feature map in matrix form with the weight matrix, and then add the bias to the matrix to output an output feature map in matrix form. In particular, in neural processing units, matrix multiplication can be performed at high speed through parallel processing, thereby enabling efficient processing of matrix multiplication operations.

[0215] As another example, the processing element may include circuitry designed to process only integer parameters as input. In this case, multiple input parameters of the processing element can be converted into integers of a specific width and stored in internal memory 30. According to the above configuration, power consumption can be effectively reduced compared to processors that support floating-point operations, and it is easy to implement on a device.

[0216] Special function unit 50 can handle the number of activation functions to give the output feature map nonlinear characteristics.

[0217] Special function unit 50 can process various activation functions to introduce nonlinear characteristics into the output feature map. The activation functions processed by special function unit 50 include, but are not limited to, the SiLU function, the Softmax function, the sigmoid function, the hyperbolic tangent (tanh) function, the ReLU function, the LeakyReLU function, the Maxout function, or the ELU function, each of which produces a nonlinear output value relative to the input value.

[0218] On the other hand, supporting all activation functions in the neural processing unit 200-1 is technically difficult. Therefore, the neural processing unit 200-1 can approximate various activation functions using a piecewise linear function approximation algorithm and a piecewise linear function processing circuitry. These activation functions can be selectively applied after multiplication and addition operations. The operation value of the activated function can be called an activation map.

[0219] On the other hand, supporting all activation functions in the neural processing unit 200-1 presents technical challenges. Therefore, the neural processing unit 200-1 can approximate various activation functions using a piecewise linear function approximation algorithm and a piecewise linear function processing circuit. These activation functions can be selectively applied after multiplication and addition operations, and the values ​​of the operations on which the activation functions are applied are called activation parameters.

[0220] In addition, the special function unit 50 can be configured to include a floating-point multiplier circuit for performing decimal point operations.

[0221] In another embodiment, the special function unit 50 may include circuitry configured to communicate with the processing element and designed to receive integer parameters from the processing element. In this case, the special function unit 50 may further integrate dequantizer circuitry responsible for converting the integer parameters to floating-point parameters. The special function unit 50 may then be configured to perform activation function operations using multiple floating-point parameters. Furthermore, the special function unit 50 may include quantization circuitry designed to convert the floating-point parameters back to integer parameters after the activation function operation. According to this configuration, the special function unit 50 can process floating-point operations by dequantizing the integer parameters when floating-point operations are required, and then requantize the resulting parameters. In other words, according to an example of the invention, the neural processing unit may include a processing element circuitry configured to process integer parameters and a special function circuitry unit connected in series with the processing element. The circuitry for the special function unit may simultaneously include quantization circuitry and dequantization circuitry, and may be configured to perform activation function operations using floating-point parameters. This configuration enables the special function unit 50 to effectively interface with processing elements that only support integer parameters, allowing it to directly convert parameter types and perform processing without additional circuitry outside the neural processing unit.

[0222] Figure 7 This is a block diagram illustrating a neural network model training system based on a heterogeneous processing device in a neural network model performance evaluation apparatus according to an example of the present invention. (Refer to...) Figure 7 The general-purpose graphics processor 100 can read parameters of the neural network model and a training dataset from memory 300 to train the model. During the training process, when multiple weights are updated, multiple weights are transferred to a target device (such as neural processing unit 200-1) to evaluate the performance of the trained neural network model. In one or more embodiments, memory 300 may receive updated weights from general-purpose graphics processor 100 and transfer them to neural processing unit 200-1, or alternatively, general-purpose graphics processor 100 may directly transfer at least a portion of the updated weights to neural processing unit 200-1. Therefore, neural processing unit 200-1 can perform the evaluation of the trained neural network model using trained model parameters stored in memory 300 and a test dataset for model evaluation.

[0223] In other words, according to an example of the present invention, the training of the neural network model is performed by a general-purpose graphics processor 100, while the evaluation of the trained neural network model is handled by a neural processing unit 200-1. Here, by means of evaluation by the neural processing unit 200-1, the inference accuracy of the trained neural network model can be determined along with multiple updated weights.

[0224] As an evaluation result, the processing performance of the trained neural network model can be obtained. In this paper, processing performance can be represented by accuracy. However, this is only an example, and other metrics such as perplexity, processing speed, and / or power consumption can be used as processing performance metrics. The evaluation results (such as the inference results processed by neural processing unit 200-1) can be stored in memory 300 and transmitted to general-purpose graphics processor 100. The baseline truth value of the evaluation dataset (such as the answer to the evaluation dataset) can be provided by the neural network model training system 10000. For example, the baseline truth value of the evaluation dataset can be stored in memory 300. The inference results are compared with the baseline truth value of the evaluation dataset. Therefore, by receiving the inference results output by neural processing unit 200-1 and comparing them with the standard answer of the evaluation dataset, the weighted accuracy of the inference by neural processing unit 200-1 trained by general-purpose graphics processor 100 or CPU 700 can be determined.

[0225] Subsequently, the general-purpose graphics processor 100 can train or retrain on an epoch-by basis, continuously updating the weight values ​​stored in memory 300 based on the evaluated weight accuracy. Multiple weight updates can be based on methods to improve accuracy. Whenever weights are updated through retraining, they are repeatedly transmitted to neural processing unit 200-1 in a repetitive manner (e.g., on an epoch-by basis) to improve the inference accuracy of the weights trained specifically for neural processing unit 200-1.

[0226] Memory 300 can be configured to store updated weights for at least one cycle. Each weight corresponding to each cycle can be stored in memory 300 for comparison purposes. Furthermore, each accuracy result corresponding to each cycle can be stored in memory 300 for comparison purposes. Memory 300 can be configured to store updated weights for multiple cycles. Memory 300 can be configured to store updated weights processed on a first processing device (such as a GPGPU) for at least one cycle, and corresponding evaluation results processed on a second processing device (such as an NPU).

[0227] The memory 300 can be configured to store the updated weights processed on the first processing device for multiple cycles and the corresponding evaluation results processed on the second processing device.

[0228] As previously described, the training and evaluation datasets used for training or retraining can be stored in memory 300. The neural network model performance evaluation device 2000 can acquire (or store) the training and evaluation datasets when the user device 1000 uploads these datasets to the server 3000. In another embodiment of the invention, the neural processing unit 200-1 can transmit output processing performance (e.g., inference accuracy) to the general-purpose graphics processor 100. In this case, the general-purpose graphics processor 100 will receive feedback on the processing performance of its trained or retrained neural network model.

[0229] As described above, according to the present invention, the general-purpose graphics processor 100 only transmits the weights updated through training or retraining of the neural network model to the neural processing unit 200-1, and the evaluation of the trained or retrained model is performed by the neural processing unit 200-1. This method allows the general-purpose graphics processor 100 to accelerate the evaluation process because it does not need to allocate resources for model evaluation, thereby reducing the time required to perform this task.

[0230] Furthermore, when the target device of the neural network model is the neural processing unit 200-1, the inference accuracy can be approximately at its highest when the model is executed directly on the target device. As mentioned earlier, when the general-purpose graphics processor 100 is used to simulate the target device, it may not perfectly replicate the behavior of the target device. Moreover, even when processing the same algorithm, the computational circuit configurations of the general-purpose graphics processor 100 and the neural processing unit 200-1 may differ. Therefore, whenever the neural network model is compiled to the neural processing unit 200-1, the computational results may differ due to the differences in computational circuitry and the parameter formats used by the model.

[0231] Furthermore, training a neural network model on a general-purpose graphics processing unit (GPGPU) 100 may not fully reflect the differences in computational circuitry between the GPGPU 100 and the neural processing unit 200-1. In other words, after the GPGPU trains the neural network model by updating its weights, the target device uses the updated weights and evaluates its inference accuracy. The inference accuracy determined by the target device can be fed back to the GPGPU to update multiple weights of the neural network model again. By using the accuracy evaluated by the target device as feedback and repeatedly updating the weights via the GPGPU, the neural network model can become specifically adapted for the target device. Therefore, the inference accuracy of the neural network model on the target device can be improved.

[0232] Figure 8 This is a block diagram illustrating the configuration of a compiler for a neural network model performance evaluation device according to an example of the present invention. (Refer to...) Figure 8The compiler 310 of the neural network model performance evaluation device 2000 can compile the neural network model into machine code based on a variety of compilation options. The compiler 310 of the neural network model performance evaluation device 2000 may include an optimization module 311, a verification module 312, and a code generation module 313. The optimization module 311 of the compiler 310 may instruct the general-purpose graphics processor 100 to modify or retrain the weights of the neural network model for deployment on the neural processing unit 200-1.

[0233] The compiler 310 can obtain structural information of a target NPU selected from one or more neural processing units 200. The structural information of the NPU may include one or more of the following: the capacity of the NPU's internal memory, the hierarchical structure of the internal memory, the number of processing units, information on special function units, etc. The compiler 310 can determine the processing order of each layer based on the structural information of the NPU and the graph information of the neural network model. In this document, the target device (e.g., a specific NPU) is configured to execute compiled machine code, which can be adapted to the computational circuitry of the target device.

[0234] The optimization module 311 is responsible for improving the neural network model represented by a directed acyclic graph (DAG) for a target NPU selected from one or more neural processing units 200. The user can select at least one option from the various options provided by the optimization module 311.

[0235] For example, optimization module 311 may provide an option to convert parameters into an integer format with a specific bit width. The specific bit width may range from 2 bits to 16 bits. Therefore, optimization module 311 can convert a neural network model based on floating-point parameters into a model based on integer parameters. In this case, one or more neural processing units 200 are designed to process integer parameters. Furthermore, optimization module 311 can convert a neural network model based on a nonlinear function into a model based on a piecewise linear function approximation. At least one neural processing unit 200 may be designed to process this piecewise linear function approximation.

[0236] For example, piecewise linear function approximation can be performed in several ways, such as dividing the entire interval into segments and approximating each segment with a linear function, dividing a portion of the interval into segments and approximating each segment with a linear function, or using a single linear function to approximate the entire interval.

[0237] In other words, the optimization module 311 can apply various optimization algorithms to reduce the size of parameters (such as weights and feature maps) in the neural network model so that it can be executed on the neural processing unit 200-1 and produce more accurate results. As part of this process, the optimization module 311 can mitigate the accuracy degradation of the optimized neural network model through various retraining algorithms.

[0238] The verification module 312 can perform a verification process to determine whether a customer's neural network model can be executed on one or more neural processing units 200. The verification module 312 can analyze the structure of the optimized neural network model and check whether the operators of each layer are supported by the hardware of one or more neural processing units 200. If the model is found to be infeasible, a separate error report file can be generated and provided to the user.

[0239] The code generation module 313 can modify the neural network model that the verification module 312 determines is runnable and that is processed by the optimization module 311, and generate machine code to execute the updated neural network model on an NPU selected from at least one neural processing unit 200. The generated machine code can be provided to the corresponding target NPU to perform performance evaluation.

[0240] Furthermore, the code generation module 313 can offload some operations of the neural network model that the verification module 312 determines to be inoperable to a heterogeneous processor (e.g., DSP, GPGPU, CPU, etc.) for execution. For example, the first machine code corresponding to the first neural network model can be generated by the first NPU in one or more neural processing units 200. The second machine code corresponding to the first neural network model can be generated by the second NPU in one or more neural processing units 200. The third machine code corresponding to the first neural network model can be generated by the third NPU in one or more neural processing units 200. The fourth machine code corresponding to the first neural network model can be generated by the fourth NPU in one or more neural processing units 200.

[0241] Figure 9 This is a block diagram illustrating the configuration of an optimization module of a neural network model performance evaluation device according to an example of the present invention. The optimization module 311 can modify the neural network model according to multiple compilation options. More specifically, the optimization module 311 can set compilation options based on the hardware information of the neural processing unit 200-1. Furthermore, the optimization module 311 can set multiple compilation options considering the parameter characteristics of the neural network model (e.g., weight parameter size, feature map size, etc.) and the characteristics of inference accuracy degradation.

[0242] The multiple compilation options set using optimization module 311 can be at least one of the following: pruning options, quantization options, model compression options, knowledge distillation options, outlier mitigation options, parameter refinement options, and retraining options.

[0243] Enabling pruning options provides techniques for reducing computation in neural network models. Pruning algorithms can replace tiny near-zero values ​​in one or more layers of weights in a neural network model with zero values, thereby sparsifying the weights. When pruning options are enabled, one or more neural processing units 200 can skip multiplication operations associated with zero-value weights to accelerate convolution operations, reduce power consumption, and reduce the parameter size in the machine code of the neural network model. Pruning specific weight parameters to zero is equivalent to breaking connections in the neurons corresponding to the weight data in the neural network. Pruning options may include: a first pruning option based on numerical values ​​(removing smaller weights) or a second pruning option based on percentages (removing a specific proportion of the smallest weight).

[0244] Enabling quantization options provides a technique for reducing the parameter size of neural network models. Quantization algorithms selectively reduce the number of bits in the weights and feature maps of each layer of a neural network model. When quantization options reduce the number of bits in specific feature maps and specific weights, the overall parameter size of the neural network model's machine code can be reduced. For example, when quantization options are enabled, a 32-bit floating-point parameter can be converted to a 2-bit to 16-bit integer parameter.

[0245] The model compression option enables the application of multiple techniques to compress the weight parameters, feature map parameters, and other parameters of a neural network model. Model compression techniques can be implemented using compression techniques known in the relevant field. This reduces the parameter size of the machine code in the neural network model using the model compression option. This model compression option can be provided to an NPU that includes a decompression decoder.

[0246] The knowledge distillation option is a technique that transfers knowledge acquired by a complex model (also known as a teacher model) to a smaller, simpler model (also known as a student model). In knowledge distillation algorithms, the teacher model typically has a larger parameter scale and higher accuracy than the student model. For example, in the retraining option described later, the accuracy of the student model can be improved through knowledge distillation options, where a neural network model trained with 32-bit floating-point parameters can be set as the teacher model, and neural network models with various optimization options can be set as student training models. The student model can be a model with at least one of the following options selected: pruning option, quantization option, model compression option, and retraining option.

[0247] The parameter refinement option is a technique that can be implemented in conjunction with the quantization option. To reduce errors that may occur during quantization and minimize memory bandwidth usage while maintaining the accuracy of the neural network model, this technique optimizes the parameters required for the quantization process. Through the parameter refinement option, the system will calculate the most suitable scaling factor and offset value for the quantization process of the floating-point parameters of the neural network model.

[0248] Outlier mitigation options are techniques that can be implemented in conjunction with quantization options. Outliers in the input values ​​and / or weights of a neural network model may exist, deviating from the actual data distribution. These outliers can amplify errors during quantization. To achieve effective quantization, appropriate compensation for outliers is necessary. Based on the outlier mitigation options, the system uses a dedicated outlier adjustment parameter to calibrate outliers in the input and weight parameters before the multiply-accumulate (MAC) operation.

[0249] The retraining option can compensate for the decrease in inference accuracy when one or more optimization options are applied. For example, the accuracy of the neural network model used for NPU inference may decrease when quantization, pruning, or model compression options are implemented. In this case, the system provides a retraining function to retrain the pruned, quantized, and / or compressed neural network model to restore its inference accuracy. This technique effectively resolves the conflict between model optimization and accuracy preservation, ensuring that the optimized model maintains ideal inference performance during actual deployment.

[0250] Specifically, retraining options may include transfer learning options, pruning perception retraining options, quantization perception retraining options, quantization perception self-distillation options, and similar options.

[0251] Enabling the quantization-aware retraining (QAT) option integrates the quantization process into the retraining phase of the neural network model, where the model fine-tunes its weights to reflect quantization errors. The quantization-aware retraining algorithm can include modifications to the loss function, gradient calculation, and optimization algorithm. The quantization-aware retraining option compensates for quantization errors by first quantizing the trained neural network model and then performing fine-tuning retraining in a manner that minimizes the loss caused by quantization.

[0252] The quantization-aware self-distillation option aims to perform QAT while avoiding underfitting during retraining. This allows retraining to consider the loss between the inferred values ​​from the model computation and the labeled values ​​from the training data, while minimizing the loss between the inferred values ​​and the results of a simulated quantized model with the same parameters. In one example, according to the quantization-aware self-distillation option, when the difference between the inferred values ​​and actual results of the pre-trained model using 32-bit floating-point parameters is the first loss, and the difference between the inferred values ​​of the quantized simulation model and the inferred values ​​of the pre-trained model with the same parameters is the second loss, the pre-trained model can update its parameters to reduce the first loss during retraining. The parameters can also be updated to reduce the second loss during retraining of the quantized simulation model.

[0253] To mitigate the problems arising from applying quantization-aware training to augment pre-trained models using data (where regularization can be overdone and lead to overgeneralization), quantization-aware self-distillation can be performed. According to the quantization-aware self-distillation method, the difference between the quantization simulation inference and the pre-trained model inference using the same parameters reflects the minimization of accuracy loss due to over-regularization.

[0254] The pruning-aware retraining (PAT) option identifies and removes less important weights from the trained neural network model and then fine-tunes the active weights. Pruning criteria can include weight values, activation values, and sensitivity analysis. The pruning-aware retraining option can reduce the size of the neural network model, improve inference speed, and compensate for overfitting during retraining.

[0255] The option to enable transfer learning allows a neural network model to learn by transferring knowledge from one task to another related task. Transfer learning algorithms are particularly effective when there is insufficient data to begin with, or when training a neural network model from scratch would be computationally expensive.

[0256] Without limitations, the optimization module 311 can apply AI-based updates to the neural network model. The AI-based optimization algorithm can be a method of generating a lightweight neural network model by applying various algorithms from compilation options. This may include exploring the neural network model structure using AI-based reinforcement learning methods; or employing techniques not based on reduction methods (such as quantization algorithms, pruning algorithms, retraining algorithms, model compression algorithms), whereby the AI ​​integrated into the optimization module 311 autonomously executes the reduction process to obtain an improved reduction result.

[0257] Figure 10 The diagram illustrates a processing element that can be applied to one of a plurality of processing elements in an example of the present invention.

[0258] According to one embodiment of the present invention, a neural processing unit 200-1 may include a plurality of processing elements 110, a neural processing unit internal memory 120 configured to store a neural network model that can be inferred by the plurality of processing elements 110, and a neural processing unit controller 130 for managing the operation of the plurality of processing elements 110 and the neural processing unit internal memory 120. The plurality of processing elements 110 may be designed to perform multiply-accumulate operations and quantize the results of the multiply-accumulate operations before output. However, embodiments of the present invention are not limited thereto.

[0259] Reference Figure 10The processing element 40-1 may be configured to include a multiplier 41, an adder 42, an accumulator 43, and a bit quantization unit 44. However, embodiments of the present invention are not limited thereto, and the processing element 40-1 may be modified according to the operational characteristics of the target neural network model.

[0260] Multiplier 41 multiplies input (N)-bit data with (M)-bit data, and outputs the result as (N+M)-bit data, where N and M are integers greater than zero. The first input terminal receiving (N)-bit data can be configured to receive parameters with variable characteristics (such as multiple activation parameters), and the second input terminal receiving (M)-bit data can be configured to receive parameters with constant characteristics (such as multiple weight parameters). However, the input data to multiplier 41 is not limited to constant parameters and variable parameters.

[0261] For example, according to an embodiment of the present invention, the input parameters of the processing element 40-1 can be reused based on the characteristics of constant parameters and variable parameters, which can improve the computational efficiency of the neural processing unit 200-1.

[0262] Here, parameters with variable characteristics refer to parameters stored at specific memory addresses whose values ​​can be dynamically updated as the input parameters change. For example, the activation parameters of each layer can represent the multiplicative and additive values ​​of the weight parameters of the integrated neural network model. When the neural network model is used for object detection in video data, the activation parameters of each layer will change frame by frame with the input image.

[0263] In this paper, parameters with constant properties refer to parameters stored at a specified memory address whose values ​​do not change as the input parameters are updated. For example, learned weight parameters can serve as unique inference criteria for neural network models. These learned weight parameters remain unchanged even when used for tasks such as object detection in video data.

[0264] In other words, multiplier 41 can be configured to receive a variable parameter and a constant parameter as input. Specifically, the variable parameter provided to the first input can represent the activation parameter of a layer in the neural network model. The activation parameter can correspond to the activation parameter of the input layer, the cumulative parameter of the hidden layer, or the cumulative parameter of the output layer. Meanwhile, the constant parameter provided to the second input can represent the weight parameter of the neural network model.

[0265] The controller 10 can be configured to improve memory reuse by taking into account the characteristics of multiple constant parameters.

[0266] Multiple variable parameters can be multiple operation values ​​of each layer, and the controller 10 can identify multiple reusable variable parameters based on the machine code of the compiled neural network model, and control the internal memory 30 to reuse the memory.

[0267] The multiple constant parameters can be multiple weight parameters of each layer, and the controller 10 can identify the multiple constant parameters that are repeatedly used based on the structural data of the neural network model or the local information of the neural network data, and control the internal memory 30 to reuse the multiple parameters stored in the memory.

[0268] In other words, controller 10 can identify multiple reusable variable parameters and multiple constant parameters based on the machine code of the compiled neural network model. Therefore, controller 10 can be configured to control internal memory 30 to reuse multiple parameters stored in internal memory 30.

[0269] Processing element 40-1 can restrict the operation of multiplier 41. For example, when zero is input to one of the first and second input terminals of multiplier 41, processing element 40-1 knows that the result is zero even if no operation is performed, so multiplier 41 may not perform the operation.

[0270] For example, when zero is input to either the first or second input of the multiplier 41, the multiplier 41 can be configured to operate in a zero-skip mode.

[0271] The number of bits for each parameter input to the first and second input terminals can be determined based on the quantization results of the activation parameters and weight parameters of each layer in the neural network model. For example, the activation parameters of the first layer can be quantized to 5 bits, while the weight parameters of the first layer can be quantized to 7 bits. In this case, the first input terminal can be configured to receive 5-bit parameters, and the second input terminal can be configured to receive 7-bit parameters. Therefore, the number of bits for the parameters input to each input terminal can be different.

[0272] Processing element 40-1 can be designed to receive quantized information of parameters input to each input terminal. Furthermore, the locality information of the neural network data can include quantized details of the input and output parameters of processing element 40-1.

[0273] The neural processing unit 200-1 can be configured to control the real-time conversion of the quantization bit width when the quantization parameters stored in the internal memory 30 are input to the processing element 40-1. Specifically, different levels can have different quantization bit widths, and the processing element 40-1 can be designed to generate the input parameters by performing real-time bit width conversion. This conversion can be performed based on bit count information provided by the neural processing unit 200-1, ensuring that the number of bits in the input parameters is properly adjusted during the conversion process.

[0274] Accumulator 43 uses adder 42 to perform L-fold cyclic accumulation to sum the result of multiplier 41 with the result of accumulator 43. Therefore, the bit width of the data at the input and output terminals of accumulator 43 can be represented as (N+M+log2(L)) bits (where L is a positive integer).

[0275] After the accumulator 43 completes the accumulation operation, the accumulator 43 can receive an initialization reset signal to initialize the data stored inside the accumulator 43 to zero. However, the examples according to the present invention are not limited to this.

[0276] Bit quantization unit 44 can be configured to reduce the bit width of the data output from accumulator 43. This bit quantization unit 44 can operate under the control of controller 10. The bit width of the quantized data can be output as (X) bits (where X is a positive integer). With this configuration, processing element 40-1 is designed to perform multiply-accumulate operations and to quantize and output the result of the multiply-accumulate operation. In particular, this quantization has the added benefit of reducing power consumption as the number of loops L increases. Low power consumption further helps to reduce heat generation in edge devices. In addition, minimizing heat generation has a key impact on reducing the likelihood of malfunction of neural processing unit 200-1 due to temperature rise.

[0277] The output parameter (X) bits from bit quantization unit 44 can represent activation parameters for the next layer or input parameters for convolution operations (or matrix multiplication). If the neural network model is quantized, bit quantization unit 44 can directly receive quantization information from the neural network model. Alternatively, in an unrestricted manner, controller 10 can be configured to analyze the neural network model to extract the necessary quantization information. Therefore, the output parameter (X) bits can be converted to the corresponding number of bits based on the size of the quantization parameters. The output parameter (X) bits from bit quantization unit 44 can then be stored in internal memory 30 with a quantized value of appropriate bit width.

[0278] According to one embodiment of the present invention, the processing element 40-1 of the neural processing unit 200-1 may include a multiplier 41, an adder 42, an accumulator 43, and a bit quantization unit 44. The bit quantization unit 44 can reduce the (N+M+log2(L)) bit data output from the accumulator 43 by the processing element 40-1 to (X) bits. The controller 10 can control the bit quantization unit 44 to reduce the number of bits in the output data from the least significant bit to the most significant bit according to a predetermined number of bits. Reducing the number of bits in the output data can have the benefits of reducing power consumption, computation, and memory usage. However, if the number of bits is lower than a certain length, the inference accuracy of the neural network model can drop sharply. Therefore, the quantization level (i.e., the reduction of the number of bits in the output data) can be determined by comparing the degree of reduction in power consumption, computation, and memory usage with the magnitude of the decrease in the inference accuracy of the neural network model. Quantization levels can also be determined by assessing the target inference accuracy used in a neural network model and progressively testing lower bit widths. Quantization levels can be determined independently for each layer of the neural network model.

[0279] By using the aforementioned processing element 40-1, by adjusting the number of bits of the (N) bit parameter and (M) bit parameter of the multiplier 41, and by reducing the number of bits of the operation value (X) bit through the bit quantization unit 44, the multiplication-addition instruction cycle can be increased while power consumption is reduced, and the convolution operation (or matrix multiplication operation) of the neural network model can be made more efficient.

[0280] Figure 11 This is an example flowchart illustrating a neural network model training method based on a heterogeneous processing device according to an example of the present invention. (Refer to...) Figure 11 The general-purpose graphics processor 100 performs the operation of training or retraining the neural network model based on the training data set (step S110).

[0281] For this process, the user can upload the training dataset to the server 3000 via the user device 1000. The user can then select, via the user device 1000, the type and number of at least one neural processing unit to be evaluated, and one or more compilation options for the neural network model to be processed by the selected neural processing unit. These selections will set the compilation options for the neural network model.

[0282] In step S110, the confirmed neural network model compilation options are verified and the model is compiled accordingly. The compiled machine code is then input to a neural processing unit selected from the available devices. As part of training or retraining the neural network model in step S110, multiple weights of the model are updated, and the updated weights are communicatively transmitted to the selected neural processing unit.

[0283] In some embodiments, the operation of training or retraining a neural network model can focus on lightweighting the weights of the neural network model. Various algorithms can be applied for this purpose, such as pruning, quantization, parameter refinement, off-point mitigation, model compression, knowledge distillation, retraining, and AI-based model optimization algorithms.

[0284] Next, in step S120, the selected neural processing unit 200-1 evaluates the trained or retrained neural network model using the evaluation dataset. For this process, the user can upload the evaluation dataset to the server 3000 via the user device 1000. During step S120, the selected neural processing unit 200-1 applies the weights updated since step S110 to the trained or retrained model to perform the evaluation task. The selected neural processing unit 200-1 then outputs the performance evaluation results in step S130. This result may include an accuracy indicator or other performance-based evaluation results.

[0285] Although not presented in Figure 11 In step S130, the selected neural processing unit 200-1 can report the performance evaluation results to the user device based on one or more predetermined methods. Furthermore, steps S110 to S130 can be executed iteratively to retrain the neural network model on a periodic basis.

[0286] In summary, as described in one embodiment, training a neural network model on a heterogeneous processing device includes training the neural network model on a first processing device using a training dataset, updating the weights of the neural network model to modify it for execution on a second processing device (hereinafter also referred to as a "lightweight model"), and then using the modified neural network model to perform inference on the second processing device. In each cycle, the first processing device compares the inference results of the current cycle with those of previous cycles based on an evaluation dataset, and selects the weight combination from the current and previous cycles that exhibits the best yield performance on the second processing device for deployment.

[0287] In one example, the first processing device may be a general-purpose graphics processor, and the second processing device may be a neural processing unit specializing in accelerating artificial intelligence inference but not configured with training operations.

[0288] For example, in conventional methods, the inference accuracy on a general-purpose graphics processing unit (GPU) may peak in one cycle (e.g., cycle 3). Therefore, multiple weights that generate the optimal inference accuracy during the dataset update in cycle 3 can be selected for deployment and execution on the neural processing unit. Conversely, according to several embodiments, weights that generate the optimal inference accuracy on the neural processing unit during another cycle (e.g., cycle 4) are selected for deployment on the neural processing unit even if the inference accuracy on the GPU is lower than in the aforementioned cycle (e.g., cycle 3). Therefore, these multiple weights that produce better performance on the target neural processing unit are selected and deployed on the target neural processing unit, rather than multiple weights resulting from better performance on the GPU.

[0289] Because the inference accuracy assessment is performed directly on the target device (i.e., the neural processing unit) rather than through general-purpose graphics processor simulation, the method of this invention achieves higher inference accuracy compared to conventional methods. Furthermore, the collaborative training and assessment using heterogeneous processing devices significantly accelerates the overall training speed.

[0290] As described, the present invention allows neural network models to be trained on general-purpose graphics processors (such as GPGPUs), but their performance is evaluated based on their execution performance on target hardware (such as NPUs), reducing the time required for evaluation and thus accelerating the overall evaluation process.

[0291] Several embodiments relate to training a neural network model based on multiple heterogeneous processing units. The neural network model is trained on a first processing unit using a training dataset and configured with multiple lightweight weights so that it can be executed on a second processing unit. It is inferred from the evaluation dataset that the multiple lightweight weights can be used for execution on the second processing unit.

[0292] In one or more embodiments, the first processing unit may be configured to train a neural network model with multiple lightweight weights based on inferences from an evaluation dataset.

[0293] In one or more embodiments, neural network model training may further include retraining the neural network model.

[0294] In one or more embodiments, the first processing unit may be a general-purpose graphics processor, and the second processing unit may be a neural processing unit.

[0295] In one or more embodiments, the second processing unit may be a neural processing unit dedicated to accelerating artificial intelligence operations, configured to support inference operations but not training operations.

[0296] In one or more embodiments, the second processing unit may include internal memory, multiple processing elements, and an activation function operation unit.

[0297] In one or more embodiments, the power consumption of the second processing unit may be relatively lower than that of the first processing unit.

[0298] In one or more embodiments, training a neural network model with lightweight weights may include: transmitting the inference result of a second processing unit to a first processing unit, and updating the lightweight weights based on the inference result.

[0299] In one or more embodiments, the inference accuracy of the neural network model can be evaluated by transferring lightweight weights trained on a first processing unit to a second processing unit. The second processing unit can use the lightweight weights to perform inferences on the evaluation dataset and compare the inference results with the correct answers.

[0300] In one or more embodiments, training a neural network model may include: setting one or more compilation options for the neural network model, compiling the neural network model according to the compilation options to generate machine code, transmitting the machine code to a second processing unit, and executing the machine code on the second processing unit.

[0301] In one or more embodiments, training a neural network model may include setting one or more compilation options, wherein the compilation options may include at least one of pruning algorithms, quantization algorithms, parameter refinement algorithms, outlier mitigation algorithms, model compression algorithms, knowledge distillation algorithms, retraining algorithms, and artificial intelligence-based optimization algorithms.

[0302] In one or more embodiments, the second processing unit may be configured to execute machine code compiled from a neural network model, the machine code being designed for the computational circuitry of the second processing unit.

[0303] In one or more embodiments, training a neural network model with lightweight weights can be performed on a periodic basis.

[0304] This invention relates to a system for training a neural network model using heterogeneous processing units. The system includes at least one memory configured to store a training dataset and an evaluation dataset; a first processing unit configured to train lightweight weights of the neural network model using the training dataset, enabling the model to be executed by a second processing unit; and the second processing unit configured to perform inference on the evaluation dataset using the lightweight weights.

[0305] In one or more embodiments, the first processing unit may be a general-purpose graphics processor (GPGPU), and the second processing unit may be a neural processing unit (NPU).

[0306] In one or more embodiments, the second processing unit may be a neural processing unit (NPU) dedicated to accelerating artificial intelligence computations, configured to support inference computations but not training computations.

[0307] In one or more embodiments, the second processing unit may be configured to execute machine code compiled from a neural network model, the machine code being designed for the computational circuitry of the second processing unit.

[0308] In one or more embodiments, the neural network model may integrate one or more algorithms, including: pruning algorithm, quantization algorithm, parameter refinement algorithm, outlier mitigation algorithm, model compression algorithm, and at least one of knowledge distillation algorithm, retraining algorithm, or artificial intelligence-based model optimization algorithm.

[0309] In one or more embodiments, the lightweight weights updated by the first processing unit may be transmitted to the second processing unit. The first processing unit may then train the lightweight weights based on the inference results obtained by the second processing unit from processing the evaluation dataset using the lightweight weights.

[0310] In one or more embodiments, the first processing unit may be configured to train multiple lightweight weights of a neural network model on a periodic basis.

[0311] The embodiments disclosed in the specification and drawings are provided only to explain the technical content of the invention and to aid in understanding the invention, and are not intended to limit the scope of the invention. Those skilled in the art can make other modifications and changes based on the technical concept of the invention, in addition to the embodiments shown.

[0312] [National research and development programs supporting this invention]

[0313] [Task ID] 00399936

[0314] [Supervisory Authority] Ministry of Science and ICT, South Korea

[0315] [Name of the planning and management (professional) organization] Korea Information and Communication Planning and Evaluation Institute

[0316] [Research Project Title] Development of Artificial Intelligence Semiconductor SW Integration Platform Technology

[0317] [Research Project Title] Development of Edge Artificial Intelligence Semiconductor (Bench Mark Test, BMT) Platform Technology

[0318] [Name of the implementing agency] DeepX Co., Ltd. (Korea)

[0319] [Research Period] April 1, 2024 – December 31, 2027

Claims

1. A method for training a neural network model, the method comprising: comprising: training or retraining a neural network model on a first processing circuit to obtain a plurality of first weights of the neural network model; transmitting the plurality of first weights to a second processing circuit having a different configuration than the first processing circuit; performing inference by the second processing circuit using the transmitted plurality of first weights; in response to the inference performed by the second processing circuit, generating a plurality of evaluation results on the performance of the inference by the second processing circuit using the transmitted plurality of first weights; and sending the plurality of evaluation results to the first processing circuit to update, by the first processing circuit, the plurality of first weights to a plurality of second weights for transmission to the second processing circuit. The plurality of third weights operable by the first processing circuit has a data volume greater than a data volume of the plurality of first weights or the plurality of second weights operable on the second processing circuit.

2. The method for training a neural network model of claim 1, wherein, The first processing circuit is a general-purpose graphics processor, and the second processing circuit is a neural processing unit.

3. The method for training a neural network model of claim 2, wherein, The second processing circuit includes:

4. The method for training a neural network model of claim 1, wherein, an internal memory; a plurality of processing elements coupled to the internal memory and configured to perform a plurality of multiply-accumulate operations using the plurality of first weights or the plurality of second weights; and an activation function operation circuit coupled to at least the internal memory or the plurality of processing elements and configured to apply an activation function to an output from the plurality of processing elements. The second processing circuit has a lower power consumption than the first processing circuit.

5. The method for training a neural network model of claim 1, wherein, further comprising:

6. The method for training a neural network model of claim 1, wherein, setting one or more compilation options related to the neural network model for training or retraining by the first processing circuit and deployment on the second processing circuit; compiling the neural network model according to the one or more compilation options to generate machine code for instantiating the neural network model on the first processing circuit; and transmitting the machine code to the second processing circuit for execution. The one or more compilation options include at least one of performing a pruning algorithm, a quantization algorithm, a parameter refinement algorithm, an outlier mitigation algorithm, a model compression algorithm, a knowledge distillation algorithm, a retraining algorithm, and an artificial intelligence-based optimization algorithm.

7. The method for training a neural network model of claim 6, wherein, The plurality of evaluation results includes at least one of a temperature distribution of the second processing circuit, a power consumption of the second processing circuit, a number of operations per unit of power consumption, a frames per second, a number of inferences per second, and an accuracy.

8. The method for training a neural network model of claim 1, wherein, The plurality of evaluation results is generated from the second processing circuit on a cycle-by-cycle basis and sent to the first processing circuit.

9. The method for training a neural network model of claim 1, wherein, comprising:

10. A system for generating a neural network model, the system comprising: a first processing circuit configured to: train or retrain a neural network model to obtain a plurality of first weights of the neural network model, receive a plurality of evaluation results, retrain the neural network model based on the received plurality of evaluation results to generate a plurality of second weights of the neural network model, and send the plurality of second weights to a second processing circuit; and a second processing circuit having a different configuration than the first processing circuit, the second processing circuit configured to: receive the plurality of first weights, perform inference using the received plurality of first weights, in response to performing the inference using the received plurality of first weights, generate a plurality of the evaluation results on the performed inference using the received plurality of first weights, ​ transmitting a plurality of the evaluation results to the first processing circuit, receiving a plurality of the second weights, and performing another inference using the received plurality of the second weights.

11. The system for generating a neural network model of claim 10, wherein, The plurality of third weights operable by the first processing circuit has a quantity of data greater than a quantity of data of the plurality of the first weights or the plurality of the second weights operable on the second processing circuit.

12. The system for generating a neural network model of claim 11, wherein, The first processing circuit is a general-purpose graphics processor, and the second processing circuit is a neural processing unit.

13. The system for generating a neural network model of claim 10, wherein, The second processing circuit includes: an internal memory; a plurality of processing elements coupled to the internal memory and configured to perform multiply-accumulate operations using the plurality of the first weights or the plurality of the second weights; and an activation function operation circuit coupled to at least the internal memory or the plurality of the processing elements and configured to apply an activation function to an output from the plurality of the processing elements.

14. The system for generating a neural network model of claim 10, wherein, The second processing circuit has a lower power consumption than the first processing circuit.

15. The system for generating a neural network model of claim 10, wherein, The one or more processors are further configured to: set one or more compilation options related to the neural network model for training or retraining of the first processing circuit and deployment on the second processing circuit; compile the neural network model according to the one or more compilation options to generate machine code for instantiating the neural network model on the first processing circuit; and transmit the machine code to the second processing circuit for execution.

16. The system for generating a neural network model of claim 15, wherein, The one or more compilation options include at least one of performing a pruning algorithm, a quantization algorithm, a parameter refinement algorithm, an outlier mitigation algorithm, a model compression algorithm, a knowledge distillation algorithm, a retraining algorithm, and an artificial intelligence-based optimization algorithm.

17. The system for generating a neural network model of claim 10, wherein, The plurality of the evaluation results includes at least one of a temperature distribution of the second processing circuit, a power consumption of the second processing circuit, a number of operations per unit of power consumption, a frames per second, an inferences per second, and an accuracy.

18. The system for generating a neural network model of claim 10, wherein, The plurality of the evaluation results is generated from the second processing circuit on a cycle-by-cycle basis and transmitted to the first processing circuit.

19. The system for generating a neural network model of claim 10, wherein, The memory is further configured to store the plurality of the first weights and the plurality of the second weights from the first processing circuit and transmit to the second processing circuit.

20. A non-transitory computer-readable storage medium, comprising: The non-transitory computer-readable storage medium is in a neural processing unit that stores a plurality of updated weights of a neural network model, the plurality of the updated weights generated by: training or retraining the neural network model on a general-purpose graphics processor to obtain a plurality of previous weights of the neural network model; transmitting the plurality of the previous weights to the neural processing unit; performing an inference by the neural processing unit using the transmitted plurality of the previous weights; in response to performing the inference by the neural processing unit, generating a plurality of evaluation results on a performance of the inference by the neural processing unit using the transmitted plurality of the previous weights; and transmitting the plurality of the evaluation results to the general-purpose graphics processor to update, by the general-purpose graphics processor, the plurality of the previous weights to the plurality of the updated weights for transmission to the neural processing unit.

Images