Neural network model conversion method, neural processor and storage medium

Abstract

The invention discloses a neural network model conversion method, a neural processor and a storage medium. The method may comprise: a step of providing information of one or more Neural Processing Units (NPUs) to a client terminal, a step of providing information of one or more NPUs to the client terminal, and a step of providing information of one or more NPUs to the client terminal, a step of providing information of a plurality of compilation options to the client terminal; a step of receiving a first neural network (Neural Network) model from the client terminal; compiling the first neural network model to generate a second neural network model according to at least one option selected by the client terminal from the plurality of compiling options; and a step of providing a result thereof to the client terminal after processing the second neural network model with the at least one NPU selected by the client terminal.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a technology of lightening a neural network model so as to be able to be driven in an edge-type low-power neural processor. BACKGROUND

[0002] A human brain is composed of a large number of nerve cells called neurons. Each neuron is connected to several hundreds to several thousands of other neurons through a connection site called a synapse. In order to imitate human intelligence, the working principle of a biological neuron and the connection relationship between neurons are modeled, which is called a neural network (NN) model. That is, a neural network is a system in which nodes imitating neurons are connected as a layer architecture.

[0003] Such a neural network model is classified into a "single-layer neural network" and a "multi-layer neural network" according to the number of layers.

[0004] A general multi-layer neural network is composed of an input layer, a hidden layer, and an output layer. The input layer is a layer that obtains external data, and the number of neurons of the input layer can correspond to the number of input variables. At least one hidden layer is positioned between the input layer and the output layer, receives a signal from the input layer, extracts a feature, and delivers it to the output layer. The output layer receives a signal from the at least one hidden layer and outputs it to the outside. An input signal between neurons is summed after being multiplied by each connection strength having a value between 0 and 1, and if the sum is greater than a threshold value of the neuron, the neuron is activated and output as an output value through an activation function.

[0005] On the other hand, a neural network in which the number of hidden layers of an artificial neural network is increased in order to implement higher artificial intelligence is called a deep neural network (DNN).

[0006] There are many kinds of DNNs, and it is known that a convolutional neural network (CNN) is easier to extract a feature of input data and master a feature pattern.

[0007] A convolutional neural network (CNN) is a neural network that performs a function similar to the visual cortex of the human brain that processes an image. It is known that a convolutional neural network is suitable for image processing.

[0008] A convolutional neural network is configured in a form in which a convolution channel and a pooling channel are repeated.

[0009] In a convolutional neural network, most of the operation time is occupied by a convolution operation. The convolutional neural network extracts image features of each channel by a kernel in the form of a matrix, and recognizes things by providing constancy of movement or distortion, etc. by means of pooling. In each channel, after a feature map is obtained by convolution of input data and a kernel, an activation function such as a rectified linear unit (ReLU) is applied to generate an activation map of the corresponding channel. Then, pooling can be applied. The neural network that actually classifies patterns is located at the back end of the feature extraction neural network, which is called a fully connected layer. In the operation processing of the convolutional neural network, most of the operations are performed by convolution or matrix multiplication.

[0010] With the development of artificial intelligence inference capabilities, various inference services such as voice recognition, speech recognition, image recognition, target sensing, driver drowsiness sensing, dangerous moment sensing, and gesture sensing are being provided by applying artificial intelligence to various electronic devices such as artificial intelligence sound boxes, smart phones, smart refrigerators, VR devices, AR devices, artificial intelligence CCTVs, artificial intelligence cleaning robots, tablet PCs, notebook computers, autonomous vehicles, humanoid robots, robotic dogs, industrial robots, etc.

[0011] Recently, with the development of deep learning technology, the performance of trained artificial neural network inference services based on big data is continuously developing. Such artificial neural network inference services are services in which an artificial neural network repeatedly learns a large amount of training data and infers various complex data through a trained neural network model. Therefore, various services are being provided to the above electronic devices using artificial neural network technology.

[0012] In addition, recently, in order to accelerate the operation speed for artificial intelligence (AI), a neural processing unit (NPU) has been developed.

[0013] However, as the functions and accuracy required for inference services using neural networks gradually increase, the data size of the neural network model, the amount of operation, and the size of the training data are increasing exponentially. Therefore, the required performance of the processor and the memory capable of performing such neural network model inference operations is also gradually increasing. SUMMARY

[0014] TECHNICAL PROBLEM The inventors of the present disclosure recognized that the conventional neural network model operation has a high power consumption, a processor operation bottleneck phenomenon caused by a relatively low memory bandwidth, and a memory latency problem. Thus, the inventors recognized the fact that there are many difficulties in improving the neural network model operation processing performance, and in this case, the lightening of the neural network model was researched in order to be able to improve this problem.

[0015] Specifically, the inventors of the present disclosure recognized that when the data size of the neural network model is large, the required data cannot be prepared in advance, and thus a delay frequently occurs. In addition, the inventors of the present disclosure recognized that in this case, the processor is in a starved or idle state in which the data to be processed cannot be obtained, and thus an actual operation cannot be performed, resulting in low operation performance.

[0016] In particular, this problem is more serious in various electronic devices used in edge computing. The edge computing means an edge or a peripheral portion in which the computation occurs, and means a terminal directly generating data or various electronic devices located in a position close to the terminal. The edge computing can be referred to as an edge device.

[0017] Further description, some of the computing systems in the cloud computing system are far from the servers of the data center, and are located at the end of the cloud computing system, and communicate with the servers of the data center, and such a computing system can be defined as an edge device. The edge device can also be used when it is required to perform the required task immediately and stably, for example, in the case of autonomous driving robots, autonomous vehicles, etc. that need to process a large amount of data within 1 / 1000 seconds. Therefore, the fields to which the edge device can be applied are rapidly increasing.

[0018] Therefore, the inventors of the present disclosure attempted to variously research and develop a neural network model lightening technology suitable for a stand-alone low-power low-cost neural processor.

[0019] That is, the inventors of the present disclosure recognized that, in order to enable the neural network model to be built-in in various electronic devices and to be independently operated, it is most important to lighten the neural network model parameters.

[0020] On the other hand, the inventors of the present disclosure also recognized that, in order to commercialize a neural processor (NPU: Neural Processing Unit) that drives the neural network model, there are various problems to be solved.

[0021] First, the user has insufficient information in selecting a neural processor for driving the developed neural network model.

[0022] Second, NPU is a technology that has just begun to be commercialized, and in order to understand whether a neural network model developed based on a GPU can be run in a specific NPU, various questionnaires and data sheets need to be researched at a level requiring technical support from engineers. In particular, the number of layers, the size of parameters, and the addition of special functions can be changed according to user needs, and thus the degree of change in the neural network model, which is difficult to implement generalization, is various.

[0023] Third, thus, it is difficult for a user to predict whether a developed neural network model can be driven in a specific NPU. That is, after purchasing an NPU, evaluation can result in a problem that a specific operation or operation is not supported and thus driving is not possible.

[0024] Fourth, it is difficult for a user to predict what performance is achieved when a developed neural network model is driven in a specific NPU. That is, it is difficult to predict whether desired power consumption and desired FPS are satisfied.

[0025] In particular, the weight value size, the feature map size, the number of layers, and the activation function characteristics of the neural network model differ for each neural network model, and thus it is difficult to predict desired performance.

[0026] Therefore, the inventors of the disclosure constructed a method and apparatus that makes it possible to select an optimal NPU product and more quickly determine model optimization conditions in the selected NPU by providing a solution or service as follows, that is, if AI code (e.g., a TensorFlowTM, PyTorchTM, ONNXTM model file, etc.) is dragged and dropped (uploaded) to a specific online simulation service, all a series of operations required by a user are uniformly performed online, and the user is provided with the greatest convenience and value.

[0027] Therefore, the technical problem to be solved by the disclosure is to make a neural network model lightweight in an optimal condition so that a specific function can be inferred above a predetermined accuracy and the minimum amount of power and memory is used.

[0028] Therefore, the technical problem to be solved by the disclosure is to variously simulate optimization options of a neural network model to optimize a neural network model running in a neural processing unit.

[0029] Therefore, the technical problem to be solved by the disclosure is to optimize parameters of each layer of a neural network model to efficiently quantize a graph-based neural network model.

[0030] Technical solutions To achieve the foregoing, according to an example of the present disclosure, a method can be provided. The method can include a step of receiving a first neural network model including one or more functions; a step of converting the one or more functions and generating a second neural network model in a directed acyclic graph (DAG) form including one or more graph modules; a step of obtaining maximum and minimum values of a parameter input in the one or more graph modules and calculating one or more scale values; a step of updating the parameter based on the one or more scale values; and a step of generating a third neural network model in a machine code form executable in a specific neural processor including the updated parameter.

[0031] The third neural network model can be generated based on hardware information of the specific neural processor.

[0032] The step of updating the parameter can be performed based on one or more selected compilation options.

[0033] The step of updating the parameter can be performed by selecting one or more of a quantization option, an outlier mitigation option, a parameter refinement option, a layer-wise training option, a Quantization Aware retraining (QAT) option, a quantization aware self-distillation (QASD) option, and a pruning option.

[0034] The step of updating the parameter can be performed according to one or more of a quantization algorithm, an outlier mitigation algorithm, a parameter refinement algorithm, a layer-wise training algorithm, a Quantization Aware retraining (QAT) algorithm, a quantization aware self-distillation (QASD) algorithm, and a pruning algorithm.

[0035] The third neural network model can be compiled such that multiply and accumulation (MAC) operations included in the third neural network model are processed in a processing unit of the specific neural processor.

[0036] The compiled third neural network model can include MAC operations, which can include at least one of a convolution operation, a general matrix multiplication operation, and a matrix multiplication operation.

[0037] Weight value parameters of the third neural network model input to the processing unit of the specific neural processor can be compiled in an integer form.

[0038] An input parameter of the third neural network model to a processing unit of the specific neural processor can be compiled as an integer modality.

[0039] The third neural network model can be compiled to cause an activation function operation included in the third neural network model to be processed in an activation function circuit of the specific neural processor.

[0040] An output parameter of a processing unit of the specific neural processor of the third neural network model can be an integer modality, and the third neural network model can be compiled to cause the output parameter to be dequantized by a dequantization circuit of the specific neural processor.

[0041] An output parameter of an activation function circuit of the specific neural processor of the third neural network model can be a floating point modality, and the third neural network model can be compiled to cause the output parameter to be quantized by a quantization circuit of the specific neural processor.

[0042] The updated parameter of the third neural network model includes one or more weight value parameters.

[0043] The third neural network model can include the one or more scale values for a dequantization operation of a dequantization circuit of the specific neural processor.

[0044] According to an example of the disclosure, a neural processor can be provided. The neural processor is an inference dedicated neural processor configured to run inference dedicated machine code including weight value parameters of a neural network model and scale values corresponding to the weight value parameters, the neural processor including: a first circuit for an on-chip memory configured to store at least a portion of data of the machine code; a second circuit for one or more processing units configured to process the weight value parameters and an input parameter and generate an output parameter; a third circuit for an activation function unit configured to apply an activation function to the output parameter and generate an activation parameter; and a fourth circuit for a control section configured to control the first to third circuits according to the machine code.

[0045] The weight value parameters included in the machine code can be updated according to one or more of a quantization algorithm, an outlier mitigation algorithm, a parameter refinement algorithm, a layer-wise training algorithm, a quantization aware retraining (QAT) algorithm, a quantization aware self-distillation (QASD) algorithm, and a pruning algorithm.

[0046] The non-quantization circuit can be configured to dequantize the output parameter of the second circuit according to the machine code.

[0047] The quantization circuit can be configured to quantize the activation parameter of the third circuit according to the machine code.

[0048] The second circuit can be configured to process an integer parameter, and the third circuit can be configured to process a floating point parameter.

[0049] According to an example of the disclosure, a non-volatile computer-readable storage medium recording instructions can be provided. As the non-volatile computer-readable storage medium recording the instructions, the instructions, when executed by one or more processors, can cause the one or more processors to perform: a step of receiving a first neural network model including one or more functions; a step of converting the one or more functions and generating a second neural network model in a directed acyclic graph (DAG) form including one or more graph modules; a step of obtaining a maximum value and a minimum value of a parameter input in the one or more graph modules and calculating one or more scale values; a step of updating the parameter based on the one or more scale values; and a step of generating a third neural network model in a machine code form executable in a specific neural processor including the updated parameter.

[0050] Inventive Effects According to an example of the disclosure, a non-graph-based neural network model can be converted into a graph-based neural network model. In addition, according to an example of the disclosure, a neural network model can be lightened.

[0051] According to another example of the disclosure, parameters of each graph module of a graph-based neural network model can be optimized for quantization of a neural network model.

[0052] Effects of the disclosure are not limited by the above examples, and various effects are included in the disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 A conceptual diagram for describing an example neural network model.

[0054] Figure 2a A graph for describing a basic architecture of a convolutional neural network (CNN).

[0055] Figure 2b A comprehensive graph for showing an operation of a convolutional neural network in an easy-to-understand manner.

[0056] Figure 3 A conceptual diagram of a neural processor for describing an example of the disclosure.

[0057] Figure 4aA conceptual diagram of one of the processing units in a plurality of processing units applicable to an example of the present disclosure.

[0058] Figure 4b A conceptual diagram of an SFU applicable to an example of the present disclosure.

[0059] Figure 5 An example diagram of a neural processor 100. Figure 3 An example diagram of a variation of the neural processor 100 shown.

[0060] Figure 6 An example diagram of a neural network model optimization apparatus 1500 and an edge device 1000 of an example of the present disclosure.

[0061] Figure 7 An example diagram of a compiler 300b-10 for describing in detail. Figure 6 An example diagram of a first conversion section 300b-11 shown.

[0062] Figure 8 An example diagram of a first conversion section 300b-11 for describing in detail. Figure 7 An example diagram of a marker adding section 300b-13 shown.

[0063] Figure 9a An example diagram of a marker adding section 300b-13 for describing in detail. Figure 7 An example diagram of a marker adding section 300b-13 shown.

[0064] Figure 9b An example diagram of a marker adding section 300b-13 for describing in detail. Figure 7 An example diagram of a marker adding section 300b-13 shown.

[0065] Figure 10 An example diagram for showing the reason why it is important to accurately determine the scale value and the offset value.

[0066] Figure 11 An example diagram of an optimization section 300b-16 shown. Figure 7 An example diagram of an optimization section 300b-16 for describing in detail.

[0067] Figure 12a An operation of a layer-by-layer training section 300b-16c of an example of the present disclosure is described in detail.

[0068] Figure 12b A conceptual diagram of a QASD 300b-16e for describing an operation of the QASD 300b-16e of an example of the present disclosure.

[0069] Figure 12c An example of an operation of a pruning section 300b-16f shown. Figure 11 An example of an operation of a pruning section 300b-16f.

[0070] Figure 12d An example of an operation of a pruning section 300b-16f. Figure 12e Examples of each step for describing an operation of an outlier mitigation section 300b-16a according to an embodiment of the present disclosure.

[0071] Figure 13a A convolution example for a first neural network model for describing an example of the present disclosure.

[0072] Figure 13b A convolution example for a second neural network model for describing an example of the present disclosure.

[0073] Figure 13c A convolution example for a third neural network model for describing an example of the present disclosure.

[0074] Figure 13d A convolution, dequantization, and quantization example for a third neural network model for describing an example of the present disclosure.

[0075] Figure 14 A block diagram showing a structure of a neural network model performance evaluation system according to another example of the present disclosure.

[0076] Figure 15 A block diagram showing a structure of a neural network model optimization apparatus according to another example of the present disclosure.

[0077] Figure 16 A block diagram showing a structure of a compiler of a neural network model optimization apparatus according to another example of the present disclosure.

[0078] Figure 17 A block diagram showing a structure of an optimization module of a neural network model processing apparatus according to another example of the present disclosure.

[0079] Figure 18a A block diagram showing an interface for selecting a compilation option for a plurality of neural processing units of a neural network model processing apparatus according to another example of the present disclosure.

[0080] Figure 18b A block diagram showing an interface for performance evaluation and proposal for a plurality of neural processing units of a neural network model processing apparatus according to another example of the present disclosure.

[0081] Figure 19a to Figure 19d A block diagram showing a structure of one neural processing unit of a neural network model optimization apparatus according to another example of the present disclosure.

[0082] Figure 20 A block diagram showing a structure of a plurality of neural processing units according to another example of the present disclosure.

[0083] Figure 21 A flowchart showing a neural network model performance evaluation method according to another example of the present disclosure.

[0084] Figure 22 A flowchart showing a neural network model performance evaluation method according to another example of the present disclosure.

[0085] Figure 23 A flowchart of a neural network model performance evaluation method for describing another example of the disclosure. DETAILED DESCRIPTION

[0086] The specific architecture and / or steps of the examples of the disclosure are merely used to describe examples of the disclosure concept. Therefore, the examples of the disclosure concept can be implemented in various forms. The examples of the disclosure concept can be implemented in various forms. The disclosure concept is not to be construed as being limited to the examples of the disclosure.

[0087] Various modifications can be made to the examples of the disclosure concept, and various forms can be taken. Therefore, specific examples are exemplarily shown in the drawings, and the disclosure or application is described in detail. However, this is not intended to limit the examples of the disclosure concept to specific disclosed forms. It should be understood that the examples of the disclosure concept include all modifications, equivalents, and alternatives included in the technical scope of the disclosure concept.

[0088] The first and / or second terms can be used to describe various components, but the components are not limited by the terms. The terms can be used only for the purpose of distinguishing one component from another component. The terms can be named as the second component when the first component is named, and similarly, the second component can be named as the first component without departing from the scope of the disclosure concept.

[0089] When it is referred to that a certain component is "connected" or "coupled" to another component, it should be understood that it can be directly connected or coupled to the other component, or there can be other components therebetween. In contrast, when it is referred to that a certain component is "directly connected" or "directly coupled" to another component, it should be understood that there is no other component therebetween. Other expressions describing the relationship between components, i.e., "between" or "directly between" or "adjacent to" or "directly adjacent to" and the like should be interpreted in the same manner.

[0090] The terms used in the disclosure are used only to describe specific examples, and are not intended to limit the intent of the disclosure. As long as the context is not explicitly indicated otherwise, the singular expression includes the plural expression. In the disclosure, the terms such as "include" or "have" are intended to specify the presence of the described features, numbers, steps, actions, components, assemblies, or combinations thereof. Therefore, it should be understood that the presence or addition possibility of one or more other features or numbers, steps, actions, components, assemblies, or combinations thereof is not precluded.

[0091] Unless defined differently, all terms used herein including technical or scientific terms have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. Terms, as commonly used in dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined in this disclosure.

[0092] In describing example aspects, technical details that are well known in the art and not directly related to the disclosure are omitted. This is to convey the gist of the present disclosure without unnecessary obscuring the disclosure.

[0093] <Definitions of Terms> The following terms used in the present disclosure are briefly organized to help understand the disclosure presented in the present disclosure.

[0094] NPU: An abbreviation of Neural Processing Unit, can mean a processor dedicated to artificial neural network model operation independent of CPU (Central processing unit), GPU.

[0095] NN: An abbreviation of artificial neural network, can mean a network in which nodes are connected in a layer architecture by connecting neurons in the human brain through synapses in order to imitate human intelligence.

[0096] DNN: An abbreviation of Deep Neural Network, can mean a neural network in which the number of hidden layers of the neural network is increased in order to achieve higher artificial intelligence.

[0097] CNN: An abbreviation of Convolutional Neural Network, is a neural network that functions similarly to processing images in the visual cortex of the human brain. It is known that the convolutional neural network is suitable for image processing, and is easy to extract features of input data and recognize feature patterns.

[0098] Transformer: A Transformer Neural Network is one of the neural network architectures widely used in natural language processing operations. A Transformer includes parameters such as input, query (Q), key (K), and value (V). The input to a Transformer model consists of a sequence of tokens. A token can be a word, subword, or character, among others. Each token of the input sequence is embedded as a high-dimensional vector. The embedding helps the model to represent the input tokens in a continuous vector space. The Transformer cannot inherently understand the order of the input tokens, so position encodings are added to the embedding. In this way, the model can obtain the position information of the tokens within the sequence. The core of a Transformer model is the self-attention mechanism. This mechanism allows determining how much attention to pay to other parts when processing a particular token while making a prediction. The attention mechanism includes three sets of vectors: query (Q), key (K), and value (V). A Transformer computes three vectors for each input token: query (Q), key (K), and value (V). These vectors are used to compute attention scores, which determine how much importance to give to different parts of the sequence when processing a particular token while making a prediction. The attention score is computed by the inner product of the query (Q) and key (Q) and divided by the square root of the dimension of the key (Q) vector. The result is processed by an exponential function to obtain an attention weight value. This weight value (i.e., scaled dot-product attention mechanism) is used to compute the weighted sum of the value (V) vectors and generate the final output for each position. To capture various relationships between words, the self-attention mechanism is usually performed multiple times in parallel. It is performed using various sets of query (Q), key (K), and value (V) parameters, and the outputs of these various attention heads (i.e., multi-head attention) are concatenated and linearly transformed. After the self-attention layer, a position feed-forward network is usually concatenated. This is a fully connected layer that is applied independently to each position of the sequence. Layer normalization and residual connections are applied around each sub-layer to help provide training stability and facilitate the flow of gradients. A Transformer is usually used as an encoder-decoder architecture for operations such as machine translation. The encoder processes the input sequence, and the decoder generates the output sequence. In short, a Transformer model employs a self-attention mechanism using query (Q), key (K), and value (V) vectors to capture the context information of the input sequence and uses multi-head attention and feed-forward networks to learn complex relationships in the data.

[0099] A Visual Transformer (ViT) is a model that extends the original Transformer model for computer vision operations. The Transformer was developed primarily for natural language processing, but the ViT applies the Transformer architecture to a variety of operations. Like the Transformer, the input to the ViT is also a sequence of tokens. In computer vision, the input tokens represent image patches. Instead of processing the entire image as a single input, the ViT divides the image into fixed-size non-overlapping patches (i.e., image patch embeddings). Each patch is linearly embedded to form a vector to generate a sequence of embeddings. Since the ViT model cannot inherently understand the order of the patches, position encodings are added to the patch embeddings to provide information about the spatial configuration (i.e., position encodings). To capture complex inter-patch relationships, the patch embeddings are linearly projected to a high-dimensional space. The patch embeddings are used as input to the Transformer encoder. Each patch embedding is treated as a sequence token. Similar to the Transformer, the ViT employs a self-attention mechanism that uses query (Q), key (K), and value (V) vectors. These vectors are computed for each patch embedding to compute attention scores that capture the dependencies between different parts of the image. To capture various inter-patch relationships, various attention heads are used (i.e., multi-head attention). The outputs of these heads are concatenated and linearly transformed. After the self-attention, a positional feed-forward network is typically applied independently to each patch embedding. In this way, the model learns local features. Similar to the Transformer, the ViT uses layer normalization and residual connections to enhance training stability and facilitate gradient flow. The ViT encoder stack processes the sequence of patch embeddings through multiple layers. Each layer includes self-attention, feed-forward, normalization, and residual connections. Unlike the Transformer, the ViT does not use the entire sequence output for prediction. Instead, a global average pooling layer is applied to obtain a fixed-size representation that is used for the classification operation.

[0100] The preferred examples of the present disclosure are described below with reference to the accompanying drawings, in order to describe the present disclosure in detail. Hereinafter, examples of the present disclosure are described in detail with reference to the accompanying drawings.

[0101] <Artificial Intelligence> Humans have intelligence that can recognize, classify, infer, predict, control / decision making, etc. Artificial intelligence (AI) means artificially imitating human intelligence.

[0102] The human brain is composed of a large number of nerve cells called neurons. Each neuron is connected to several hundreds to several thousands of other neurons through a connection site called a synapse. In order to mimic human intelligence, the working principle of a biological neuron and the connection relationship between neurons are modeled, which is called a neural network model. That is, a neural network is a system that connects nodes that mimic neurons in a layer architecture.

[0103] This neural network model is divided into a "single-layer neural network" and a "multi-layer neural network" according to the number of layers. A general multi-layer neural network is composed of an input layer, a hidden layer, and an output layer. (1) The input layer is a layer that obtains external data, and the number of neurons of the input layer is the same as the number of input variables. (2) The hidden layer is located between the input layer and the output layer, receives signals from the input layer, extracts features, and delivers them to the output layer. (3) The output layer receives signals from the hidden layer and outputs them to the outside. The input signals between neurons are multiplied by each connection strength having a value between 0 and 1 and then added. If the sum is greater than the threshold value of the neuron, the neuron is activated, and an activation function is implemented as an output value.

[0104] On the other hand, a neural network in which the number of hidden layers is increased in order to achieve higher artificial intelligence is called a deep neural network (DNN).

[0105] DNNs are being developed in various architectures. For example, a convolutional neural network (CNN), which is an example of a DNN, is known to easily extract features of input values (images or pictures) and recognize patterns of the extracted output values. The CNN can be configured in a form in which a convolution operation, an activation function operation, a pooling operation, and the like are processed in a certain order.

[0106] For example, in each layer of the DNN, parameters (i.e., input values, output values, weights, or kernels, etc.) can be matrices composed of multiple channels. The parameters can be processed in the NPU through convolution or matrix multiplication. The output values are generated after the operation processing in each layer.

[0107] For example, a vision Transformer or Transformer is a DNN based on an attention technique. The Transformer heavily utilizes matrix multiplication operations. The Transformer can obtain an attention (Q, K, V) as an output value using input values and parameters such as a query vector (query: Q), a key vector (key: K), and a value vector (value: V). The Transformer can process various inference operations based on the output value (i.e., the attention (Q, K, V)). The Transformer has a tendency to exhibit superior inference performance than a CNN.

[0108] Figure 1 A conceptual diagram for describing an example neural network model.

[0109] The following describes an operation of an example neural network model 110a that can be executed in the neural processor 100.

[0110] Figure 1 The example neural network model 110a can be a neural network trained to perform various inference functions such as object recognition, speech recognition, etc.

[0111] The neural network model 110a can be a deep neural network (DNN).

[0112] However, the neural network model 110a of the present disclosure example is not limited to a deep neural network.

[0113] For example, the neural network model 110a can be implemented by a Siamese Network, a Triplet Network, a ContrastiveLoss, a FaceNet, a DeepID, a SphereFace, an ArcFace, a Florence-2, a DaViT, a MobileViT, a ViT, a Swin-Transformer, a Transformer, a YOLO, a CNN, a PIDNet, a BiseNet, a RCNN, a VGG, a VGG16, a DenseNet, a SegNet, a DeconvNet, a DeepLAB V3+, a U-net, a SqueezeNet, an Alexnet, a ResNet18, a MobileNet-v2, a GoogLeNet, a Resnet-v2, a Resnet50, a Resnet101, an Inception-v3, etc. model. However, the present disclosure is not limited to the above-described models. In addition, the neural network model 110a can also be an ensemble model based on at least two different models.

[0114] The inference process performed by the exemplary neural network model 110a is described below.

[0115] The neural network model 110a is an exemplary deep neural network model including an input layer 110a-1, a first connection network 110a-2, a first hidden layer 110a-3, a second connection network 110a-4, a second hidden layer 110a-5, a third connection network 110a-6, and an output layer 110a-7. However, the present disclosure is not limited to the illustrated neural network model. The first hidden layer 110a-3 and the second hidden layer 110a-5 can also be referred to as a plurality of hidden layers. Figure 1 The illustrated neural network model. The first hidden layer 110a-3 and the second hidden layer 110a-5 can also be referred to as a plurality of hidden layers.

[0116] The input layer 110a-1 can exemplarily include x1 and x2 input nodes. That is, the input layer 110a-1 can include information of 2 input values.

[0117] The first connection network 110a-2 can exemplarily include information of 6 weight values for connecting each node of the input layer 110a-1 to each node of the first hidden layer 110a-3. Each weight value is multiplied by an input node value, and an accumulated value of the product values is stored in the first hidden layer 110a-3. The weight values and the input node values can be referred to as parameters of the neural network model.

[0118] The first hidden layer 110a-3 can exemplarily include a1, a2, and a3 nodes. That is, the first hidden layer 110a-3 can include information of 3 node values.

[0119] Figure 1 The first processing element PE1 of the first processing unit 110a-1 can process the operation of the a1 node.

[0120] Figure 1 The second processing element PE2 of the first processing unit 110a-1 can process the operation of the a2 node.

[0121] Figure 1 The third processing element PE3 of the first processing unit 110a-1 can process the operation of the a3 node. The second connection network 110a-4 can exemplarily include information of 9 weight values for connecting each node of the first hidden layer 110a-3 to each node of the second hidden layer 110a-5. The weight values of the second connection network 110a-4 are respectively multiplied by the node values input from the first hidden layer 110a-3, and an accumulated value of the product values is stored in the second hidden layer 110a-5.

[0122] The second hidden layer 110a-5 can exemplarily include b1, b2, and b3 nodes. That is, the second hidden layer 110a-5 can include information of 3 node values.

[0123] Figure 1 The fourth processing element PE4 of the second processing unit 110a-2 can process the operation of the b1 node.

[0124] Figure 1 The fifth processing element PE5 of the fifth processing unit 110a-4 can process the operation of the b2 node.

[0125] Figure 1 The sixth processing element PE6 of the sixth processing unit 110a-5 can process the operation of the b3 node.

[0126] The third connection net 110a-6 can exemplarily include information of 6 weight values connecting each node of the second hidden layer 110a-5 and each node of the output layer 110a-7. The weight values of the third connection net 110a-6 are multiplied with the node values input from the second hidden layer 110a-5 respectively, and the accumulated value of the product values is stored in the output layer 110a-7.

[0127] The output layer 110a-7 can exemplarily include y1 and y2 nodes. That is, the output layer 110a-7 can include information of 2 node values.

[0128] Figure 1 The seventh processing element PE7 of the seventh processing unit 110a-6 can process the operation of the y1 node.

[0129] Figure 1 The eighth processing element PE8 of the eighth processing unit 110a-7 can process the operation of the y2 node.

[0130] Each node can correspond to a feature value, and the feature value can correspond to a feature map.

[0131] Figure 2a FIG. 1 is a diagram to describe a basic architecture of a convolutional neural network (CNN).

[0132] Referring to FIG. 1, Figure 2a An input image can be displayed as a two-dimensional matrix composed of rows of a certain size and columns of a certain size. The input image can have a plurality of channels, where a channel can represent the number of color components of the input data image.

[0133] The convolution process means that a kernel and a convolution operation are performed while traversing the input image at a specified interval.

[0134] The convolutional neural network can have an architecture that passes the output value (convolution or matrix multiplication) of the current layer as the input value of the next layer.

[0135] For example, the convolution or matrix multiplication is defined by two main parameters (input feature map and kernel). The parameters can include input feature map, output feature map, activation map, weight value, kernel, and attention (Q, K, V), etc.

[0136] The convolution slides the kernel window on the input feature map. The step size by which the kernel slides the input feature map is called the stride.

[0137] Pooling can be applied after convolution. Additionally, fully-connected (FC) layers can be configured at the end of the convolutional neural network.

[0138] For ease of description, the following description will focus on convolution; however, specific layers of a neural network model may include various operations such as matrix multiplication.

[0139] Figure 2b A comprehensive diagram illustrating the operation of a convolutional neural network in an easily understandable way.

[0140] Reference Figure 2b For example, the input image is shown as a two-dimensional matrix of size 6×6. Additionally, in Figure 2b The example shown uses 3 nodes, namely channel 1, channel 2, and channel 3.

[0141] First, the convolution operation will be described.

[0142] Input image (in) Figure 2b In the example shown (6×6 size), at the first node, is kernel 1 for channel 1 (in... Figure 2b In the example shown, a 3×3 convolution is used to output features as a result. Figure 1 (exist Figure 2b (Example shown in the image is 4×4 in size). Additionally, the input image (in...) Figure 2b In the example shown (6×6 size), at the second node, kernel 2 for channel 2 (in... Figure 2b In the example shown, a 3×3 convolution is performed, and as a result, the output feature map 2 (in...) Figure 2b In the example shown, it is 4×4 in size. Additionally, the input image at the third node is associated with kernel 3 for channel 3 (in...). Figure 2b In the example shown, a 3×3 convolution is used to output features as a result. Figure 3 (exist Figure 2b (Example shown is 4×4 size).

[0143] In order to process each convolution, the processing units PE1 to PE12 of the neural processor 100 are configured to perform MAC operations.

[0144] The operation of the activation function is described below.

[0145] Features that can be applied to the output of self-convolution operations Figure 1 Feature diagram 2 and features Figure 3 (exist Figure 2b In the example shown, each size is 4×4) to which the activation function is applied. The output after the activation function is applied can be, for example, 4×4 in size.

[0146] The pooling operation is described below.

[0147] The features output from the activation function Figure 1 , feature map 2, features Figure 3 (in Figure 2b , the size of each is exemplarily shown as 4×4) are input to 3 nodes. It is possible to receive the feature map input from the activation function output and perform pooling. The so-called pooling can reduce the size or emphasize specific values within the matrix. Pooling methods include max pooling, average pooling, and min pooling. Max pooling is used to aggregate the maximum value within a specific area of the matrix, and average pooling can be used to find the mean within a specific area.

[0148] In Figure 2b 's example, the feature map shown as 4×4 size is reduced to 2×2 size by pooling.

[0149] Specifically, after the first node receives the feature Figure 1 for channel 1 as input and performs pooling, for example, the output is a 2×2 matrix. After the second node receives the feature map 2 for channel 2 as input and performs pooling, for example, the output is a 2×2 matrix. After the third node receives the feature Figure 3 for channel 3 as input and performs pooling, for example, the output is a 2×2 matrix.

[0150] The aforementioned convolution, activation function, and pooling are repeatedly executed, and finally, the output can be fully connected as Figure 2a shown.

[0151] The CNN described so far is also the most used method in the field of computer vision (Vision) among various deep neural network (DNN) methods. In particular, CNN shows amazing performance in various research fields performing various tasks such as image classification and object detection.

[0152] <Hardware resources required for NN operations> Figure 3 A conceptual schematic diagram of a neural processor for describing an example of the present disclosure.

[0153] Figure 3 The neural processing unit (NPU) 100 shown is a processor specifically designed to perform operations for neural networks.

[0154] A neural network means a network aggregated by artificial neurons that multiply and add respective weights if a plurality of inputs or stimuli come in, further deform a value obtained by adding a bias through an activation function, and pass it. The neural network thus trained can be used to output an inference result from input data.

[0155] The neural processor 100 can be a semiconductor implemented in an electrical / electronic circuit. The electrical / electronic circuit can mean one including a large number of electronic elements such as transistors, capacitors.

[0156] For a Transformer and / or a CNN-based neural network model, the neural processor 100 can discriminate and process matrix multiplication operations, convolution operations, etc. according to the architecture of the neural network.

[0157] For example, in each layer of a convolutional neural network CNN, an input feature map corresponding to input data and a kernel corresponding to a weight can be a tensor or a matrix composed of a plurality of channels. The convolution operation of the input feature map and the kernel is performed, and an output feature map is generated in each channel by the convolution operation and pooling. An activation function is applied to the output feature map, and an activation map of the corresponding channel is generated. Then, the activation map can be subjected to pooling. In general, the activation map can be referred to as an output feature map. Hereinafter, the activation map will be described as an output feature map for convenience of description.

[0158] However, examples of the disclosure are not limited thereto, and the output feature map means a map to which a matrix multiplication operation or a convolution operation, etc. is applied.

[0159] Further description, the output feature map of the examples of the disclosure should be interpreted in a broad sense. For example, the output feature map can be a matrix multiplication operation or a convolution operation result value. Therefore, the plurality of processing units 110 can also be implemented to further include a processing circuit portion for a complementary algorithm. That is, a part of the circuit unit of the SFU 150 described later can also be configured to be included in the plurality of processing units 110.

[0160] The neural processor 100 can be configured to include a plurality of processing units 110 for processing the convolution and matrix multiplication required for the above-described neural network operation.

[0161] The neural processor 100 can be configured to include various processing circuits optimized for matrix multiplication operations, convolution operations, activation function operations, pooling operations, stride operations, batch normalization operations, skip connection operations, concatenation operations, quantization operations, clipping operations, padding operations required for the above-described neural network operations.

[0162] For example, the neural processor 100 can be configured to include the SFU 150 for processing at least one of the activation function operations, the pooling operations, the stride operations, the batch normalization operations, the skip connection operations, the concatenation operations, the quantization operations, the clipping operations, the padding operations in the above-described algorithms.

[0163] Specifically, the neural processor 100 can include a plurality of processing elements (PEs) 110, the SFU 150, an NPU internal memory 120, an NPU controller 130, and an NPU interface 140. The plurality of processing elements 110, the SFU 150, the NPU internal memory 120, the NPU controller 130, and the NPU interface 140 can each be a semiconductor circuit connected with a large number of transistors. Thus, a part thereof can be difficult to be recognized or distinguished by the naked eye, and can be recognized only by operation.

[0164] For example, any circuit can be operated with the plurality of processing elements 110 or with the NPU controller 130. The NPU controller 130 can be configured to perform a control part function configured to control a neural network inference operation of the neural processor 100.

[0165] The neural processor 100 can include the NPU internal memory 120 configured to store parameters of a neural network model that can be inferred in the plurality of processing elements 110 and the SFU 150, and the NPU controller 130 configured to control operation scheduling of the plurality of processing elements 110, the SFU 150, and the NPU internal memory 120.

[0166] The neural processor 100 can be configured to process a feature map corresponding to an encoding and decoding method using SFC (scalable feature-map coding) or SFC (scalable feature-map coding). The method is a technology that can change the amount of data transmission variably according to the actual effective bandwidth and the signal to noise ratio (SNR) of a communication channel or a communication bus. That is, the neural processor 100 can also be configured to further include an encoder and a decoder.

[0167] The plurality of processing elements 110 can perform a part of operations for a neural network.

[0168] The SFU 150 can perform another part of operations for the neural network.

[0169] The neural processor 100 can be configured to accelerate operations of the neural network model on hardware using the plurality of processing units 110 and the SFU 150.

[0170] The NPU interface 140 can communicate with various components connected with the neural processor 100 through a system bus, for example, a memory.

[0171] The NPU controller 130 can be configured to control operations of the plurality of processing units 110 for inference operations of the neural processor 100, operations of the SFU 150, and read / write operations of the NPU internal memory 120.

[0172] The NPU controller 130 can be configured to control the plurality of processing units 110, the SFU 150, and the NPU internal memory 120 based on control information included in the compiled neural network model.

[0173] The NPU controller 130 can analyze an architecture of the neural network model to be operated in the plurality of processing units 110 and the SFU 150 or obtain analyzed information. The analyzed information can be information generated by a compiler. For example, data of the neural network model can include at least one of node data (i.e., feature map) of each layer, configuration data of the layer, local information or architecture information, a connection net connecting nodes of each layer, and respective weight value data (i.e., weight kernel). The data of the neural network model can be stored in a memory provided inside the NPU controller 130 or the NPU internal memory 120. However, it is not limited thereto, and the data of the neural network model can be stored in a separate cache memory or a register file provided in the NPU or an SoC including the NPU.

[0174] The NPU controller 130 can obtain scheduling information of an operation order of the neural network model to be executed by the neural processor 100 based on data local information or architecture information of the compiled neural network model. For example, the scheduling information can be information generated by a compiler. The scheduling information generated by the compiler can be referred to as machine code or binary code, etc.

[0175] The NPU controller 130 can obtain scheduling information of a sequence of neural network model operations to be performed by the neural processing unit 100 based on a directed acyclic graph (DAG) of the neural network model compiled by the compiler. The compiler determines the number of processing units 110 of the neural processing unit 100, the size of the NPU internal memory 120, the size of parameters of each layer of the neural network model, etc., to determine an operation schedule that can accelerate the neural network model operations. According to the operation schedule, the NPU controller 130 can be configured to control the number of processing units 110 required for each operation step, and control the read and write operations of the parameters required for each operation step to the NPU internal memory 120.

[0176] That is, the scheduling information used by the NPU controller 130 can be information generated by the compiler based on the data locality or architecture of the neural network model. Further, the compiler can efficiently perform scheduling of the neural processing unit 100 according to the degree of understanding and reconstruction of the unique characteristics of the neural network model, i.e., the neural network data locality.

[0177] Further, the compiler can efficiently perform scheduling of the NPU according to the degree of understanding of the hardware architecture and performance of the neural processing unit 100.

[0178] Further, when the neural network model is compiled by the compiler to run in the neural processing unit 100, the neural network data locality can be reconstructed. The neural network data locality can be reconstructed according to the algorithm applied by the neural network model and the operating characteristics of the processor.

[0179] Further, the scheduling information can be reconstructed according to the way in which the corresponding neural network model is processed by the neural processing unit 100, such as feature map tiling, stationary manner of processing units.

[0180] Further, the scheduling information can be reconstructed according to the number of processing units of the neural processing unit 100, the capacity of the internal memory.

[0181] Further, the scheduling information can be reconstructed according to the bandwidth of the memory in communication with the neural processing unit 100.

[0182] This is because, due to the above factors, even if the same neural network model is processed, the neural processing unit 100 determines the sequence of data required at each instant differently by clock unit.

[0183] The compiler can determine the data locality based on the sequence of data required for the operations of the neural network model and the operation sequence of the layers, unit convolutions, and / or matrix multiplications of the neural network, and generate the compiled machine code.

[0184] The NPU controller 130 can be configured to use the scheduling information included in the machine code.

[0185] The NPU controller 130 can obtain the feature map of the layer of the neural network model and the memory address value storing the weight value data based on the scheduling information.

[0186] For example, the NPU controller 130 can obtain the feature map of the layer of the neural network model and the memory address value storing the weight value data stored in the NPU internal memory. Accordingly, the NPU controller 130 can fetch the feature map of the layer of the neural network model to be driven and the weight value data from the main memory and store them in the NPU internal memory 120.

[0187] For example, the neural processor 100 can set the memory mapping of the main memory for efficient read / write operation of the neural network model parameters (e.g., weight values and feature maps) based on the data locality information of the neural network model to reduce the data transfer delay between the main memory (not shown) and the NPU internal memory 120.

[0188] The feature map of each layer can have a corresponding memory address value.

[0189] Each weight value data can have a corresponding memory address value.

[0190] The NPU controller 130 can obtain scheduling information of the operation order of the plurality of processing units 110 based on the data locality information or the architecture information of the neural network model, for example, based on the batch data, the locality information, or the architecture information of the layer of the neural network of the neural network model. The scheduling information can be generated in the compiling step.

[0191] The NPU controller 130 can operate based on the information scheduled based on the data locality information or the architecture information of the neural network model, and thus can operate differently from the scheduling concept of the general CPU. The general CPU scheduling considers fairness, efficiency, stability, reaction time, etc. to operate so as to be able to exert optimal efficiency. That is, scheduling is performed considering priority, operation time, etc. so as to perform the most processing in the same time.

[0192] The conventional CPU uses an algorithm that schedules operations considering the priority of each processing, operation processing time, etc.

[0193] Unlike this, the NPU controller 130 can control the neural processor 100 in the order of processing determined based on the data locality information or the architecture information of the neural network model of the neural processor 100.

[0194] Further, the NPU controller 130 can drive the neural processor 100 in the order of processing determined based on the data locality information or the architecture information of the neural network model and / or the data locality information or the architecture information of the neural processor 100 to be used.

[0195] That is, a cache policy (e.g., LRU, FIFO, LFU) used on a von Neumann architecture is inefficient for controlling the NPU internal memory 120 of the neural processing unit 100. That is, since a neural network model is not a simple chain structure algorithm, but has a Directed Acyclic Graph (DAG) algorithm structure, the operation of the neural processing unit 100 is more efficient using a cache policy capable of recognizing data locality of the neural network model.

[0196] However, the present disclosure is not limited to data locality information or architecture information of the neural processing unit 100.

[0197] The NPU controller 130 can be configured to store data locality information or architecture information of a neural network.

[0198] That is, the NPU controller 130 can determine a processing order even by using only data locality information or architecture information of a neural network of at least a neural network model.

[0199] Further, the NPU controller 130 can determine a processing order of the neural processing unit 100 in consideration of data locality information or architecture information of a neural network model and data locality information or hardware architecture information of the neural processing unit 100. In addition, processing optimization of the neural processing unit 100 can also be performed according to the determined processing order.

[0200] That is, the NPU controller 130 can be configured to run based on machine code compiled by a compiler, but in another example, the NPU controller 130 can also be configured to have an embedded compiler built-in. According to the above configuration, the neural processing unit 100 can be configured to obtain a file in the form of a framework of various AI software to generate machine code. For example, the framework of AI software is TensorFlow, PyTorch, Keras, XGBoost, mxnet, DARKNET, ONNX, etc.

[0201] The plurality of processing units 110 means a structure configured with a plurality of processing elements PE1~PE12 configured to perform operations on feature map and weight value data of a neural network. Each processing unit can include a multiply and accumulate (MAC) operator and / or an Arithmetic Logic Unit (ALU) operator. However, examples of the present disclosure are not limited thereto.

[0202] Each processing unit can be configured to selectively further include an additional special function unit circuit for processing an additional special function.

[0203] For example, the processing element (PE) can also be implemented in a manner that also includes a batch-normalization unit, an activation function unit, an interpolation unit, and the like.

[0204] The SFU 150 can include a circuit unit configured to discriminate and process a function unit for a skip connection operation, a function unit for an activation function operation, a function unit for a pooling operation, a function unit for a dequantization operation, a function unit for a quantization operation, a function unit for a non-maximum suppression operation, a function unit for a batch normalization operation, a function unit for an interpolation operation, a function unit for a concatenation operation, and a function unit for a bias operation, and the like, according to the architecture of the neural network model. That is, the SFU 150 can include a plurality of special function operation processing circuit units. The circuit unit included in the SFU 150 processes various operations that are difficult for the processing unit to process.

[0205] The plurality of processing units are exemplarily shown in Figure 3 However, the plurality of processing units 110 can also be configured in parallel with an operator implemented by a plurality of multipliers and an adder tree instead of a MAC within one processing unit. At this time, the plurality of processing units 110 can also refer to at least one processing unit including a plurality of operators.

[0206] The plurality of processing units 110 can be configured to include a plurality of processing units PE1-PE12. Figure 3 The plurality of processing units PE1-PE12 shown are only examples for ease of description, and the number of the plurality of processing units PE1-PE12 is not limited. The size or number of the plurality of processing units 110 can be determined according to the number of the plurality of processing units PE1-PE12. The size of the plurality of processing units 110 can be implemented in the form of an N×M matrix. Here, N and M are integers greater than 0. The plurality of processing units 110 can include N×M processing units. That is, the processing unit can be one or more.

[0207] The size of the plurality of processing units 110 can be designed in consideration of the characteristics of the neural network model run by the neural processor 100.

[0208] The plurality of processing units 110 are configured to perform addition, multiplication, accumulation, and the like functions required for neural network operations. In other words, the plurality of processing units 110 can be configured to perform MAC (multiplication and accumulation) operations.

[0209] Hereinafter, a first processing unit PE1 in the plurality of processing units 110 will be described as an example.

[0210] Figure 4aA conceptual diagram of one processing element of a plurality of processing elements applicable to one example of the present disclosure.

[0211] The neural processor 100 of one example of the present disclosure can include a plurality of processing elements 110, an NPU internal memory 120 configured to store a neural network model that can be inferred by the plurality of processing elements 110, and an NPU controller 130 configured to control the plurality of processing elements 110 and the NPU internal memory 120, wherein the plurality of processing elements 110 can be configured to perform a MAC operation, and the plurality of processing elements 110 can be configured to quantize and output a result of the MAC operation. However, examples of the present disclosure are not limited thereto.

[0212] The NPU internal memory 120 can store all or a part of the neural network model according to a memory size and a data size of the neural network model.

[0213] The first processing element PE1 can include a multiplier 111, an adder 112, an accumulator 113, and a bit quantization unit 114. However, examples of the present disclosure are not limited thereto, and the plurality of processing elements 110 can be variously implemented in consideration of an operation characteristic of a neural network.

[0214] The multiplier 111 multiplies (N)bit data inputted and (M)bit data. An operation value of the multiplier 111 is outputted as (N+M)bit data.

[0215] The multiplier 111 can be configured to input one weight value parameter and one feature map parameter.

[0216] In this case, when the multiplier 111 inputs the weight value parameter or the feature map parameter having a value of 0, the multiplier 111 can be inactivated. Accordingly, the multiplier 111 can be configured to reduce power consumption of the plurality of processing elements 110 when processing a weight value parameter to which a pruning algorithm is applied, or when a value of a feature map parameter is 0. Accordingly, the processing element including the multiplier 111 can be inactivated.

[0217] The accumulator 113 accumulates the operation value of the multiplier 111 and the operation value of the accumulator 113 using the adder 112 according to the number of (L) loops. Accordingly, a bit width of the output and input data of the accumulator 113 can be outputted as (N+M+log2(L))bit. Herein, L is an integer greater than 0.

[0218] After the accumulator 113 finishes accumulation, an initialization reset can be inputted to initialize the data stored in the accumulator 113 to 0. However, examples of the present disclosure are not limited thereto.

[0219] The bit quantization unit 114 can reduce the bit width of the data outputted by the accumulator 113. The bit quantization unit 114 can be controlled by the NPU controller 130. The bit width of the quantized data can be outputted as (X) bit. Wherein, X is an integer greater than 0. According to the above configuration, the plurality of processing units 110 are configured to perform MAC operation, and the plurality of processing units 110 can have the effect of quantizing the MAC operation result and outputting. In particular, such quantization has the effect of further saving power consumption as the (L) loops increase. In addition, if the power consumption is reduced, it also has the effect of reducing the heat generation. In particular, if the heat generation is reduced, it can have the effect of reducing the possibility of the neural processor 100 malfunctioning due to high temperature.

[0220] The output data (X) bit of the bit quantization unit 114 can become the node data of the next layer or the input data of the convolution. If the neural network model has been quantized, the bit quantization unit 114 can be configured to obtain the quantized information from the neural network model. However, it is not limited thereto, and the NPU controller 130 can also be configured to analyze the neural network model and extract the quantized information. Therefore, the output data (X) bit can be converted to the quantized bit width and outputted in a manner corresponding to the size of the quantized data. The output data (X) bit of the bit quantization unit 114 can be stored in the NPU internal memory 120 according to the quantized bit width.

[0221] The plurality of processing units 110 of the neural processor 100 of an example of the present disclosure include the multiplier 111, the adder 112, and the accumulator 113. The bit quantization unit 114 can be selectively selected according to whether quantization is applied. In another example, the bit quantization unit can also be configured to be included in the SFU 150.

[0222] Figure 4b A conceptual diagram of the SFU applicable to an example of the present disclosure is described.

[0223] Reference Figure 4b The SFU 150 includes a plurality of functional units. Each functional unit can be selectively operated. Each functional unit can be selectively turned on or off. That is, each functional unit can be set.

[0224] In other words, the SFU 150 can include a plurality of circuit units required for neural network inference operation.

[0225] For example, the circuit units of the SFU 150 can include functional units for skip-connection operations, functional units for activation function operations, functional units for pooling operations, functional units for quantization operations, functional units for non-maximum suppression (NMS) operations, functional units for batch-normalization operations, functional units for interpolation operations, functional units for concatenation operations, and functional units for bias operations, etc. In addition, a certain functional unit needs to be processed with a floating-point parameter, and thus the conversion of the floating-point parameter and the integer parameter can be selectively performed in the SFU 150. Each functional unit can be composed of respective circuit units. The functional unit for the quantization operation and the functional unit for the dequantization operation can also be integrated into one circuit.

[0226] According to examples of the present disclosure, the floating-point can be various formats configured to have a mantissa composed of a certain number of bits and an exponent composed of a certain number of bits, and the present disclosure is not limited to the number of bits of the mantissa and the number of bits of the exponent of the parameter to be processed by a certain neural processor. In some examples, the floating-point can be replaced with a Brain-floating number.

[0227] The functional units of the SFU 150 can be selectively turned on or off according to the data local information of the neural network model. The data local information of the neural network model can include the off or off-related control information of the corresponding functional unit when performing the operation for a certain layer.

[0228] The activated units among the functional units of the SFU 150 can be turned on. In selectively turning off a part of the functional units of the SFU 150, the power consumption of the neural processor 100 can be saved. On the other hand, in order to turn off a part of the functional units, power gating can be used. Alternatively, in order to turn off a part of the functional units, clock gating can also be performed.

[0229] Figure 5 To illustrate Figure 3 An example diagram of a variation of the illustrated neural processor 100.

[0230] Figure 5 The illustrated neural processor 100 and Figure 3The neural processor 100 shown in the figure is substantially the same as the neural processor 100 shown in the figure except for the plurality of processing units 110, and thus the following description is not repeated for the sake of convenience.

[0231] Figure 5 The plurality of processing units 110 shown in the figure can further include a plurality of register files RF1-RF12 corresponding to the plurality of processing units PE1-PE12, in addition to the plurality of processing units PE1-PE12.

[0232] Figure 5 The plurality of processing units PE1-PE12 and the plurality of register files RF1-RF12 shown in the figure are only examples for the sake of convenience, and the number of the plurality of processing units PE1-PE12 and the plurality of register files RF1-RF12 is not limited.

[0233] The size or number of the plurality of processing units 110 can be determined according to the number of the plurality of processing units PE1-PE12 and the plurality of register files RF1-RF12. The plurality of processing units 110 and the plurality of register files RF1-RF12 can be implemented in the form of an N x M matrix. Here, N and M are integers greater than 0.

[0234] The array size of the plurality of processing units 110 can be designed in consideration of the characteristics of the neural network model to be executed by the neural processor 100. Further, the memory size of the register file can be determined in consideration of the data size of the neural network model to be executed, the required execution speed, the required power consumption, and the like.

[0235] The register files RF1-RF12 of the neural processor 100 are static memory units directly connected to the processing units PE1-PE12. The register files RF1-RF12 can be configured by, for example, flip-flops and / or latches. The register files RF1-RF12 can be configured to store MAC operation values of the corresponding processing units PE1-PE12. The register files RF1-RF12 can be configured to provide or obtain the NPU internal memory 120 and weight value data and / or node data.

[0236] The register files RF1-RF12 can also be configured to perform the function of a temporary memory of an accumulator at the time of MAC operation.

[0237] Technical difficulties discovered by the inventors of the present disclosure The neural processor 100 specialized for AI operation to accelerate AI operation can have various optimized circuit structures on hardware. In contrast, the conventional neural network model is a neural network model trained without taking into account the hardware characteristics of the neural processor 100. That is, the conventional neural network model is trained without considering the hardware limitations of the neural processor 100. Therefore, when processing the conventional neural network model, the processing performance of the corresponding neural processor 100 can not be optimized. For example, the low processing performance can be caused by inefficient memory management and the massive amount of operation of the neural network model. Therefore, the conventional neural processor 100 processing the conventional neural network model can have problems of high power consumption or low operation processing speed.

[0238] <Disclosure of the Patent> The neural network model optimization apparatus 1500 according to an example of the disclosure is configured to optimize the neural network model using the structure data of the neural network model or the hardware characteristic data of the neural processor 100.

[0239] Therefore, the optimized neural network model has the effect of obtaining relatively higher performance and lower power consumption when processed in the neural processor 100 compared to the unoptimized neural network model.

[0240] The neural network model running in the neural processor 100 can be processed in the corresponding dedicated circuit part of the neural processor 100 at each step, and the input / output parameters processed for each dedicated circuit part are subjected to quantization and dequantization, which has the effect of reducing power consumption of the neural processor 100, improving processing speed, reducing memory bandwidth, reducing inference accuracy degradation, etc.

[0241] The neural network model optimization apparatus 1500 can be configured to optimize the neural network model for the neural processor 100.

[0242] Figure 6 An example diagram of the neural network model optimization apparatus 1500 and the edge device 1000 according to an example of the disclosure.

[0243] As shown in the figure, the neural network model optimization apparatus 1500 is an independent external system, which means an apparatus configured to optimize the neural network model used by the neural processor 100a in the edge device 1000 according to an example of the disclosure.

[0244] Therefore, the neural network model optimization apparatus 1500 can also be referred to as a dedicated neural network model simulator or a neural network model simulator for the neural processor 100a in the edge device 1000.

[0245] The edge device 1000 can include the neural processor 100a, the memory 200a, the CPU 300a, and the interface 400a.

[0246] Also, the neural network model optimization apparatus 1500 can include a neural processing unit (NPU) or a graphics processing unit (GPU) 100b, a memory 200b, a CPU 300b, and an interface 400b.

[0247] The neural network model optimization apparatus 1500 can communicate with the neural processing unit 100a within the edge device 1000. To this end, the interface 400b of the neural network model optimization apparatus 1500 can establish a connection or a session with the interface 400a of the edge device 1000. The interface can be an interface based on IEEE 802.3 for a wired local area network or IEEE 802.11 for a wireless local area network. Alternatively, the interface can be a peripheral component interconnect express (PCIe) interface-based interface or a personal computer memory card international association (PCMCIA)-based interface. Alternatively, the interface can be a universal serial bus (USB)-based interface. However, examples of the disclosure are not limited to a specific interface, and various interfaces can be applied.

[0248] The neural network model optimization apparatus 1500 can optimize the neural network model to be driven by the neural processing unit 100a within the edge device 1000. To this end, the neural network model optimization apparatus 1500 can obtain the neural network model from the edge device 1000. Alternatively, the neural network model optimization apparatus 1500 can also be configured to independently receive the neural network model to be driven by the neural processing unit 100a within the edge device 1000 from the outside.

[0249] After obtaining the neural network model to be driven by the neural processing unit 100a within the edge device 1000, the neural network model optimization apparatus 1500 can store the neural network model in the memory 200b within the neural network model optimization apparatus 1500.

[0250] When the received neural network model is generated by a specific machine learning framework software, the neural network model can not be directly run in the edge device 1000. Accordingly, the compiler 300b-10 of the neural network model optimization apparatus 1500 can be configured to compile the neural network model so as to generate machine code that can be run in the neural processing unit 100a of the edge device 1000.

[0251] The CPU 300b in the neural network model optimization apparatus 1500 can drive a compiler 300b-10. The compiler 300b-10 can be a semiconductor circuit or software stored in the memory 200b and run by the CPU 300b. The compiler 300b-10 can be one software or multiple software interacting with each other. For example, specific sub-modules of the compiler 300b-10 can be included in a first software, and the remaining sub-modules can be included in a second software.

[0252] The compiler 300b-10 can optimize and compile a neural network model stored in the memory 200b according to the neural processor 100a of the edge device 1000.

[0253] To optimize the neural network model, the neural network model optimization apparatus 1500 can be configured to analyze the neural network model to be optimized.

[0254] Specifically, the neural network model can be analyzed in the compiler 300b-10 of the neural network model optimization apparatus 1500.

[0255] The neural network model optimization apparatus 1500 can analyze parameter information of each layer of the neural network model. The neural network model optimization apparatus 1500 can analyze the size of weight value parameters and feature map parameters of each layer. The neural network model optimization apparatus 1500 can analyze the connection relationship between each layer. The neural network model optimization apparatus 1500 can analyze the size of input parameters and output parameters of each layer. The parameters of a multi-dimensional matrix can be referred to as tensors. The neural network model optimization apparatus 1500 can analyze function modules applied in each layer. The neural network model optimization apparatus 1500 can analyze branch points of specific layers. The neural network model optimization apparatus 1500 can analyze merging points of specific layers.

[0256] Further, the neural network model optimization apparatus 1500 can analyze non-graph-based function modules applied in each layer. Further, the neural network model optimization apparatus 1500 can be configured to convert non-graph-based function modules into graph-based modules.

[0257] For example, examples of non-graph-based functions included in each layer can be an addition function, a subtraction function, a multiplication function, a division function, a convolution, a matrix multiplication, a slice function, a concatenation function, a tensor view function, a reshape function, a transpose function, a softmax function, a permute function, a chunk function, a split function, a clamp function, a flatten function, a tensor mean function, and a sum function, etc. of each parameter. Further description, the functions can be provided as non-graph-based functions in a specific machine learning framework software. Among them, the neural network model optimization device 1500 can be configured to search for the non-graph-based functions.

[0258] A slice function can extract a part of a tensor. The slice function can be used to select a specific element or range on a specific dimension of a tensor.

[0259] A concatenation function can combine two or more tensors along a specified axis. The concatenation function can be used when connecting tensors to form a larger tensor, and can be mainly used to combine data along the configuration or feature dimension.

[0260] A tensor view function can re-determine the shape of a tensor without changing the data. The tensor view function can provide different representations of the same data to change the shape of the tensor to be compatible with various operations.

[0261] A reshape function can change the shape of a tensor. The reshape function is used to modify the dimensions of a tensor, and if the new shape is not compatible with the original data, the original data can be changed.

[0262] A transpose function can exchange the dimensions of a tensor. The transpose function can be mainly used to change the dimensions of a tensor in order to perform operations such as matrix multiplication.

[0263] A softmax function can convert a real number vector into a probability distribution. The softmax function can be mainly used to obtain class probabilities in the output layer of a neural network in a multi-class classification problem.

[0264] A permute function can change the dimensions of a tensor in a specified order. The permute function is similar to the transpose function, but can arbitrarily rearrange the dimensions.

[0265] A chunk function can split a tensor into a specified number of chunks along a specified dimension. A chunk function can be used when splitting a tensor into equal or specified sizes.

[0266] A split function can split a tensor into multiple tensors along a specified dimension. A split function is different from a chunk, which can provide more flexible functionality that can specify the final chunk size.

[0267] A clamp function can cut off tensor values within a specified range. A clamp function can be used to limit tensor values to a specific range in an optimization scenario.

[0268] A flatten function can convert a multi-dimensional tensor into a one-dimensional tensor. A flatten function can be mainly used when converting from a convolution layer to a fully connected layer in a neural network.

[0269] A tensor mean function can calculate the mean of a tensor along a specified dimension. A tensor mean function can be mainly used for normalization or data aggregation, which can be used when obtaining the mean of a tensor along a specific axis.

[0270] The neural network model optimization apparatus 1500 can be configured to further receive hardware data of the neural processor 100a within the edge device 1000. The hardware data of the neural processor 100a can include information of an internal memory 120 within the neural processor 100a (e.g., internal memory size, bit width of read / write operation of the internal memory, information of internal memory type / structure / speed), information of whether integer or floating point operation is possible, number of bits of integer that can be operated in the case of integer operation (e.g., int8, etc.), number of bits of floating point that can be operated in the case of floating point operation, operation frequency information, PE number information, type of special function unit, etc. However, the disclosure is not limited thereto.

[0271] To optimize the neural network model, the compiler 300b-10 can include Figure 7 The components shown. This will be described later.

[0272] The neural processor (NPU) or GPU 100b within the neural network model optimization apparatus 1500 can perform retraining of the neural network model according to Figure 7 the control of the compiler 300b-10 shown.

[0273] As described previously, when the compiler 300b-10 is implemented in software, the memory 200b within the neural network model optimization apparatus 1500 can store the software. The CPU 300b of the neural network model optimization apparatus 1500 can execute the software.

[0274] The memory 200b within the neural network model optimization apparatus 1500 can store a neural network model to be driven by the neural processor 100a within the edge device 1000. In addition, after the neural network model optimization apparatus 1500 performs optimization on the neural network model, the memory 200b within the neural network model optimization apparatus 1500 can store the optimized neural network model.

[0275] Figure 7 To show the details of the compiler 300b-10 in more detail, Figure 6 An example diagram of the compiler 300b-10 is shown.

[0276] Referring to FIG. 10, Figure 7 As can be seen, the compiler 300b-10 can include a first conversion part 300b-11, a graph generation part 300b-12, a mark adding part 300b-13, a calibration part 300b-14, a second conversion part 300b-15, an optimization part 300b-16, a third conversion part 300b-17, and an extraction part 300b-18. The optimization part 300b-16 can be selectively executed according to a compilation option.

[0277] Before describing the compiler 300b-10 of an example of the present disclosure, the difference between a non-graph-based neural network model and a graph-based neural network model will be described. A non-graph-based neural network model processes at least a part of the operation of each layer in a plurality of layers in a function call manner. The function call manner is a manner in which a predefined function is called and a corresponding input parameter is input to the function to process neural network operations. When designing a neural network model, the corresponding manner is convenient at the coding level.

[0278] However, in order to compile a non-graph-based neural network model (i.e., a first neural network model) so as to perform accelerated computation in a neural processor 100a of an edge device 1000, several technical problems must be solved.

[0279] First, a non-graph-based (i.e., function call manner) neural network model can not be compiled in a compiler for a neural processor 100a of a specific structure. That is, a compiler for a neural processor 100a of a specific structure can be designed to compile only a graph-based neural network model. That is, the compiler can not compile a function call manner neural network model. This is because a function call manner neural network model does not explicitly define the connection relationship of the operation steps of each layer. That is, the flow of each operation step of each layer of a non-graph-based (i.e., function call manner) neural network model (i.e., the connection relationship between each graph module) can not be explicitly defined. Specifically, a function call manner is only run when a function is called, and thus cannot track inputs and outputs outside the neural network model. If such a function call manner function is converted into a graph module, the graph module can be predefined. Therefore, the compiler 300b-10 can track the inputs and outputs of the graph modules of the neural network model to be compiled. In addition, for the graph module, the function of the inheritance module class can be predefined, and thus the graph modules can be connected to generate a directed acyclic graph (DAG).

[0280] Second, for a neural processor 100a of an edge device 1000, the capacity of the internal memory (on-chip memory) can be limited, and in an operating scenario with a small memory capacity, the cache efficiency of data can have a greater impact on the performance of the edge device 1000. That is, when a neural network model that does not pre-analyze the connection relationship between each operation step is compiled, the data cache efficiency in the neural processor 100a of the edge device 1000 can be low. When the cache efficiency is low, the amount of data transmission between the NPU internal memory 120 and the main memory 200a of the neural processor 100a of the edge device 1000 can be unnecessarily increased (e.g., repeated copying of data, unnecessary data movement, deletion of data for subsequent use, etc.).

[0281] For the graph-based neural network model (i.e., the second neural network model) of an example of the present disclosure, the connection relationship between each layer thereof can be explicitly analyzed by using the graph module converted in the first conversion unit 300b-11 of the compiler 300b-10. For example, the compiler 300b-10 can analyze the connection relationship that the output data of the first layer of the general neural network model is used as the input data of the second layer connected with the first layer. In addition, a series of operation steps included in each layer internal can also be represented by the graph module, so that the connection relationship in each layer internal can also be explicitly defined. Therefore, the compiler 300b-10 can use the connection relationship in the compiling step to maximize the memory management and optimization (e.g., cache efficiency) of the internal memory 120 of the NPU of the neural processing unit 100a of the edge device 1000. Further description, the compiler 300b-10 can determine the job scheduling of the neural processing unit 100a processing a specific neural network model based on the connection relationship in the compiling step.

[0282] Therefore, in order to maximize the acceleration of the operation of the neural network model in the neural processing unit 100a of the edge device 1000, it is necessary to convert the non-graph-based neural network model into the graph-based neural network model. In addition, compared with compiling the non-graph-based neural network model, compiling the graph-based neural network model can reduce the number of unpredictable situations at the time of compiling, so that the graph-based neural network model can be more efficiently compiled than the function call manner neural network model.

[0283] The following describes a method of quantizing the parameters of the neural network model after the function call manner neural network model is converted into the graph-based neural network model by the compiler 300b-10.

[0284] First, the first conversion unit 300b-11 is configured to receive the first neural network model. At this time, at least one layer of the first neural network model can include at least one function call instruction. That is, the first neural network model can be a neural network model including at least one function call instruction. Wherein, the compiler 300b-10 is configured to perform a series of steps of optimizing the first neural network model.

[0285] The first conversion unit 300b-11 can convert a plurality of function call instructions in the first neural network model into corresponding graph modules. For the first conversion unit 300b-11, it will be described with reference to Figure 8 .

[0286] The first conversion section 300b-11 of one example of the present disclosure can be configured to receive a non-graph-based or graph-based first neural network model. The first neural network model can be a neural network model generated based on a first machine learning framework software. The first machine learning framework software can be a software configured to support both graph-based and non-graph-based neural network models.

[0287] The compiler 300b-10 of one example of the present disclosure can be a software configured to receive a non-graph-based neural network model and perform quantization after converting it into a graph-based neural network model. For example, the first neural network model can be a neural network model generated based on a machine learning framework software such as Pytorch™, TensorFlow™, etc. However, the present disclosure is not limited to a specific machine learning framework software.

[0288] According to one example of the present disclosure, the first conversion section 300b-11 can convert a plurality of operation functions within the first neural network model into corresponding graph modules. Accordingly, the compiler 300b-10 can connect the converted graph modules and configure a graph-based neural network model. Herein, the first conversion section 300b-11 can be configured to convert all function calls of the first neural network model into corresponding graph modules.

[0289] Then, the graph generation section 300b-12 can analyze the relationship (i.e., connection relationship) between the input and output of a plurality of modules within the first neural network model using the graph modules converted by the first conversion section 300b-11. Accordingly, the graph modules having analyzed the relationship can be connected to each other according to the relationship.

[0290] The graph generation section 300b-12 can generate a graph-based second neural network model based on the converted graph modules and the analyzed relationship. That is, the second neural network model can be generated based on the first neural network model. Specifically, based on the analyzed connection relationship of the graph modules converted in the first conversion section 300b-11, the graph generation section 300b-12 can generate a graph-based second neural network model connected by a plurality of graph modules. More specifically, the graph generation section 300b-12 can map at least one input of the plurality of modules to at least one output, thereby generating a second neural network model including a plurality of modules connected as a graph. Herein, the graph-based modules already applied in the first neural network model can be applied to the second neural network model without being converted again. The graph module can be referred to as a module. Accordingly, by configuring the second neural network model, the compiler 300b-10 can analyze a series of operation sequences that could not be analyzed in the first neural network model.

[0291] Non-graph-based function calls, for example, non-graph-based function call instructions can include addition functions, subtraction functions, multiplication functions, division functions, slicing functions, concatenation functions, tensor view functions, reshape functions, transpose functions, soft-max functions, permutation functions, blocking functions, split functions, clipping functions, flatten functions, tensor mean functions, and sum functions, etc.

[0292] The compiler 300b-10 receives a neural network model generated by the first machine learning framework software, converts non-graph-based function calls into corresponding graph modules, and connects the graph modules to each other according to analyzed relationships of the modules. Accordingly, the second neural network model can be expressed as a directed acyclic graph (DAG) in which the graph modules are connected to each other.

[0293] Figure 8 To illustrate in detail Figure 7 An example graph of the first conversion section 300b-11 is shown.

[0294] Referring to Figure 8 The first conversion section 300b-11 can convert a plurality of operation functions within the first neural network model into a plurality of graph-based modules (e.g., graph modules).

[0295] For example, a function call instruction of the first machine learning framework software shown on the left side can be converted into a graph module shown on the right side. Figure 8 Figure 8

[0296] Specifically, Figure 8 x = x1 + x2 on the left side is a function that is not defined in advance. Such a function is executed only when the function is called, and thus it is not possible to track inputs and outputs outside the neural network model.

[0297] On the other hand, Figure 8 The add(x1, x2) graph module on the right side is defined in advance, and thus it is possible to track inputs and outputs. In addition, for the graph module, since a function that inherits a module category is defined in advance, a graph can be generated, and can be configured to selectively add a tag to inputs and outputs as needed.

[0298] Further description, the first machine learning framework software includes basic four arithmetic operations and function call instructions, but is not stored in the smallest operation unit, but is accessed in the module unit. Thus, it is not possible to monitor the input and output of the smallest operation unit. However, if converted into a graph-based module, it is possible to monitor the input and output of all operations, and thus a graph can be generated. In other words, the difference between the function call method and the graph-based module is whether it is possible to monitor and track all operations.

[0299] Specifically, Figure 8 The graph of the first machine learning framework software shown on the left side includes​​ 、 、 and addition (+) operation. Among them 、 、 Although it is a graph module, however, the addition (+) operation is a function call manner. Therefore, the addition (+) operation can be converted into an add graph module. Conv means a convolution graph module. Bn means a batch-normalization graph module. Relu means a relu activation function graph module. A plurality of graph modules can be grouped, and the graph modules after grouping can also be referred to as group sub-graph modules. That is, the first conversion unit 300b-11 is configured to convert all function call instructions shown on the left into corresponding graph modules. Figure 8

[0300] Then, the mark adding unit 300b-13 can add marks for tracking to each module of the second neural network model. Through the marks added to the second neural network model, correction data can be collected in the input and output of each graph module. For the marks, it will be described later with reference to Figure 9a and Figure 9b . Exemplarily, the correction data can be used to reduce the inference accuracy degradation when quantizing the parameters of the second neural network model. The marks can be referred to as tracking modules. The correction data can be pre-prepared data or real-time input data.

[0301] Figure 9a To show an example of the mark adding unit 300b-13 shown in Figure 7 in detail.

[0302] The mark adding unit 300b-13 can add marks for tracking to each module of the second neural network model, that is, marks.

[0303] As can be seen from Figure 9a , marks can be added at the input end and the output end of the Relu module and the input end and the output end of the Conv module, respectively.

[0304] The marks added to each module can measure the input value and the output value.

[0305] Figure 9b To show another example of the mark adding unit 300b-13 shown in Figure 7 in detail.

[0306] As can be seen from Figure 9b , marks can be added at the input end and the output end of the Conv module. At this time, marks can also be added at the input end inputting the weight value parameters of the Conv module.

[0307] ​Then, a module that adds a marker to the second neural network model to collect correction data is referred to as a calibration section 300b-14. However, the marker can be selectively added to the modules in the entire graph module for which correction data is to be collected, and it can not be necessary to add the marker to all the graph modules. The marker can be added to the input and the output of each graph module, respectively. Thus, correction data can be obtained from the input and the output of each corresponding graph module, respectively. For example, the marker can be added to each graph module that uses a parameter quantized in the second neural network model.

[0308] Referring again to Figure 7 , the calibration section 300b-14 can input a correction dataset to the second neural network model to obtain correction data. The correction dataset can be, for example, a set of tens to hundreds of inference test images. The higher the relevance of the correction dataset to the dataset used to train the second neural network model, the more effective it will be.

[0309] For example, when the second neural network model is a neural network model trained for autonomous driving, it is preferable that the correction dataset also consists of a dataset related to autonomous driving. For example, when the second neural network model is a neural network model trained for target sensing of a camera of a drone, it is preferable that the correction dataset also consists of a dataset related to target sensing of a camera of a drone. For example, when the second neural network model is a neural network model trained to distinguish the gender of a person, it is preferable that the correction dataset also consists of a dataset related to the gender of a person. For example, when the second neural network model is a neural network model trained to discriminate a defect of a specific product, it is preferable that the correction dataset also consists of a dataset related to the product. For example, when the second neural network model is a neural network model trained to discriminate a vehicle license plate, it is preferable that the correction dataset also consists of a dataset related to a vehicle license plate. That is, the correction dataset can be a dataset corresponding to the inference purpose of the second neural network model.

[0310] When the correction dataset is input to the second neural network model, the calibration section 300b-14 can collect correction data (i.e., the input value and the output value of the graph module to which the marker is added) through each graph module to which the marker is added, respectively. That is, the correction data can be generated independently for each marker. That is, it should be understood that the correction data includes each correction data corresponding to a plurality of markers.

[0311] The calibration section 300b-14 of the first conversion section 300b-11 can collect values measured when the correction dataset is input to the second neural network model to generate correction data. That is, the number of correction data can correspond to the number of markers added in the second neural network model. For example, when a marker is added to the input and the output of one graph module, respectively, the correction data can be generated in a manner corresponding to each of the input and the output of the graph module.

[0312] The correction data obtained by inputting the correction data set into the second neural network model can be stored in the memory 200b. In addition, the correction data set can also be stored in the memory 200b. Therefore, each correction data collected in each graph module can be stored in the memory 200b. Therefore, the generation of the second neural network model correction data can be completed in the calibration unit 300b-14.

[0313] Then, the second conversion unit 300b-15 is configured to simulate the quantization of the parameters of the second neural network model. That is, the parameters of the second neural network model are floating points, but the results after the quantization of the parameters can be simulated. Exemplarily, the parameters of the second neural network model input into the second conversion unit 300b-15 can be 32-bit floating points. Among them, the parameters of the neural network model in the examples of the present disclosure can include feature maps, weight values, etc. The feature maps can be referred to as input feature maps and output feature maps, activation maps, etc. The input feature maps can be referred to as input parameters, and the output feature maps can be referred to as output parameters. The input feature maps can be referred to as input activation values, and the output feature maps can be referred to as output activation values. The output feature maps can become the input feature maps of the next layer, so the output feature maps and the input feature maps may, according to the circumstances, refer to substantially the same parameters. The weight values can also be referred to as kernels. When the neural network model is of the Transformer series, the parameters can be referred to as queries (Q), keys (K), and values (V), attention (Q, K, V), etc.

[0314] Therefore, the second conversion unit 300b-15 can calculate the corresponding quantized parameters for the floating point parameters of the second neural network model based on the correction data generated by the calibration unit 300b-14. The method of simulating the quantization of the parameters of the second neural network model is described in detail below.

[0315] The compiler 300b-10 can calculate the scale value for the quantization of the floating point parameters based on the correction data. The compiler 300b-10 can calculate the scale value and the offset value for the quantization of the floating point parameters based on the correction data. That is, the offset value can be reflected, or the offset value can be omitted.

[0316] More specifically, the scale value and the offset value can be calculated according to the following mathematical formula 1. Among them, the scale value and the offset value can be calculated respectively according to the correction data generated in each mark.

[0317] For example, the first scale value and the first offset value of a specific graph module connected to the first mark can be calculated based on the first maximum value, the first minimum value of the first correction data measured in the first mark, and the target quantization bit width.

[0318] For example, a second scale value and a second offset value of a specific graph module connected to the second mark can be calculated based on the second maximum value, the second minimum value and the target quantization bit width of the second correction data measured in the second mark.

[0319] For example, the first marker can be configured to collect the input values ​​of the first graph module, and the second marker can be configured to collect the output values ​​of the first graph module. That is, in the example above, a first scale value and a first offset value corresponding to the input values ​​of the first graph module can be calculated, and a second scale value and a second offset value corresponding to the output values ​​of the first graph module can be calculated. The calculation is described in detail below with reference to Mathematical Formula 1.

[0320]

Mathematical Formula 1

[0321] In Equation 1, `max` refers to the maximum value among the correction data collected at a specific marker, `min` refers to the minimum value, and `bitwidth` represents the target quantization bitwidth. That is, within a graph module, the quantization levels of the input and output can be the same or different. Furthermore, the quantization levels of different graph modules can be the same or different.

[0322] Therefore, the max and min values ​​of specific calibration data corresponding to a specific graph module can be input into mathematical formula 1. The scale value and offset value can be used to reduce inference accuracy degradation due to quantization errors when quantizing the parameters of the second neural network model (e.g., feature maps and weight values). Further, if quantization is performed using scale values ​​and offset values ​​reflecting the data distribution characteristics of a specific graph module, inference accuracy degradation due to quantization errors can be reduced. Further, if quantization is processed using scale values ​​and offset values ​​reflecting the data distribution characteristics in multiple graph modules included in the second neural network model, inference accuracy degradation due to quantization of the second neural network model can be reduced. Further, the collected calibration data may include at least one of the data distribution, minimum value, maximum value, and average value. The scale value can be pre-calculated for each graph module or dynamically calculated by determining the maximum and minimum values ​​of the parameters input to each graph module during inference operations at the NPU.

[0323] The scale value corresponding to the feature map can be called s. f The scale value corresponding to the weight value can be called s. w The offset value corresponding to the feature map can be called o. f The offset value corresponding to the weight value can be called o. w .

[0324] Next, we will describe mathematical formula 2, which can be used to describe the feature map parameters. fp Quantization is used to reflect the feature map parameters of the corrected data.int ).

[0325] [Math. 2]

[0326] wherein, feature int denotes a quantized feature map, feature fp denotes a feature map to be quantized, o f denotes an offset value of Math. 1 of the feature map to be quantized, s f denotes a scale value of Math. 1 of the feature map to be quantized, denotes a round and clip operation. Wherein, Q min means -2 n-1 , Q max means 2 n-1 -1, n means bit width. Thus, the feature map of the floating point reflecting the correction data can be quantized by Math. 2. However, feature int as a value after analog quantization, can be basically stored in the memory 200b in the form of floating point data. Further described, the value obtained by the calculation of Math. 2, although having a quantized integer value, can be basically processed in the form of floating point in the compiler 300b-10. That is, in the second conversion part 300b-15, feature int may be a pseudo integer. That is, feautre int basically denotes a value after quantization, however, the form of storage can be stored in the memory 200b in the form of floating point.

[0327] Wherein, according to the input data, the feature map can further include outliers. Such outliers can become a cause of a large quantization error at the time of quantization. Thus, it is preferable to appropriately compensate the outliers. For example, the outliers can be compensated by applying a moving average algorithm to the correction data. If the moving average algorithm is applied to each correction data, the minimum value and the maximum value of the outliers can be obtained. However, examples of the present disclosure are not limited thereto, and can be configured to compensate the outliers of the feature map by various compensation algorithms. That is, the outliers of the correction data can also be removed at the time of quantization to reduce the influence of the outliers of the feature map. Thus, according to an example of the present disclosure, each correction data corresponding to the feature map using Math. 1 and Math. 2 can include the max value and the min value of the outliers compensated. Thus, the feature map can be an input value (for example, an input feature map) or an output value (for example, an output feature map) of a corresponding graph module.

[0328] The quantized feature map can be stored in the memory 200b.

[0329] Next, a description is made of mathematical expression 3 that can quantize the weight value parameter (weight fp ) to reflect the weight value parameter (weight int ) of the correction data.

[0330]

Mathematical expression 3

[0331] where weight int represents the quantized weight value, weight fp represents the weight value of the floating point to be quantized, s w represents the scale value of mathematical expression 1 of the weight value of the floating point to be quantized, represents a round and clip operation. Here, Q min means -2 n-1 , Q max means 2 n-1 -1, and n means the bit width. Thus, the weight value parameter reflecting the correction data can be quantized by mathematical expression 3. However, weight in t as a value after analog quantization can be basically stored in the memory 200b in the form of a floating point data. That is, the value calculated by mathematical expression 3, although having an integer value after quantization, can be basically processed in the form of a floating point in the compiler 300b-10. That is, weight int may be a pseudo integer in the second conversion section 300b-15. That is, weight int may basically represent a value after quantization, but the form of storage can be a floating point stored in the memory 200b.

[0332] The quantized weight value can be stored in the memory 200b.

[0333] Further description is made that the second neural network model can include a plurality of layers each including at least one graph module. At this time, when the plurality of graph modules are connected to each other, the quantization error is accumulated each time a graph module is passed. Thus, the more complex the structure of the second neural network model and the more the number of layers increases, the more the quantization based on mathematical expression 1 to mathematical expression 3 can reduce the accumulation of deterioration of inference accuracy of the second neural network model due to the quantization error. That is, in analyzing the data distribution and quantizing the floating point parameter to an integer parameter, deterioration of inference accuracy of the second neural network model due to quantization can be reduced.

[0334] The quantization of the correction data generated by analyzing the data distribution according to an example of the disclosure can be referred to as clipping quantization. The clipping quantization using mathematical formula 1 to mathematical formula 3 can quantize within the effective data distribution using the maximum value (max) and the minimum value (min) of the correction data. In particular, when there are outliers that can affect the accuracy of quantization, clipping quantization is useful. The compiler 300b-10 can perform the quantization in a manner that selectively processes the outliers of the feature map.

[0335] Figure 10 An example graph to show why it is important to accurately determine the scale value and the offset value.

[0336] Referring to Figure 10 , the X-axis represents the degree of outliers. The interval of the outliers of 0 means the global minimum value of the loss value. The further the degree of outliers from the global minimum value, the more the loss of the quantized neural network model increases. Using mathematical formula 1 to mathematical formula 3, the floating-point parameter of the second neural network model is quantized to a value of a certain bit width (A point of Figure 10 , the likelihood that the value of the quantized parameter becomes a value relatively close to the global minimum value increases. If quantization is not performed using mathematical formula 1 to mathematical formula 3, the quantized value becomes a value B relatively far from the global minimum value, rather than a value A relatively close to the global minimum value. In other words, the quantization error is greater than the case where mathematical formula 1 to mathematical formula 3 are used.

[0337] After the parameter quantization calculation of the second neural network model is completed, the second conversion part 300b-15 can remove the markers added for tracking from the second neural network model. That is, the markers added in the second neural network model can be deleted from the second conversion part 300b-15 after the correction data is obtained through the calibration part 300b-14. That is, if the parameters quantized based on the correction data are obtained, the markers are no longer needed in the second neural network model. However, examples of the disclosure are not limited thereto.

[0338] Referring again to Figure 7 , the optimization part 300b-16 can perform optimization with respect to the quantized parameters calculated by the second conversion part 300b-15. If the optimization part 300b-16 performs optimization on the quantized parameters (e.g., the scale value and / or the offset value), the second conversion part 300b-15 can generate a third neural network model including quantized weight value parameters in an integer format based on the optimized scale value and the optimized offset value, and based on the second neural network model.

[0339] Figure 11 To show in detail Figure 7An example diagram of the optimization unit 300b-16.

[0340] The compiler 300b-10 according to an example of the present disclosure can selectively optimize the parameters of the neural network model in the optimization unit 300b-16 according to a compilation option. In an example, the optimization unit 300b-16 can optimize one or more parameters included in the second neural network model in order to prevent inference accuracy degradation caused by the quantization simulation on the graph-based second neural network model.

[0341] The second conversion unit 300b-15 can calculate the corresponding quantized parameters for the floating-point parameters of the second neural network model based on the correction data generated by the calibration unit 300b-14. The compiler 300b-10 can selectively optimize the quantized parameters in the optimization unit 300b-16 according to a compilation option.

[0342] The optimization unit 300b-16 can include at least one of an outlier mitigation unit 300b-16a, a parameter refinement unit 300b-16b, a layer-wise training unit 300b-16c, a quantization aware retraining unit (QAT) 300b-16d, a quantization aware self-distillation unit (QASD) 300b-16e, and a pruning unit 300b-16f. The components of the optimization unit 300b-16 can be independently executed, and the execution results of the components do not affect other components. In various embodiments, the execution order of the components of the optimization unit 300b-16 can be adaptively determined, for example, such that the parameter refinement can be performed after the outlier mitigation is performed.

[0343] The optimization unit 300b-16 can optimize the parameters of the second neural network model using at least one of the outlier mitigation unit 300b-16a, the parameter refinement unit 300b-16b, the layer-wise training unit 300b-16c, the pruning unit 300b-16f, and the retraining unit 300b-16g according to a compilation option. The compilation option can be selected based on hardware information of the neural processing unit 100a of the edge device 1000. In addition, the compilation option can be selected corresponding to parameter characteristics (e.g., size of weight values, size of feature maps, etc.) and inference accuracy degradation characteristics of the second neural network model.

[0344] The outlier mitigation unit 300b-16a can be associated with a graph module including a multiply and accumulation (MAC) operation (e.g., a convolution or a matrix multiplication operation), and weight value parameters included in input parameters are weighted according to a degree of mitigation of outliers while the outliers are mitigated. For example, the outlier mitigation unit 300b-16a can calculate a parameter constant for adjusting outliers with respect to input values of a first graph module of a second neural network model and weight values of the first graph module, multiply the input values of the first graph module by an inverse of the parameter constant, and multiply the weight values of the first graph module by the parameter constant, thereby causing a part of the outliers included in the input values to be shared to the weight values. The outlier mitigation unit 300b-16a does not remove the outliers, but shares a degree of the outliers between parameters of a MAC operation, and thus the MAC operation result can be considered to include the outliers even if quantization is finally performed on the parameters.

[0345] The following describes in detail an operation of the outlier mitigation unit 300b-16a of an example of the present disclosure.

[0346] The outlier mitigation unit 300b-16a can mitigate outliers of each of the operation objects while the MAC operation result remains unchanged by partially transferring the outliers included in the operation objects of the MAC operation between the operation objects. In an example, this is the same as converting the operation to At this time, the is referred to as an adjustment value for adjusting the outliers.

[0347] The outlier mitigation unit 300b-16a can calculate the adjustment value based on first correction data collected from the input parameters and the weight value parameters using a flag added to each graph module. In an example, the outlier mitigation unit 300b-16a can perform 50 times of calibration to collect the first correction data using the flag added to each graph module. In an example, the outlier mitigation unit 300b-16a can calculate the adjustment value using a maximum value of the input parameters and a maximum value of the weight value parameters. The adjustment value is used to adjust a data range, and the outlier mitigation unit 300b-16a can calculate a maximum value of absolute values of the input parameters and a maximum value of absolute values of the weight value parameters to calculate a maximum positive number.

[0348] A form of the adjustment value can be determined according to a form of the operation objects. For example, when the operation objects are matrices, the adjustment value can also be a matrix. When a first operation object is an M*I matrix and a second operation object is an I*N matrix, an adjustment value matrix 1*I can be generated with respect to a channel I.

[0349] ​For example, when Activation A is a 2*4 matrix and Weight W is a 4*3 matrix, it is assumed that an operation included in the graph module is a convolution operation, and A and W correspond to operation objects of the convolution operation.

[0350] The outlier mitigation part 300b-16a can show the maximum value among the absolute values of each channel with respect to the first operation object and the second operation object of the MAC operation. In the aforementioned example, the maximum value set of each channel of the A matrix can be {Amax1, Amax2, Amax3, Amax4}. For example, the maximum value set of each channel of the W matrix can be {Wmax1, Wmax2, Wmax3, Wmax4}.

[0351] In an example of the disclosure, the adjustment value can be found as shown in the mathematical formula In the mathematical formula, adPi means the value of the adjustment value of channel i, Amaxi means the maximum value among the absolute values of all elements of channel i of the input parameter, and Wmaxi means the maximum value among the absolute values of all elements of channel i of the weight value parameter. However, examples of the disclosure are not limited to the mathematical formula, and the adjustment value can be determined using various formulas.

[0352] The outlier mitigation part 300b-16a can multiply the input parameter of the first graph module including the MAC operation by the reciprocal of the adjustment value (for example, the first adjustment value) and multiply the weight value parameter of the first graph module by the adjustment value (for example, the second adjustment value) so as to optimize the input parameter and the weight value parameter from the perspective of reducing quantization error based on the adjustment value for adjusting the outlier of each graph module of the second neural network model.

[0353] In an example, the outlier mitigation part 300b-16a can perform optimization of the input parameter and the weight value parameter of the first graph module before executing the first graph module. The outlier mitigation part 300b-16a can not add additional operations, but reflect the adjustment value through the multiplication operation performed before the first graph module so that the parameter optimization operation is executed together with the original operation.

[0354] In an example, in the previous step of the first graph module, a layer normalization graph module can be further included. The layer normalization step can include a multiplication operation, and the adjustment value can be reflected using the multiplication operation included in the layer normalization without adding additional multiplication operations. Accordingly, the layer normalization graph module can perform an operation of multiplying the input parameter by the first adjustment value. Then, in the first graph module, an operation of multiplying the weight value parameter reflecting the second adjustment value can be performed. For example, when a graph included in a normalization layer configured before the MAC operation includes a function of , the variable , and the variable multiply the first adjustment value , using and a modification function. In this case, the variables and are constants, and thus can be calculated in the optimization unit 300b-16 and stored as constant parameters. At this time, resource waste that occurs when multiplication operations for optimizing parameters are independently performed (e.g., input parameters multiplied by the first adjustment value) can be reduced. In addition, multiplication operations of the second adjustment value and the weight value parameter can be performed and stored as constant parameters. In this case, resource waste that occurs when multiplication operations for optimizing parameters are independently performed can be reduced.

[0355] In various examples, the outlier mitigation unit 300b-16a can reflect the parameter optimization operation in a predetermined multiplication operation before the operation of the graph module. In yet another example, when a MAC operation (e.g., a matmul operation) is not included in the graph module or a multiplication operation is not included in the previous step of the graph module, the parameter optimization operation can not be performed, considering the cost that occurs when multiplication operations for optimizing parameters are independently performed.

[0356] The input parameters and the weight value parameters can be optimized by applying adjustment values to reduce the quantization error of outliers. Each adjustment value (e.g., the first adjustment value and the second adjustment value) can be calculated in a compilation step of a neural network model and stored as a constant parameter. In particular, it is preferable to calculate and store as a constant parameter in advance in order to reduce the power consumption of inference operations of a neural processing unit and to improve the inference speed.

[0357] The outlier mitigation unit 300b-16a can multiply each element of the input parameters by the reciprocal of the adjustment value to optimize the input parameter value. In the foregoing example, the calculation can be performed as shown in A 11 *(adP1) -1 =A' 11 , A 21 *(adP1) -1 =A' 21 , A 12 *(adP2) -1 =A' 12 , etc. The calculation can be performed in a layer normalization step performed before the graph module. The outlier mitigation unit 300b-16a can reflect the parameter optimization operation in a multiplication operation included in the layer normalization step before each graph module. The parameter optimization multiplication operation is included in the original multiplication operation, and thus does not incur additional operation costs. That is, the outlier adjustment of the input parameters has the advantage that, when the layer normalization is located before the MAC operation, the variable The third neural network model can be provided without an additional inference resource. Thus, the third neural network model to which the outlier mitigation value generated in the third conversion unit 300b-17 is applied basically does not require an additional resource for mitigating outliers.

[0358] The outlier mitigation unit 300b-16a can multiply each element of the weight value parameter by the adjustment value to optimize the weight value parameter value. In the aforementioned example, the calculation can be performed as shown in W 11 *adP1=W' 11 , W 12 *adP1=W' 12 , W 21 *adP2=W' 21 .

[0359] In an example of the present disclosure, the input parameter and the weight value parameter to which the outlier mitigation is applied can be applied in the quantization step and the steps thereafter. For example, when the outlier mitigation unit 300b-16a performs the outlier mitigation with respect to the second neural network model by means of the optimization unit 300b-16, the input value feature_in int may mean that the outlier mitigation is applied.

[0360] In an example of the present disclosure, the outlier mitigation unit 300b-16a needs to obtain correction data for mitigating outliers, at this time, a unit independent of the calibration unit 300b-14 can be further included. The input value and the weight value collected in the markers included in each graph module can be obtained as the correction data using any correction data in the correction data set. The correction data generated by the calibration unit 300b-14 can be used to calculate the scale value and the offset value of each parameter in the second conversion unit 300b-15. The outlier mitigation unit 300b-16a can mitigate the outliers of the input parameter and the weight value parameter independently of the operation of the second conversion unit 300b-15. The optimization unit 300b-16 can perform the parameter refinement after performing the outlier mitigation, and both the outlier mitigation and the parameter refinement can be reflected in the quantization simulation of the second neural network model. When the outlier mitigation is performed, the quantization simulation process of the second neural network model and the input parameter in which the outliers are mitigated in the process can be reflected. That is, the third conversion unit can generate the third neural network model based on the quantization simulation of the second neural network model using the input parameter and the weight value parameter in which the adjustment value in which the outliers are mitigated is reflected. After the adjustment value in which the outliers are mitigated is determined, the third conversion unit can reflect each adjustment value to the input parameter and the weight value parameter of the corresponding neural network model.

[0361] The parameter refinement unit 300b-16b can perform optimization on parameters required for the quantization process to reduce errors that can occur as a result of quantization and improve computational performance resulting from quantization while maintaining the accuracy of the neural network model. The parameter refinement unit 300b-16b can calculate optimal values for each of the scale value and the offset value for quantization of floating-point parameters of the neural network model.

[0362] The following describes the operation of the parameter refinement unit 300b-16b of an example of the present disclosure in detail.

[0363] The parameter refinement unit 300b-16b can calculate optimal values for each of the scale value and the offset value for quantization of floating-point parameters calculated by the second conversion unit 300b-15. For ease of description, the scale value calculated by the second conversion unit 300b-15 is referred to as Scaledefault, and the offset value calculated by the second conversion unit 300b-15 is referred to as Offsetdefault.

[0364] Cosine similarity is a measure of similarity between two vectors in an inner product space. Cosine similarity can be measured using the cosine of the angle between two vectors to determine whether the two vectors point in almost the same direction. The higher the cosine similarity of the output values when quantization is not performed and when quantization is performed, the smaller the quantization error, and thus the final inference accuracy of the neural network model can be determined to be maintained, according to the parameter refinement unit 300b-16b. That is, the parameter refinement unit 300b-16b can use the cosine similarity of the output values when quantization is not performed and when quantization is performed as a reference to perform optimization on the scale value and the offset value for performing quantization. The parameter refinement unit 300b-16b can calculate optimal values for each of the scale value Scaledefault calculated by the second conversion unit 300b-15 and the offset value Offsetdefault calculated by the second conversion unit 300b-15. In an example, the parameter refinement unit 300b-16b can select an optimal value from among the peripheral values of the scale value Scaledefault calculated by the second conversion unit 300b-15. In addition, the parameter refinement unit 300b-16b can select an optimal value from among the peripheral values of the offset value Offsetdefault calculated by the second conversion unit 300b-15.

[0365] The second neural network model can include a plurality of layers, each layer including at least one graph module. The compiler 300b-10 can calculate a scale value and an offset value for a specific graph module connected to a marker based on correction data measured in the marker added to each graph module. Referring to Figure 9b Markers are added to the input and output ends of the Conv module and the input end for inputting a weight value parameter, respectively, and a scale value and an offset value can be calculated based on correction data measured in each marker, respectively.

[0366] For example, the first scale value and the first offset value of the input parameter of the Conv module can be calculated using Mathematical Formula 1 based on the first maximum value, the first minimum value of the first correction data, and the target quantization bit width, wherein the first maximum value, the first minimum value of the first correction data are measured from the first marker added to the input end of the Conv module. Figure 9b

[0367] For example, the second scale value of the weight value parameter of the Conv module can be calculated using Mathematical Formula 1 based on the second maximum value, the second minimum value of the second correction data, and the target quantization bit width, wherein the second maximum value, the second minimum value of the second correction data are measured from the second marker added to the input end for the weight value parameter of the Conv module. Figure 9b

[0368] For example, Figure 9b The output parameter of the Conv module can be calculated by the first scale value and the second offset value of the input parameter of the Conv module and the second scale value of the weight value parameter. The output of the Conv module is in an integer form, and the scale value and the offset value of the dequantization thereof can be obtained as the first and second scale values / offset values. The output value of the Conv module after the dequantization is the input value of the next module, and thus corresponds to the first scale value of the next module.

[0369] The parameter refinement unit 300b-16b can perform optimization for the first scale value and the first offset value of the input parameter of the Conv module and the second scale value of the weight value parameter of the Conv module, respectively. The output parameter of the Conv module corresponds to the input parameter of the next module connected to the Conv module, and thus optimization can be performed in the next module.

[0370] The optimization unit 300b-16 can selectively perform outlier mitigation and parameter refinement according to a compilation option.

[0371] In an example, when only the outlier mitigation is performed, the outlier mitigation unit 300b-16a can perform the outlier mitigation on the quantization parameter based on the correction data before the quantization parameter is calculated by the second conversion unit 300b-15.

[0372] ​​In an example, when only parameter refinement is performed, the parameter refinement unit 300b-16b can perform optimization on the quantization parameters after the quantization parameters are calculated by the second conversion unit 300b-15. However, when there is an abnormal value in the parameters, if the scale value and the offset value are calculated according to the mathematical formula 1 using the maximum value and the minimum value of the correction data, it is easy to include errors caused by the abnormal value. When the abnormal value mitigation and the parameter refinement are both performed, the optimization unit 300b-16 can perform the parameter refinement after performing the abnormal value mitigation.

[0373] In an example, the optimization unit 300b-16 can sequentially perform the following operations in order to optimize the parameters: 1) while the abnormal values included in the input parameters are mitigated by the abnormal value mitigation unit 300b-16a, the weight value parameters are weighted according to the mitigation degree of the abnormal values, 2) the quantization parameters (the scale value and the offset value) are calculated by the second conversion unit 300b-15 based on the correction data using the mathematical formula 1, and 3) optimization is performed on the parameters (for example, the input parameter scale value, the input parameter offset value, and / or the weight value parameter scale value) calculated by the parameter refinement unit 300b-16b.

[0374] The parameter refinement unit 300b-16b can optimize the corresponding scale value or offset value for the quantization parameters of each graph module of the second neural network model.

[0375] In an example, the parameter refinement unit 300b-16b can determine the optimal value of the scale value or the offset value in the order from the first graph module to the last graph module based on the connection relationship between the graph modules included in the second neural network model. For example, the parameter refinement unit 300b-16b can optimize the offset value in order from the first graph module determined by the connection relationship of the graph modules to the last graph module for the plurality of graph modules included in the second neural network model. The optimization order can be one of forward, reverse, or a specific order. After the optimization of the offset value is completed, the parameter refinement unit 300b-16b can optimize the scale value in order from the first layer to the last layer. The optimization order can be one of forward, reverse, or a specific order.

[0376] In an example, the parameter refinement unit 300b-16b can perform optimization for a part of the connected graph modules. For example, the parameter refinement unit 300b-16b can perform optimization for the first graph module, not perform optimization for the second graph module, and perform optimization for the third graph module among all the connected graph modules. The parameter refinement unit 300b-16b can perform parameter refinement for all the graph modules using the same method.

[0377] The parameter refinement unit 300b-16b can experimentally select an optimization order. In an example, the parameter refinement unit 300b-16b can determine an optimization order of a plurality of quantization parameters. The parameter refinement unit 300b-16b can optimize a scale value of a parameter after optimizing an offset value of the parameter. And / or the parameter refinement unit 300b-16b can optimize an input parameter before optimizing a weight value parameter. For example, the parameter refinement unit 300b-16b can, for a layer including an input activation map, a weight value map, 1) optimize an offset value of the activation map first, 2) then optimize a scale value of the activation map, and 3) finally optimize a scale value of the weight value map. The parameter refinement unit 300b-16b can determine an optimal value of an offset value first, and then determine an optimal value of a scale value for a second neural network model in which the optimal offset value is reflected in each of a plurality of layers.

[0378] The parameter refinement unit 300b-16b can select a surrounding value for a scale value and an offset value to be optimized, and generate an optimization candidate group. The parameter refinement unit 300b-16b can compare the optimization candidate group with a result value obtained by not performing quantization through a quantization simulation, and thus determine one of the optimization candidate group as an optimal value. That is, the parameter refinement unit 300b-16b can calculate a cosine similarity of an operation result value of each of the map modules of the second neural network model and each candidate included in the optimization candidate group, each of which performs a quantization simulation, and determine a candidate having the highest cosine similarity value among the candidates included in the optimization candidate group as an optimal value.

[0379] The parameter refinement unit 300b-16b can experimentally determine a candidate group of a scale value or an offset value to be optimized. The parameter refinement unit 300b-16b can select an optimal candidate group of a scale value to be optimized in a predetermined number within a surrounding, i.e., a predetermined range, including a scale value calculated using Mathematical Formula 1. In addition, the parameter refinement unit 300b-16b can select an optimal candidate group of an offset value to be optimized in a predetermined number within a predetermined range including an offset value calculated using Mathematical Formula 1.

[0380] In an example, the parameter refinement unit 300b-16b can select a candidate according to a search space within an under bound coefficient α and an upper bound coefficient β. The parameter refinement unit 300b-16b can select a candidate in a search space number within a range from Scale default * α to Scale default *β. The parameter refinement unit 300b-16b can select a candidate in a search space number within a range from Scale default * α to Scale default* The candidates are selected equally in the range of β by the number of search spaces. For example, when the scale S value is 3, α is 0.5, β is 2, and the search space is 10, the candidate group can be {1.5, 2, 2.5, 3, 3.5, 4, 4.5, 5, 5.5, 6}. In the case of the example, the scale S value is included in the candidate group, but depending on the case, the scale S can be included in the candidate group in the case where the scale S value is not included in the candidate group. For example, when the scale S value is 3, α is 0.5, β is 3, and the search space is 10, the candidate group can be {1.5, 2.33, 3, 3.16, 3.99, 4.82, 5.65, 6.48, 7.31, 8.14, 9}. The parameter refining section 300b-16b can generate the candidate group using a permutation generating function. For example, the parameter refining section 300b-16b can generate the candidate group using the np.linspace(scale* α, scale* β, search_space) function. In another example, the parameter refining section 300b-16b can determine the candidate group unequally among the peripheral values with the scale value or the offset value calculated by the second conversion section 300b-15 as a reference.

[0381] A specific method in which the parameter refining section 300b-16b optimizes the scale value of the current graph module is described. An example for description is as follows. For the parameter to be optimized, when the scale value Scale default is 3, α is 0.5, β is 2, and the search space is 10, the scale optimization candidate group is {1.5, 2, 2.5, 3, 3.5, 4, 4.5, 5, 5.5, 6}. The current scale value S l may be set to 0. The parameter refining section 300b-16b can use a part of the correction data set as input data for the optimization process. For example, when the correction data set includes 50 data, the parameter refining section 300b-16b can use optional 2 of the correction data set as input data for the optimization process. The parameter refining section 300b-16b can experimentally determine the kind and the number of the input data.

[0382] The parameter refining section 300b-16b can calculate the operation result value O l generated by a module (original module) in which quantization is not performed on the first input value in the input data.

[0383] The parameter refining section 300b-16b can calculate the operation result value O lThe module that performs the quantization simulation (Q-module) can be the second transformation unit 300b-15. The parameter refinement unit 300b-16b can calculate the parameter using the i-th candidate s. l i The results of performing the quantization simulation l i At this time, Ô l i For integer values, the cosine similarity can be calculated after dequantization in floating-point form. The specific method for performing dequantization on the quantization simulation results will be explained in mathematical formulas 7 to 8. Figure 13d It will be described in the detailed description.

[0384] The parameter refinement section 300b-16b calculates the results of the operation without quantization. l With the use of optimized candidate s l i The result of the quantization simulation. l i The cosine similarity, and compared with the current scale value S as the baseline. l Compare with the cosine similarity value (MAX) without quantization. When based on optimized candidates s l i The result of the operation Ô l i The result of the calculation without quantization O l When the cosine similarity is greater than the baseline value, the parameter refinement unit 300b-16b can adjust the current scale value S. l Updated to optimize candidate s l i The parameter refinement section 300b-16b can refine the next optimization candidate s. l i+1 The above process is repeated. The parameter refinement unit 300b-16b can repeatedly perform the above process for all candidates included in the optimization candidate group, calculating the scale value calculated by the second conversion unit 300b-15. default The optimal value.

[0385] The module (Q-module) that performs the quantization simulation can be a separate module other than the second conversion unit 300b-15. In this case, the separate module may include the following steps: quantizing each input value of each graph module using a scale and offset value, performing calculations for each graph module using the quantized input values, and then inverting the calculation results. That is, when it is a separate module, it may include the inverting quantization step along with the second conversion unit 300b-15.

[0386] The parameter refinement unit 300b-16b can repeatedly perform the above process for the second input value in the input data.

[0387] The parameter refinement unit 300b-16b can pass the second neural network model to the second conversion unit 300b-15, which is a model for a scale value Scale default The optimization is performed, and the optimized scale value is applied to each graph module connected based on the connection relationship of the graph module.

[0388] The operation of the quantization-aware retraining unit 300b-16d of an embodiment of the disclosure is described below in detail.

[0389] The quantization-aware retraining unit 300b-16d integrates quantization in the training step of the neural network model to reflect the quantization error in the neural network model and fine-tune the weight value. The quantization-aware retraining algorithm can include a loss function, gradient calculation, and optimization algorithm modification. The quantization-aware retraining unit 300b-16d can perform fine-tuning to compensate for the quantization error after quantizing the trained neural network model, in which the fine-tuning is retraining in a direction to minimize the loss caused by quantization.

[0390] To reduce the quantization error of the trained neural network model, the quantization-aware retraining unit 300b-16d can fine-tune the parameters so that the loss (the difference between the result value of the training data and the output value of the second neural network model) is minimized through the retraining process. The quantization-aware retraining unit 300b-16d can perform retraining to update only the parameter value for optimization after the second conversion unit 300b-15 performs quantization simulation on the second neural network model in the form of a directed acyclic graph. That is, to optimize the parameter value of the trained neural network model, the quantization-aware retraining unit 300b-16d can perform retraining and update the parameter value so that the difference (loss) between the result value (label) of the training data and the output value (output) of the second neural network model is minimized.

[0391] In an example of the disclosure, the second conversion unit 300b-15 can perform quantization simulation on the parameters of the second neural network model. As described above, the second conversion unit 300b-15 can perform quantization on the parameters of the second neural network model using Mathematical Formula 1 to Mathematical Formula 3.

[0392] In an example of the disclosure, the quantization-aware retraining part 300b-16d can find the optimal value of the weight value parameter for the neural network model including the quantized parameter using a gradient decent method. The gradient decent method is a method of repeatedly performing the following process gradually, i.e., starting from an initial weight value in the correlation between the weight and the cost, subtracting the gradient of the graph between the weight and the cost, i.e., the degree of loss caused by the change in the weight, in order to reach the point of minimum cost. Here, the cost can be the difference between the actual result value and the output value of the neural network model, i.e., the loss value.

[0393] The quantization-aware retraining part 300b-16d can update the weight value parameter value using the following mathematical formula 4 while performing retraining on the second neural network model on which quantization is performed.

[0394] [Mathematical Formula 4]

[0395] is the current weight value parameter value, is the loss value, is the updated weight value parameter value. is the learning rate. The learning rate represents the degree of retraining execution refinement, and the smaller the learning rate, the more refined the degree of change in the weight value parameter can be applied in the retraining process. There can be a trade-off between the retraining execution time and the size of the learning rate. Therefore, the quantization-aware retraining part 300b-16d can experimentally determine the learning rate according to the retraining execution time, or determine it according to the user's selected option. In an example, the quantization-aware retraining part 300b-16d can determine the degree of change in the current parameter, i.e., the learning rate, according to the user's option or the retraining execution time.

[0396] The quantization-aware retraining part 300b-16d can subtract the degree of loss caused by the change in the weight value from the current weight value, i.e., the gradient, and find a new weight value as shown in the mathematical formula 4. The quantization-aware retraining part 300b-16d can subtract the loss difference caused by the change in the current parameter and update the current parameter in each step of the quantization-aware retraining of the quantized second neural network model.

[0397] The quantization-aware retraining unit 300b-16d can determine the termination conditions for quantization-aware retraining. In one example, the quantization-aware retraining unit 300b-16d can pre-determine the target loss, and terminate the quantization-aware retraining when the pre-determined target loss (critical value) is reached. In another example, the quantization-aware retraining unit 300b-16d can terminate the quantization-aware retraining within a limited runtime. The runtime can also be set in epochs. The limited runtime can be pre-determined according to user options.

[0398] The Quantization-Aware Retraining Unit 300b-16d is used for computation on neural network models that have undergone quantization. The loss change calculation function can be applied to the calculation of the graph module, whereby the... This refers to the loss change of the weight value parameter.

[0399] First, we describe how to calculate the loss caused by changes in weight parameters in a neural network model without quantization.

[0400] For example, the input value of the first graph module Weight values The output value can be When the actual result value of the first graph module is called... At that time, the loss can be calculated as Forward computation is... This is used to confirm the degree of loss of the weight value parameters. The backward computation of ) can utilize according to Differential Defined as However, if the input value is... Weight values If quantization is performed, the extent of loss due to changes in weight parameters cannot be confirmed during reverse calculation.

[0401] Specifically, if quantization is performed on the first graph module, the input value is The weight value is .in, The input feature map parameters of the graph module are represented. This represents the scale value of the input feature map parameters. This represents the offset value of the input feature map parameters. This represents the weight value parameter of the graph module. This represents the scale value of the weight parameter. Furthermore, the forward calculation of the first graph module is... If the forward calculation is simply differentiated, the loss change of the weight parameters cannot be confirmed in the reverse calculation.

[0402] To address this problem, the quantization-sensing retraining unit 300b-16d of one embodiment of this disclosure, as shown in the following mathematical formula 5, uses the detach function as the loss change calculation function so that, although it is not actually calculated in the forward calculation, the loss change of the weight value parameters can be confirmed during the differentiation process of the backward calculation. That is, the detach function can be called the loss change calculation function.

[0403] The quantization-aware retraining unit 300b-16d can add a loss change calculation function to the forward calculation of each of the multiple graph modules included in the quantized second neural network model, corresponding to each added quantization module (Act Quant, Weight Quant). Based on the loss change calculation function, the quantization-aware retraining unit 300b-16d can confirm the changes in the output values ​​of each graph module caused by the changes in each parameter during the backward calculation of each of the multiple graph modules.

[0404]

Mathematical Expression 5

[0405] In mathematical formula 5, The input feature map parameters of the graph module are represented. represents the scale value of the input feature map parameters, o represents the offset value of the input feature map parameters, and w represents the weight value of the graph module. This represents the scale value of the weight parameter. For description, a substitution process is included in mathematical formula 5 to derive the result of the operation of the first graph module (Gemm function in Figure 12) from formula 1) which includes the detach function, i.e., 3). In equation 1) which includes the detach function, the input parameters are... Replace with The weight value parameter Replace with This can form equation 2). In equation 2), Replace with ,Will Replace with This can form equation 3).

[0406] Equations 1) to 3) above can be used to confirm the change in loss caused by changes in the scale value of the input feature map parameters, the offset value of the input feature map parameters, and the scale value of the weight parameters.

[0407] In one example, the quantization-aware retraining unit 300b-16d can, as shown in Equation 6 below, use the differentiation of Equation 5 to determine the scale value of the input feature map parameters. Offset values ​​of input feature map parameters Scale value of the weight parameter Changes in various losses caused by changes .

[0408]

Mathematical Expression 6

[0409] Mathematical formula 6 includes the loss function according to The differential equation 1). Mathematical expression 6 contains the derivative of mathematical expression 5, equation 3). and Differentiated expressions. Expression 6 contains expressions that will be differentiated from Expression 5. The corresponding formula is as follows and Differentiated expressions. Expression 6 contains expressions that will be differentiated from Expression 5. The corresponding formula is as follows and The formulas are differentiated separately. Mathematical formula 6 can be used to confirm the scale values ​​of the input feature map parameters based on processes 2) to 6). Offset values ​​of input feature map parameters Scale value of the weight parameter Changes in various losses caused by changes .

[0410] Referring again to mathematical formula 4, the quantization perception retraining unit 300b-16d can subtract the scale value of the current weight parameter from the current weight parameter value. Changes in losses caused by changes in [their] nature Then, update the next weight parameter value. At this point, when the degree of loss change reaches the predetermined target loss, the quantization perception retraining unit 300b-16d can end the retraining.

[0411] The pruning unit 300b-16f can prune the weight parameters of each layer of the second neural network model without compromising the inference accuracy of the second neural network model. In one example, after removing a portion of the weight parameters of each layer of the neural network model, the pruning unit 300b-16f can mark the corresponding components when the inference accuracy of the neural network model is within a critical range. The method for selecting which component to remove can vary. In one embodiment, it can be selected within a range of weight values, and then selected based on the location of the component, or it can be selected arbitrarily. The pruning unit 300b-16f can perform maximum pruning while gradually increasing the degree of pruning during the period when the inference accuracy of the neural network model is within a critical range. Alternatively, the pruning unit 300b-16f can gradually increase the degree of pruning until a predetermined degree of pruning is reached while the inference accuracy of the neural network model is within a critical range.

[0412] As an example, the pruning unit 300b-16f can perform magnitude-based pruning. The pruning unit 300b-16f can set a threshold value 'a' and set weights with absolute values ​​less than 'a' to 0, thereby performing pruning. The pruning unit 300b-16f can increase the pruning ratio while the neural network model's inference accuracy is within a critical range, while simultaneously resetting the threshold value 'a'.

[0413] As an example, the pruning unit 300b-16f can determine the importance of weight parameters and perform pruning. The importance of neural network weights can be determined by sensitivity or significance. Sensitivity represents the importance of a weight and can be measured using the magnitude of the derivative of the weight's loss function. The higher the sensitivity of a weight, the more important that weight is considered. The pruning unit 300b-16f can use the sensitivity of weights as a benchmark to identify important weights in the neural network and use this as a tool to prune the least important parts while minimizing the impact on network performance (inference accuracy). There are various methods for calculating sensitivity.

[0414] As an example, the pruning section 300b-16f can prune the weight parameters based on the change in loss. The loss change benchmark assesses the difference in loss of the neural network when specific weights are removed versus not removed, and measures the importance of the weights. The pruning section 300b-16f can utilize first-order Taylor expansion to measure the change in loss corresponding to smaller changes. First-order Taylor expansion is a useful mathematical tool when dealing with differentiable functions, and the pruning section 300b-16f can compute the change in the loss function within the neural network. Training can be performed in the direction that minimizes the loss function using gradient descent in the neural network, where the rate of change of the loss function plays a crucial role. For example, for the function... The first-order Taylor expansion can be expressed as . express The differential (derivative function). This represents a minimal change. (Regarding the loss function) , These represent the parameters of the model. Methods for calculating the rate of change of the parameters of the loss function include, for example,... The result is shown. The rate of change of. Among them, It is the loss function By parameters The value obtained by partial derivative, Indicates a very small amount of change. The rate of change of the loss function calculated through the above process can be used to update the parameters. If the rate of change is negative, it can be updated in the direction of reducing the loss, and if the rate of change is positive, it can be updated in the direction of increasing the loss. The pruning part 300b-16f can reduce the loss in the direction of reducing the loss, that is, reducing the degree of progressive pruning, or interrupting the pruning. The pruning part 300b-16f can repeatedly perform the progressive pruning again in the direction of increasing the loss, or increase the degree of pruning.

[0415] The components included in the compiler 300b-10 can be independently executed, respectively, and do not affect each other. For example, when n optimization components among the components included in the compiler 300b-10 are executed, the execution order can be determined in n! combinations. In various embodiments, as an example of maximizing optimization efficiency, outlier mitigation can be performed first, and parameter refinement can be performed next. After performing the pruning process, for each parameter, a flag or the like can be stored to store a flag corresponding to pruning, and at the time of final compilation, it can be made so that no operation is performed with respect to the components that are pruned and deleted. For example, zero skip processing can be performed with respect to the component values that are pruned and deleted. Since each parameter requires an additional bit for storing a flag indicating the result of pruning execution after pruning execution, in consideration of factors such as memory capacity, pruning can be determined to be executed last in the order of optimization within the compilation.

[0416] Figure 12a The operation of the layer-by-layer training part 300b-16c of one example of the present disclosure is described in detail.

[0417] The layer-by-layer training part 300b-16c of one example of the present disclosure can optimize one or more parameters of the neural network model through layer-by-layer training. The second conversion part 300b-15 can simulate quantization of the second neural network model. As described above, the second conversion part 300b-15 can quantize the weight value parameter (W fp ) to a weight value parameter (W int ) reflecting the correction data using Mathematical Formula 3. The layer-by-layer training part 300b-16c of one example of the present disclosure can minimize the quantization loss that can occur in a function that quantizes the parameter and optimize the parameter. In one example, the layer-by-layer training part 300b-16c can minimize the quantization loss that occurs due to the rounding operation included in Mathematical Formula 3. In another example, the layer-by-layer training part 300b-16c can optimize the weight value parameter by optimizing by element such that the quantization loss is minimized with respect to the scale included in Mathematical Formula 3. The layer-by-layer training part 300b-16c can optimize the parameter by updating at least a part (operation or variable, etc.) of the parameter quantization function so as to reflect the output value (output int ) of the layer in which quantization is performed layer by layer with respect to the plurality of layers included in the loop-free graph-based neural network model.int ) to minimize the difference (loss) from the output value (output fp ) of the layer for which quantization is not performed.

[0418] Referring to Figure 12a , for a graph module (Gemm function) included in the first layer, the layer input x fp and the weight value w fp The result of performing the operation within the graph module is y fp When quantization is performed, a quantization module can be added to the input value and the weight value, respectively. The layer input x fp becomes x int by means of the quantization module, and the weight value W fp becomes W int The layer-by-layer training part 300b-16c can update the rounding operation or the scale value within the weight value quantization module through learning so that the difference between the output value y fp of the layer for which quantization is not performed and the output value W int of the layer for which quantization is performed is minimized.

[0419] In an example, the second conversion part 300b-15, when simulating quantization of the second neural network model, the mathematical expression 3 in which the weight value parameter is quantized includes a rounding operation (round). The use of rounding when quantizing the weight value parameter can be the optimal method to reduce the quantization loss of the weight value itself, but rounding can not be considered the optimal method to reduce quantization error in terms of neural network model inference accuracy. The layer-by-layer training part 300b-16c of an example of the present disclosure can determine whether to round or round each element of the weight value parameter so that the quantization loss of the result value (output) predicted by the quantized neural network model is minimized. In an example, the layer-by-layer training part 300b-16c, when quantizing the neural network model, can determine which of rounding or rounding to select through layer-by-layer training of the neural network model instead of the rounding operation.

[0420] The layer-by-layer training part 300b-16c can optimize the parameters layer by layer in order to effectively quantize the neural network model based on the acyclic graph. In an example, the layer-by-layer training part 300b-16c can update the weight value parameter of each layer so that the quantization loss is minimized in each layer with respect to the plurality of layers included in the second neural network model. The layer-by-layer training part 300b-16c can apply the parameter value for which quantization is performed only to the current layer for which the parameter is optimized, and the remaining layers directly retain the original parameter value for which quantization is not performed, and in this way, perform an operation on the neural network model. The quantization function of the weight value parameter of an example of the present disclosure is shown in mathematical expression 7.

[0421]

Mathematical Expression 7

[0422] wherein, is an original weight value. is a weight value after dequantization. is a scale (or scale vector). is an element-wise parameter of training, has the same shape as . For example, when is an M*N matrix, is also an M*N matrix. and represent the minimum and maximum values in the performance after quantization. is a Rectified Sigmoid having a value between 0 and 1. The initial value of . For example, the Rectified Sigmoid function may be defined as . However, the disclosure is not limited to the example, and various functions outputting a value between 0 and 1 can be applied.

[0423] Unlike the mathematical expression 3 which is a weight value parameter quantization function including a rounding operation, according to the mathematical expression 7, the learning parameter reflecting the minimum quantization loss for each layer output value can be substituted instead of the rounding operation, thereby selectively applying the rounding operation.

[0424] The layer-by-layer training part 300b-16c can train the learning parameter included in the quantization function so that the difference (loss) between the output value (e.g., FP32 matmul output) of the neural network model which is not quantized layer by layer and the output value (e.g., INT8 matmul output) of the neural network model which is quantized is minimized. The rounding loss can be introduced so that is close to 0 or 1 while training . The loss function of one example of the disclosure is shown in mathematical expression 8.

[0425]

Mathematical Expression 8

[0426] wherein, is an input of each layer. This is the activation function. This refers to a normalization function (e.g., the Frobenius norm). Besides the Frobenius norm, various other normalization functions can be applied. This represents the output values ​​of each layer of the neural network model, including floating-point parameters, before quantization. Figure 12a In, corresponding to y fp . This represents the output values ​​of each layer in a neural network model that includes integer parameters. Figure 12a In, corresponding to y int According to mathematical formula 8, of The loss is calculated using the mean squared error (MSE) between the result values ​​of the quantized layer and the original result values ​​of the unquantized layer.

[0427] For use in induction A function that converges to either 0 or 1. For example, Can be defined as a of . It converges to a real value of 0 or 1, but is not an integer; this is called floating-point form. In one example, the learned parameters... After completing the learning, you can The value is rounded to either 0 or 1 to be converted to an actual integer value. In this case, rounding loss may occur. As a normalization function to minimize this loss, a normalization function can be introduced. .but, Not limited to the examples described, various methods can be applied to make The included function is a regularization function that minimizes rounding loss. In Equation 8, variables that are repeated in Equation 7 are found in Equation 7.

[0428] The layer-by-layer training unit 300b-16c can train learning parameters. When performing learning, set a learning rate. To minimize rounding loss, for example, when the batch size is 1, the learning rate can be set to 0.001.

[0429] The layer-by-layer training unit 300b-16c can optimize the weight value parameter through layer-by-layer training of the neural network model using mathematical formulas 7 and 8. The layer-by-layer training unit 300b-16c can prevent a decrease in inference accuracy due to quantization of the neural network model by minimizing a loss of output values of each layer, rather than minimizing a loss of the weight values themselves before and after quantization.

[0430] As still another example, the second conversion unit 300b-15, when simulating quantization of the second neural network model, the mathematical formula 3 for quantizing the weight value parameter includes an operation of dividing the weight value weight fp by a scale s w . The weight value parameter can have a form including a plurality of elements. For example, the weight value can be an M*N matrix. In an example of the disclosure, the scale value for performing quantization can be calculated as shown in mathematical formula 1. At this time, the scale value is calculated for one parameter, and one scale value is uniformly applied to all elements included in the parameter. The layer-by-layer training unit 300b-16c of an example of the disclosure can optimize the scale value element by element for each element of the parameter. The layer-by-layer training unit 300b-16c can determine the scale value for each element of the weight value parameter for each layer of the neural network model, so as to minimize the quantization loss of the result value (output) predicted by the neural network model after quantization.

[0431] The layer-by-layer training unit 300b-16c can train the elements included in the weight value parameter with respect to the scale s w for quantizing the weight value parameter, thereby updating the weight value parameter as shown in mathematical formula 9.

[0432] [Mathematical Formula 9]

[0433] wherein, is the weight value (original weight). is the weight value after dequantization (DeQuantize weight). is the scale (or scale vector). is the initial scale value. For example, may be a value calculated according to mathematical formula 1. is a learning parameter having the same form as the weight value parameter. By one scale value can be expressed as the value of each element of the weight value parameter. The update makes the quantization loss minimum for each channel's learning parameter. For the Fully-connected layer, the scale value is updated according to the first equation. For the Convolutional layer, the scale value is updated according to the second equation. For the Convolutional layer, the learning parameter .

[0434] For the Fully-connected layer, when , it can be defined as , .

[0435] For the Convolutional layer, when , it can be defined as , , .

[0436] The layer-by-layer training part 300b-16c can train the scale value included in the quantization function, which is updated as shown in Mathematical Formula 9, so that the difference (loss) between the output value (e.g., FP32 matmul output) of the neural network model, for which quantization is not performed layer by layer, and the output value (e.g., INT8 matmul output) of the neural network model, for which quantization is performed, is minimized. At this time, the loss function can be defined as shown in Mathematical Formula 10 below.

[0437] [Equation 10]

[0438] wherein, is a weight value (original weight). is a weight value after quantization and dequantization (DeQuantize weight). is an input (input) of each layer. is a normalization function (e.g., Frobenius norm). Various normalization functions can be applied in addition to the Frobenius norm. represents an output value of each layer of the neural network model including a parameter in a floating point form before quantization is performed, and corresponds to y Figure 12a in Mathematical Formula 10. fp . represents an output value of each layer of the neural network model including a parameter in an integer form, and corresponds to y Figure 12a in Mathematical Formula 10. intThe layer-by-layer training section 300b-16c can update the scale value as shown in Mathematical Formula 9 so as to minimize the loss function of Mathematical Formula 10 layer by layer according to the layers of the neural network model.

[0439] The layer-by-layer training section 300b-16c can update the learning parameters The learning rate is set when training is performed so as to minimize the quantization loss. For example, when the batch size is 1, the learning rate can be set to 0.00003.

[0440] The quantization-aware self-distillation retraining (QASD) operation of the retraining section 300b-16e of one example of the disclosure will be described in detail below.

[0441] Figure 12b A conceptual diagram for describing a quantization-aware self-distillation method of one example of the disclosure.

[0442] The quantization-aware self-distillation section (QASD) 300b-16e can perform self-distillation in the same neural network model with reference to the output value of the neural network model including the parameters of the floating-point form in which quantization is not performed, the self-distillation calculating the loss of the neural network model in which quantization is performed. In other words, the QASD 300b-16e of one example of the disclosure can perform quantization-aware retraining by applying self-distillation with reference to the output value of the model before quantization for the same neural network model, the self-distillation being retraining performed on the model in which quantization is performed.

[0443] The optimization section 300b-16 of one example of the disclosure can improve the inference accuracy of the neural network model by an optimization process for the parameters such as outlier mitigation or parameter refinement using the outlier mitigation section 300b-16a and the parameter refinement section 300b-16b after the training of the neural network model is completed. In addition, the optimization section 300b-16 can perform retraining to optimize the parameters to reduce the quantization error of the neural network model quantized by the QAT 300b-16d.

[0444] Re-training of a neural network model means re-training the model using additional data to the initially trained model. With an initially trained model, new data can be used to adjust or update the original model. This can be used to reflect new knowledge, mainly in the case of added or changed data. Re-training generally refers to re-training the trained weight values using a data set. In connection with re-training, data augmentation can be used to improve the performance of the model in deep neural network models. Data augmentation refers to a process of deforming or augmenting the original training data to generate new training data. Data augmentation can be applied to various types of data such as images, text, audio, etc. For example, when it is for image data, rotation, shift, size adjustment, flip, brightness adjustment, etc. transformations can be applied to generate a new image set. When it is for text data, a new data set can be created by changing the summary to different words or reconstructing the sentence. When data augmentation is applied to a deep neural network model, the original data used to train the model can be collected first. These data are related to what the model actually tries to predict. Various methods can be applied to the original data to augment the data. The augmented data can be added to the original training data set to re-train the model. In this way, the model can learn various data patterns more effectively, improving the generalization performance. The performance of the re-trained model can be evaluated using a validation data set. In this process, the accuracy or other performance indicators (e.g., running time) of the model can be evaluated to confirm how the augmented data improves the model performance. By repeating this process, additional data augmentation can be performed as needed and the model performance can be continuously improved. Data augmentation can be usefully used even in the case of insufficient data or difficult annotation.

[0445] In a deep learning model, quantization aware training is different from retraining of a general neural network model, and is performed while considering quantization. When a neural network model is quantized, weight values and activation functions in a floating point form are expressed in an integer form with a number of bits and lightened, and thus inference accuracy of the neural network model can be affected. Accordingly, to reduce quantization error, the QAT 300b-16dsk of an example of the disclosure can consider a difference between a correct answer (label value) of retraining data and a result value (output value) predicted by a quantized neural network model as a loss, and update a parameter value so that the loss is minimized. When the data augmentation method used in retraining of the neural network model described above is directly applied to quantization aware retraining, an overfitting problem that is too optimized for retraining data can occur. Specifically, data augmentation is a method of improving generalization performance to avoid overfitting, and quantization aware retraining also improves generalization performance, and thus when data augmentation and quantization aware retraining are simultaneously performed, overgeneralization can occur, and thus performance of the neural network model can be degraded. To solve this problem, the optimization part 300b-16 of an example of the disclosure can prevent overgeneralization by using self-distillation when performing quantization aware retraining.

[0446] In an example, a quantization aware self-distillation part (QASD) 300b-16e can perform self-distillation using an output value of a neural network model including a parameter in a floating point form on which quantization is not performed as a reference, and calculate a loss of a neural network model on which quantization is performed. In other words, the QASD 300b-16e of an example of the disclosure can perform quantization aware retraining by applying self-distillation to a model on which quantization is performed, using an output value of a model before quantization as a reference, with respect to the same neural network model.

[0447] Referring to Figure 12b , the compiler 300b-10 of an example of the disclosure can generate a simulated quantization model including a parameter in an integer form having a predetermined target number of bits from a neural network model (pre-trained model) including a parameter in a floating point form. For convenience of description, the neural network model (pre-trained model) including a parameter in a floating point form will be referred to as a P neural network model, and the simulated quantization model will be referred to as a Q neural network model.

[0448] The QASD 300b-16e can retrain the P neural network model with the retraining data. The QASD 300b-16e can calculate a first loss (loss1) as a difference between an output value (FP32_output) of the P neural network model and a label of the retraining data. The QASD 300b-16e can optimize parameters of the P neural network model during the retraining to minimize the first loss.

[0449] In an example, the QASD 300b-16e can generate the Q neural network model with the mathematical formulas 1 to 3 for the P neural network model. The P neural network model and the Q neural network model are essentially the same neural network model with different parameter representations. The P neural network model and the Q neural network model have the same neural network structure, and each layer has substantially the same parameters (e.g., weight value parameters and bias parameters). At this time, the parameters of the P neural network model can be in a floating point form, and the parameters of the Q neural network model can be in an integer form.

[0450] The QASD 300b-16e can retrain the Q neural network model with the retraining data. The QASD 300b-16e can calculate a second loss as a difference between an output value (sq_output) of the Q neural network model and the output value (FP32_output) of the P neural network model. The QASD 300b-16e can optimize parameters of the Q neural network model during the retraining to minimize the second loss.

[0451] For example, the k-th training data is described, when the actual result value of the k-th training data is called , the output value of the P neural network model for the k-th training data is , and the output value of the Q neural network model for the k-th training data is . The QASD 300b-16e can calculate the first loss as shown in , and calculate the second loss as shown in . The QASD 300b-16e can update the weight value parameters of the P neural network model during the retraining of the P neural network model to minimize the first loss. The QASD 300b-16e can update the weight value parameters of the Q neural network model during the retraining of the Q neural network model to minimize the second loss.

[0452] In an example of the disclosure, the retraining data can be the same as the initial training data. Alternatively, in another example of the disclosure, the retraining data can be different from the initial training data. However, the retraining data can be determined in association with the initial training data. For example, when the initial training data is the image image, the retraining data can also be the image image. The retraining data can be generated by expanding the initial training data with data augmentation.

[0453] In an example of the disclosure, the QASD 300b-16e can find the optimal value of the weight value parameter for the neural network model including the quantized parameters using the gradient descent method. The gradient descent method is a method of repeatedly performing the following process gradually, i.e., in the correlation between the weight value and the cost, in order to reach the point of minimum cost from the initial weight value, the graph gradient between the weight value and the cost, i.e., the degree of loss due to the change in the weight value, is subtracted. At this time, the QASD 300b-16e can calculate the cost with the predicted value (i.e., the output value) of the neural network model having the same floating point form of parameters as the neural network model on which the quantization is performed, not the actual result value (i.e., the label value) of the retraining data. Specifically, the cost (i.e., the loss due to quantization) can be calculated by subtracting the output value of the neural network model on which the quantization is performed from the output value of the neural network model having the floating point form of parameters

[0454] The QASD 300b-16e can update the weight value parameter value using Mathematical Formula 4 while performing the retraining on the Q neural network model, which is obtained by quantizing the P neural network model according to Mathematical Formulas 1 to 3.

[0455] In an example, the QASD 300b-16e can perform the retraining on the P neural network model first. The P neural network model has not been quantized, and thus can update the parameters based on the label value of the retraining data according to the general retraining method. The QASD 300b-16e can store the output value of the P neural network model for the retraining data. In another example, when the retraining data is the same as the initial training data, the retraining can not be performed on the P neural network model. The output value of the P neural network model for the initial training data can be directly used.

[0456] ​​​QASD 300b-16e can utilize Formula 5 to generate prediction values ​​while retrained data is propagated forward through the Q-neural network model. QASD 300b-16e can calculate the loss function. The loss function This represents the prediction result value of the P neural network model. The predicted values ​​of the Q-neural network model The difference between them. At this point, the loss function reflects the quantization-related loss. QASD 300b-16e can use Equation 6 to calculate the gradient of the loss function during back propagation. The parameters can be updated based on the gradient value. For example, the weight parameters can be updated based on the change in the loss function caused by the difference in weight values. If this process is repeated, the weight parameters of a particular layer will converge to their optimal values.

[0457] The compiler 300b-10 of this disclosure, as an example, can add multiple tags to multiple graph modules included in a first neural network model of a directed acyclic graph (DAG) shape using a tag addition unit 300b-13. The compiler 300b-10 uses a calibration unit 300b-14 to collect the input and output values ​​of each of the multiple graph modules using the multiple tags, thereby generating calibration data. The second conversion unit 300b-15 of the compiler 300b-10 can determine, based on the calibration data, a scale value and an offset value applicable to the first neural network model according to mathematical formula 1. The second conversion unit 300b-15 can perform quantization on the first neural network model with floating-point morphological parameters based on the scale value and the offset value, generating a second neural network model with quantized parameters in integer format. QASD 300b-16e can obtain the output value of the first neural network model for retraining data. QASD 300b-16e can perform quantization-aware retraining on the second neural network model based on the output value of the first neural network model, updating at least one weight value parameter included in the second neural network model.

[0458] QASD 300b-16e allows updating the parameters of each of the multiple graph modules included in a second neural network model using gradient descent to minimize the loss caused by parameter changes. In this case, the loss represents the difference between the output value of the graph module of the first neural network model corresponding to the graph module of the second neural network model and the output value of the graph module of the second neural network model. In other words, during quantization-aware retraining, the loss is the predicted value of the P neural network model. The predicted values ​​of the Q-neural network model The difference between them. QASD 300b-16e can use Equation 4 to subtract the loss difference caused by the change in the current weight parameters and update the current weight parameters. Equation 4 includes the learning rate, which represents the degree of parameter change. The learning rate indicates the finesse of retraining; a smaller learning rate allows for more precise application of changes in weight parameters during retraining. QASD 300b-16e determines the learning rate based on user options or the retraining execution time, thus allowing selection of the current parameter variation. QASD 300b-16e allows termination when the loss reaches a predetermined threshold or exceeds a predetermined runtime. The runtime can also be set in epochs. The defined runtime can be predetermined by user options.

[0459] QASD 300b-16e can correspond to the addition of a quantization module to each of the multiple graph modules included in the second neural network model, as shown in Equation 5. A loss change calculation function is added to the forward calculation of each of the multiple graph modules, and the output value of each graph module is confirmed in the backward calculation of each of the multiple graph modules, as shown in Equation 6, corresponding to the change of each parameter.

[0460] In one example, the graph module may be a Gemm function (General Matrix Multiply function), a matrix multiplication function, or a convolution function, but this disclosure is not limited to such functions.

[0461] The loss calculation function does not affect the results of the forward calculation, allowing the original formula stored in the reverse calculation to be removed by the rounding and pruning operations included in the quantization module. For example, for the forward calculation, QASD 300b-16e can be used in the graph module's calculation as shown in Equation 5. It includes a detach function as the function for calculating the loss change. The loss can be applied to the output value of the first neural network model. The output value of the second neural network model Defined as loss function At this point, QASD 300b-16e can convert equation 6 (1) Change to The formula for differentiation Only the variables change; the mathematical formula remains the same. With the change in 1), the variables in 2) to 7) of mathematical formula 6 also change. QASD 300b-16e can use the formula obtained after changing the differential formula of the loss function in mathematical formula 6 to confirm the scale values ​​of the input feature map parameters during backpropagation. Offset values ​​of input feature map parameters Scale value of the weight parameter Changes in various losses caused by changes .

[0462] QASD 300b-16e can again refer to mathematical expression 4, subtract a loss change caused by a change in a scale value of a current weight value parameter from a current weight value parameter value , and update a next weight value parameter value. At this time, when the degree of loss change reaches a predetermined target loss, QASD 300b-16e can end retraining.

[0463] Figure 12c is an example of the operation of the pruning section 300b-16f shown in FIG. 17. Figure 11 The pruning section 300b-16d of an example of the disclosure can provide a technology capable of reducing the amount of computation of a neural network model. The pruning algorithm can be configured to replace smaller values close to 0 with 0 in weight values of all layers of the neural network model. The plurality of neural processors 2200a can skip a multiplication operation related to a 0 weight value to improve the operation speed of convolution and reduce power consumption, and can reduce the parameter size of the machine code of the neural network model to which the pruning option is applied. Further description is made that, when a specific weight value parameter is replaced with 0 according to the pruning option, the pruning algorithm can provide substantially the same effect as cutting off the connection net of the neural network model having the corresponding weight value data. For example, the pruning option can provide a first pruning option based on size that removes a weight value having a small size and a second pruning option based on percentage that removes a specific proportion of the smallest weight value.

[0464] The pruning section 300b-16f performs pruning with respect to weight value parameters included in each layer of the second neural network model. The pruning section 300b-16f can remove values that do not affect the result value of the neural network model from the values of the weight value parameters to reduce the amount of computation and improve the overall operation speed. The pruning section 300b-16f can perform pruning on the second neural network model based on a unidirectional graph, and perform pruning on the second neural network model in which at least a part of the parameters are optimized.

[0465] In an example, the pruning section 300b-16f can perform pruning on the weight value parameters layer by layer for each layer of the second neural network model. The pruning section 300b-16f can perform pruning on the weight value parameters with respect to a scale value of each layer of the second neural network model.

[0466] In an example, the pruning section 300b-16f can perform pruning on the weight value parameters layer by layer for each layer of the second neural network model. The pruning section 300b-16f can perform pruning on the weight value parameters with respect to a scale value of each layer of the second neural network model. ​The pruning methods include 1) a method of pruning components corresponding to a preset pattern, 2) a method of pruning components having a weight value close to 0 based on a threshold value a, 3) a method of pruning components corresponding to a predetermined channel, or 4) a method of pruning components corresponding to a predetermined row. Components pruned according to each method can be stored as a mask. The mask has a value of 0 or 1 and can have the same form as the weight value parameter. When the mask value is 1, the value of the weight value parameter is maintained, and when the mask value is 0, the value of the weight value parameter can be pruned and skipped at the time of calculation. Referring to Figure 12c , the diagonal portion of the masks 1 to 4 is pruned as a pruning target and has a value of 0, and the remaining portion has a value of 1 so as to maintain the component value.

[0467] Referring to Figure 12c Mask 1 of FIG. 1 is an example of 1) a method of pruning components corresponding to a preset pattern. The pruning part 300b-16f can reflect characteristics of input data, structural characteristics of a neural network model, etc., and various patterns can be defined in advance. The pruning part 300b-16f can select one of the preset patterns, perform pruning on the neural network model, and the preset pattern can add a direction determination step to the pruning region and gradually increase the pruning ratio.

[0468] Referring to Figure 12c Mask 2 of FIG. 2 is an example of 2) a method of pruning components having a weight value close to 0 based on a threshold value a. The threshold value a can be a ratio or an absolute value. For example, when the threshold value a is a ratio, a% of the weight value close to 0 can be pruned. Alternatively, when the threshold value a is an absolute value, the weight value less than the threshold value a can be pruned. The threshold value a can be adaptively determined by reflecting characteristics of input data, structural characteristics of a neural network model, etc.

[0469] Referring to Figure 12c Mask 3 of FIG. 3 is an example of 3) a method of pruning components corresponding to a predetermined channel. The pruning part 300b-16f can perform pruning in a channel unit. When pruning in a channel unit, the effect of pruning can be improved in hardware operation. The pruning part 300b-16f can perform mask processing as shown in Mask 3 and reduce the form of the weight value parameter. For example, when channels 1 and 4 among channels 1, 2, 3, and 4 are subjected to mask processing, the size of the weight value parameter matrix can be pruned from 4x4 to 2x4 after copying channels 2 and 3 to channel 1 and channel 2.

[0470] Referring to Figure 12bMask 4, is an example of pruning the components corresponding to the predetermined row. The pruning part 300b-16f can perform pruning in a row unit. Like pruning in a channel unit, when performing mask processing in a row unit, the shape of the weight value parameter itself can be reduced. For example, in row 1, row 2, row 3, and row 4, after performing mask processing on row 2 and row 3, copying row 4 to channel 2, the size of the weight value parameter matrix can be pruned from 4x4 to 4x2.

[0471] The pruning part 300b-16f can gradually update the mask using each pruning method. After applying the initial mask value to the weight value parameter, the loss degree of the neural network model can be confirmed to determine whether to perform mask processing. When the loss degree is greater than a predetermined threshold value, the corresponding mask processing can not be used. The pruning part 300b-16f can gradually increase the pruning degree and repeatedly implement.

[0472] In various embodiments, the pruning part 300b-16f can set a separate pruning flag for each weight value parameter of each layer, and when the corresponding components of the weight value parameter are mask-processed, the flag can be checked.

[0473] Figure 12d and Figure 12e is an example for describing each step of the operation of the outlier mitigation part 300b-16a according to an embodiment of the disclosure.

[0474] The outlier mitigation part 300b-16a can mitigate the outliers of each operation object by partially transferring the outliers included in the operation objects of the MAC operation between the operation objects, while the MAC operation result remains unchanged. In an example, this is the same as converting the operation to . At this time, is referred to as an adjustment value for adjusting the outliers.

[0475] The outlier mitigation part 300b-16a can calculate the adjustment value based on the first correction data collected from the input parameters and the weight value parameters using the flag added in each graph module. In an example, the outlier mitigation part 300b-16a can perform 50 times of calibration to collect the first correction data using the flag added in each graph module. In an example, the outlier mitigation part 300b-16a can calculate the adjustment value using the maximum value of the input parameters and the maximum value of the weight value parameters. The adjustment value is used to adjust the data range, and the outlier mitigation part 300b-16a can find the maximum value of the absolute value of the input parameters, find the maximum value of the absolute value of the weight value parameters, and find the maximum positive number.

[0476] The form of the adjustment value can be determined according to the form of the operation object. For example, when the operation object is a matrix, the adjustment value can also be a matrix. When the first operation object is an M*I matrix and the second operation object is an I*N matrix, an adjustment value matrix 1*I can be generated for the channel I. For example, Activation A is a 2*4 matrix, and Weight W is a 4*3 matrix, and the operation objects correspond to a convolution operation.

[0477] The outlier mitigation unit 300b-16a can show the maximum value among the absolute values of each channel for the first operation object and the second operation object of the MAC operation, respectively. For example, the maximum value set of each channel of the A matrix can be {Amax1, Amax2, Amax3, Amax4}. For example, the maximum value set of each channel of the W matrix can be {Wmax1, Wmax2, Wmax3, Wmax4}.

[0478] In an example of the disclosure, the adjustment value can be obtained using the following mathematical formula. However, examples of the disclosure are not limited to the mathematical formula, and the adjustment value can be determined using various formulas.

[0479] adPi means the value of the adjustment value of channel i, Amaxi means the maximum value among the absolute values of all elements of channel i of the input parameter, and Wmaxi means the maximum value among the absolute values of all elements of channel i of the weight value parameter.

[0480] The outlier mitigation unit 300b-16a can multiply the input parameter of the first graph module including the MAC operation by the reciprocal of the adjustment value (for example, the first adjustment value), and multiply the weight value parameter of the first graph module by the adjustment value (for example, the second adjustment value), in order to optimize the input parameter and the weight value parameter from the perspective of reducing quantization error based on the adjustment value for adjusting the outlier of each graph module of the second neural network model.

[0481] In an example, the outlier mitigation unit 300b-16a can perform optimization of the input parameter and the weight value parameter of the first graph module before executing the first graph module. The outlier mitigation unit 300b-16a can not add additional operations, and reflect the adjustment value by multiplication operations performed before the first graph module, so that the parameter optimization operation is executed together with the original operation.

[0482] In an example, the layer normalization graph module can be further included in the previous step of the first graph module. The layer normalization step can include a multiplication operation, and the multiplication operation included in the layer normalization can be utilized without adding an additional multiplication operation to reflect the adjustment value. Accordingly, the layer normalization graph module can perform an operation of multiplying the input parameter by the first adjustment value. Then, in the first graph module, an operation of multiplying the weight value parameter reflecting the second adjustment value can be performed. For example, when the graph included in the normalization layer configured before the MAC operation includes a function of A * B and the variable is multiplied by the first adjustment value , the function is modified using and . Here, the variables and are constants, and thus can be calculated in the optimization unit 300b-16 and stored as constant parameters. At this time, resource waste that occurs when the multiplication operation for optimizing the parameter (e.g., multiplication of the input parameter by the first adjustment value) is independently performed can be reduced. In addition, the multiplication operation of the second adjustment value and the weight value parameter can be performed and stored as a constant parameter. In this case, resource waste that occurs when the multiplication operation for optimizing the parameter is independently performed can be reduced.

[0483] In various examples, the outlier mitigation unit 300b-16a can reflect the parameter optimization operation before the operation of the graph module in a predetermined multiplication operation. In yet another example, when the MAC operation (e.g., matmul operation) is not included in the graph module or the multiplication operation is not included in the previous step of the graph module, the parameter optimization operation can not be performed in consideration of the cost that occurs when the multiplication operation for optimizing the parameter is independently performed.

[0484] The input parameter and the weight value parameter can be optimized by applying the adjustment value to reduce the quantization error of the outlier. Each adjustment value (e.g., the first adjustment value and the second adjustment value) can be calculated and stored as a constant parameter in the compilation step of the neural network model. In particular, it is preferable to calculate and store as a constant parameter in advance in order to reduce the power consumption of the inference operation of the neural processing unit and to improve the inference speed.

[0485] Referring to Figure 12d , the outlier mitigation unit 300b-16a can multiply each element of the input parameter by the reciprocal of the adjustment value to optimize the input parameter value. In the foregoing example, A 11 *(adP1) -1 =A' 11 , A 21 *(adP1) -1 =A' 21 , A 12 *(adP2)-1 = A' = W 12 The calculation can be performed in a layer normalization step performed before the graph module. The outlier mitigation unit 300b-16a can reflect the parameter optimization operation in a multiplication operation included in the layer normalization step located before each graph module. The parameter optimization multiplication operation is included in the original multiplication operation, and thus does not incur additional operation costs. That is, the outlier adjustment of the input parameter has the advantage that, when the layer normalization is located before the MAC operation, the variable , can be provided to the third neural network model without increasing additional inference resources. Therefore, the third neural network model to which the outlier mitigation value generated in the third conversion unit 300b-17 is applied basically does not need to increase resources required to mitigate outliers.

[0486] Referring to Figure 12e , the outlier mitigation unit 300b-16a can multiply each element of the weight value parameter by an adjustment value to optimize the weight value parameter value. In the foregoing example, the calculation can be performed as shown in W 11 * adP1 = W 11 , W 12 * adP1 = W 12 , W 21 * adP2 = W 21 .

[0487] In an example of the disclosure, the input parameter and the weight value parameter to which the outlier mitigation is applied can be applied in a quantization step and steps thereafter. For example, when the outlier mitigation unit 300b-16a performs outlier mitigation with respect to the second neural network model by means of the optimization unit 300b-16, the input value feature_in int of the third neural network model can mean that the outlier mitigation is applied.

[0488] In one example of the present disclosure, the outlier mitigation unit 300b-16a obtains correction data for mitigating outliers, and at this time, a component independent of the calibration unit 300b-14 can be further included. The input values and weight values collected in the marks included in each graph module can be obtained as correction data using any correction data in the correction data set. The correction data generated by the calibration unit 300b-14 can be used to calculate the scale values and offset values of each parameter in the second conversion unit 300b-15. The outlier mitigation unit 300b-16a can mitigate outliers of the input parameter and the weight value parameter independently of the operation of the second conversion unit 300b-15. The optimization unit 300b-16 can perform parameter refinement after performing outlier mitigation, and both the outlier mitigation and the parameter refinement can be reflected in the quantization simulation of the second neural network model. When the outlier mitigation is performed, the quantization simulation process of the second neural network model and the input parameters in which outliers are mitigated in the process can be reflected. That is, the third conversion unit can generate a third neural network model based on the quantization simulation of the second neural network model using the input parameters and the weight value parameters in which the adjustment values of the mitigated outliers are reflected. After the adjustment values in which the outliers are mitigated are determined, the third conversion unit can reflect each adjustment value to the input parameter and the weight value parameter of the corresponding neural network model. Figure 13a and Mathematical Formula 11 is a convolution example for describing the first neural network model of one example of the present disclosure.

[0489] The convolution of the first neural network model can be expressed by Figure 13a and Mathematical Formula 11. Figure 13a A graph module (Conv) corresponding to the convolution is shown. Each graph module is provided with an input parameter. The input / output parameters of the graph module can be referred to Mathematical Formula 11. Figure 13a The graph module shown can constitute a directed acyclic graph (DAG). The first neural network model is an example of the most basic neural network model, which is a neural network model in which all operations are processed using floating point parameters. The first neural network model can be a model that can only be run in the GPU 100b of the neural network model optimization device 1500, and can be a model including a function call instruction.

[0490]

Mathematical Formula 11

[0491] wherein feature_out fp represents a floating point output feature map, feature_in fp represents a floating point input feature map, and weight fp represents a floating point weight value. Herein, means convolution. Herein, Mathematical Formula 11 represents substantially the same operation as Figure 13a

[0492] ​Figure 13b and mathematical formula 12 is a convolution example for describing a second neural network model of an example of the present disclosure.

[0493] The convolution of the second neural network model can be expressed as Figure 13b and mathematical formula 12. Figure 13b The graph modules corresponding to the convolution (Conv), the graph module corresponding to the subtraction (Sub), the graph module corresponding to the division (Div), the graph module corresponding to the Round (Round), the graph module corresponding to the Clip (Clip), and the graph module corresponding to the addition (Add) are shown in the mathematical formula 12. Each graph module is provided with an input parameter. The parameters of each graph module can refer to mathematical formula 12. Figure 13b Some of the graph modules of the mathematical formula 12 can be function call instructions converted in the graph generation unit 300b-12. Figure 13b Each graph module shown in the mathematical formula 12 can be connected to each other to form a directed acyclic graph (DAG). The second neural network model is an example of a neural network model that can simulate quantization of the first neural network model, is a neural network model in which all operations are processed with floating-point parameters, and can calculate inference accuracy degradation, quantization error, and the like caused by quantization.

[0494]

Mathematical formula 12

[0495] wherein feature_out fp represents a floating-point output feature map simulated for quantization, feature_in fp represents a floating-point input feature map, o f represents an offset value of the mathematical formula 1 for the input feature map to be quantized, s f represents a scale value of the mathematical formula 1 for the input feature map to be quantized, weight fp represents a weight value of the floating-point to be quantized, s w represents a scale value of the mathematical formula 1 for the weight value to be quantized, represents a rounding (round) and clipping (Clip) operation, means convolution. The mathematical formula 12 represents substantially the same operation as Figure 13b

[0496] Therefore, the compiler 300b-10 can simulate quantization of the first neural network model using the second neural network model. The compiler 300b-10 can perform an inference accuracy degradation degree evaluation by using quantization simulation of the second neural network model. The inference accuracy degradation degree can be different according to the level of the target quantization (for example, 16-bit, 8-bit, 4-bit, 2-bit quantization level) and the clipping degree, and various bit-width quantizations can be simulated according to the settings of the compiler 300b-10. ​

[0497] To further describe, compiler 300b-10 can set the quantization level of each graph module uniformly. Compiler 300b-10 can also set the quantization level differently for each graph module. Compiler 300b-10 can set the quantization levels of the input and output parameters of graph modules differently for each other. Compiler 300b-10 can also set the quantization levels of the input and output parameters of graph modules uniformly.

[0498] Then, the third conversion unit 300b-17 can convert the second neural network model into a third neural network model that can run in the neural processor 100a of the edge device 1000. That is, the third conversion unit 300b-17 can perform the operation of generating the third neural network model based on the quantization simulation of the second neural network model.

[0499] The first neural network model and the second neural network model can be models that can run on the GPU 100b, which is capable of reasoning and learning, and the third neural network model can be a model that can run on the neural processor 100a of the edge device 1000, which is capable of reasoning only.

[0500] That is, the third neural network model can refer to a neural network model optimized for inference. Therefore, the edge device 1000 can receive the third neural network model from the neural network model optimization device 1500. The third neural network model can be a compiled neural network model, which may be referred to as binary code or machine code, etc. The third neural network model can be stored in the memory 200a of the edge device 1000. The third neural network model is configured to run in the neural processor 100a of the edge device 1000.

[0501] Figure 13c Formula 13 is a convolution example used to describe a third neural network model of an example of this disclosure.

[0502] The convolution of the third neural network model can be used Figure 13c And mathematical expression 13. In Figure 13c The diagram shows the graph module (Conv) corresponding to convolution. Each graph module has input parameters set. Figure 13c The input / output parameters of the graph module can be found in mathematical formula 13. Figure 13c The graph modules shown can form a directed acyclic graph (DAG).

[0503] exist Figure 13cAn example of a quantized convolution of a third neural network model is described. The processing unit (not shown) of the neural processor 100a of the edge device 1000 may be a circuit configured to process the convolution of the third neural network model. The processing unit may be a circuit configured to receive and output integer parameters. The processing unit may be an arithmetic unit configured to process multiply and accumulate (MAC) operations. For example, multiple processing units (not shown) of the neural processor 100a may correspond to... Figure 3 , Figure 4a , Figure 5 The multiple processing units 110 shown. Figure 3 , Figure 4a , Figure 5 The neural processor 100 shown can correspond to Figure 6 The edge device 1000 includes a neural processor 100a.

[0504]

Mathematical Expression 13

[0505] Among them, feature_out int The output feature map representing integers, feature_in int Input feature map representing integers, weight int Weight values ​​representing integers, This refers to convolution. Specifically, mathematical formula 13 and... Figure 13c This indicates that the operations are essentially the same.

[0506] For example, feature_in int Can be entered in Figure 4a The first input section of the first processing unit PE1. Wherein, feature_in int This can be a parameter quantized to 8 bits. However, this disclosure is not limited to this; feature_in int The bit width can be 2 to 16 bits.

[0507] To elaborate further, the feature_in of mathematical formula 13 int This can be quantified using mathematical formula 2. Alternatively, feature_in int It can be configured to receive signals from sensors such as image sensors, microphones, radar, and LiDAR connected via interface 400a of edge device 1000. Among these, feature_in int The value can be stored in memory 200b in real time (e.g., in frame units, line buffer units, etc.) via interface 400a of edge device 1000. For example, feature_in intThe 8-bit resolution RGB image outputtable by the camera. Thus, the edge device 1000 can process the operation of the third neural network model using the quantized integer form feature map.

[0508] For example, weight int may be input to the second input unit of the first processing element PE1 of Figure 4a . Among them, weight int may be a parameter quantized in 8 bits. However, the present disclosure is not limited thereto, and weight int may be 2 to 16 bits.

[0509] Further description, weight int may be pre-calculated by mathematical formula 3. If the weight value of the second neural network model is trained, weight fp and s w become constants whose values are unchanged. Thus, the compiler 300b-10 can pre-calculate the value of weight int and store it as a constant in the memory 200b. In addition, the quantized weight value (weight int ) can be transmitted to the memory 200a of the edge device 1000. Thus, the edge device 1000 can process the operation of the third neural network model using the quantized integer form weight value.

[0510] According to an example of the present disclosure, the bit width of the input parameter (e.g., input feature map) and the output parameter (e.g., output feature map) of the convolution graph module in the graph module of the third neural network model can be different from each other.

[0511] Referring to Figure 4a , for example, the bit width (X) bit of feature_in int may be 8, and the bit width (X) bit of feature_out int may be 24. Further description, the value can be accumulated in the convolution, and when feature_out int is an 8-bit integer, overflow can occur. Thus, in order to prevent overflow, the bit width (X) bit of the output feature map can be appropriately set.

[0512] Further description, the size of the value accumulated in the accumulator 113 can have a larger bit width (e.g., (X) bit of feature_out Figure 4a ) than the bit width (e.g., (N) bit and (M) bit of feature_in Figure 4a ) of the input integer parameter according to the convolution operation amount.

[0513] For example, a bit width of an input parameter (e.g., an input feature map) of a convolution graph module in a graph module of the third neural network model can be smaller than a bit width of an output parameter (e.g., an output feature map).

[0514] For example, a bit width of an output parameter (e.g., an output feature map) of a convolution graph module in a graph module of the third neural network model can be larger than a bit width of an input parameter (e.g., an input feature map).

[0515] Figure 13d and Mathematical Formulae 14 to 16 are examples of convolution, dequantization, and quantization of the third neural network model for describing one example of the present disclosure.

[0516] Dequantization and quantization after convolution of the third neural network model can be performed by Figure 13d and Mathematical Formulae 2, 13, 14, 15, and 16. In Figure 13d A graph module (Conv) corresponding to convolution, a graph module (Mul(dequant), Add(dequant)) corresponding to dequantization, and a graph module (Sub(o f ), Div(s f ), Round, Clip) corresponding to quantization are illustrated in Figure 13d Parameters of the graph module of Figure 13d The graph module illustrated in

[0517] After convolution of the third neural network model (the convolution can refer to Mathematical Formula 13), a parameter quantized in an integer according to a graph module that can be included in the third neural network model needs to be converted to a floating point.

[0518] Therefore, examples of convolution, dequantization, and quantization of the third neural network model are described in Figure 13d

[0519] A processing unit (not illustrated) of the neural processing unit 100a of the edge device 1000 can be a circuit configured to process convolution of the third neural network model. The processing unit can be a circuit configured to receive an integer parameter and output an integer parameter. The processing unit can be an operator configured to process multiply and accumulation (MAC) operations. Figure 13d Convolution of Figure 13c Convolution of Figure 3 , Figure 4a , Figure 5 The plurality of processing units 110 illustrated in Figure 3 , Figure 4a ​, Figure 5 The neural processor 100 shown can correspond to Figure 6 the neural processor 100a included in the edge device 1000 of

[0520] The SFU (not shown) of the neural processor 100a of the edge device 1000 can be configured to include a circuit configured to process dequantization and quantization of the third neural network model. For example, the SFU (not shown) of the neural processor 100a of the edge device 1000 can correspond to Figure 3 , Figure 4b , Figure 5 the SFU 150 shown. The Figure 3 , Figure 4b , Figure 5 The neural processor 100 shown can correspond to Figure 6 the neural processor 100a included in the edge device 1000 of

[0521] Specifically, for example, the dequantization circuit of the SFU 150 can be a circuit designed to process dequantization of mathematical formulas 15 and 16, and the quantization circuit of the SFU 150 can be a circuit designed to process quantization of mathematical formula 2. That is, the dequantization circuit receives an integer parameter and converts it to a floating point parameter for output. The quantization circuit receives a floating point parameter and converts it to an integer parameter for output.

[0522] That is, Figure 13d The convolution graph module (Conv) of the third neural network model shown can be configured to be processed in the processing unit of the neural processor of an example of the present disclosure, the dequantization graph module (Mul (dequant) and Add (dequant)) of the third neural network model can be configured to be processed in the dequantization circuit of the neural processor of an example of the present disclosure, and the quantization graph module (Sub (of), Div (sf), Round, Clip) of the third neural network model can be configured to be processed in the quantization circuit of the neural processor of an example of the present disclosure.

[0523] Convolution, dequantization, and quantization are described below with reference to mathematical formulas 14 to 16.

[0524] For example, in the SFU 150 of Figure 4b , the Activation function circuit and the Batch Normalization circuit can be configured to receive a floating point parameter.

[0525] [Mathematical formula 14]

[0526]

[0527]

[0528] feature_out of mathematical expression 14 int This represents the output feature map of integer parameters after dequantization and requantization. (Feature_in from Equation 14) int The input feature map represents integer parameters, and the weights are... int This represents the weight value of the integer parameter. This refers to convolution, which is essentially the same as in equation 13. The dequant in equation 14... mul The dequant, already defined in equation 15, is in equation 14. add Defined in equation 16. Dequantization can be performed using equations 15 and 16. That is, dequantization... mul and dequant add After applying it to mathematical expression 14, Figure 13d The output of the Conv module, i.e., feature_out int Can be converted to feature_out fp s in mathematical expression 14 f and o f This can be calculated using mathematical formula 1. Then, the output of the Conv graph module, i.e., feature_out... int via dequant mul and dequant add Dequantization to feature_out fp Then, feature_out fp It can be provided to the corresponding functional unit in the SFU 150 to process the required operations. The value calculated in the SFU 150 can be quantized again. Among them, mathematical expression 14 and Figure 13d This represents essentially the same operation. Therefore, feature_out fp It can be provided to the SFU 150, providing circuit units for processing the required floating-point operations.

[0529]

Mathematical Expression 15

[0530] dequant of mathematical formula 15 mul s is a floating-point constant parameter. f and s w This is a constant floating-point parameter. Further description: s f and s wThe calculation can be performed in the second conversion unit 300b-15 of the compiler 300b-10. In addition, o f and s w are constants, so dequant mul can be calculated in advance. Therefore, dequant mul can be a constant parameter of the third neural network model calculated in advance. Thus, dequant mul may be stored in the memory 200a of the edge device 1000. The operation of mathematical formula 15 can be omitted in the neural processor 100a. Thus, the operation of the neural processor 100a processing the third neural network model can be accelerated, power consumption can be reduced, and the amount of memory 200a used for the operation of mathematical formula 15 can be reduced.

[0531] [mathematical formula 16]

[0532]

[0533] dequant add is a floating-point constant parameter, o f and s w are floating-point constant parameters. dequant add may be tensor data. Further description, o f , weight int and s w can be calculated in the second conversion unit 300b-15 of the compiler 300b-10. In addition, o f , weight int and s w are constants, so dequant add can be calculated in advance. Therefore, dequant add can be a constant parameter of the third neural network model calculated in advance. Thus, dequant add may be stored in the memory 200a of the edge device 1000. The operation of mathematical formula 16 can be omitted in the neural processor 100a. Thus, the operation of the neural processor 100a processing the third neural network model can be accelerated, power consumption can be reduced, and the amount of memory 200a used for the operation of mathematical formula 16 can be reduced.

[0534] In Figure 14 , how the integer parameters and floating-point parameters of the third neural network model that can run in the neural processor 100a run in the corresponding respective specific circuits of the neural processor 100a are described.

[0535] An example of the present disclosure described from the perspective of an integer parameter can cause an integer parameter quantized by a certain bit width to be input to a plurality of processing units of a neural processor to process a convolution or a matrix multiplication. In particular, the convolution or the matrix multiplication accounts for the largest portion of the total amount of operations of a neural network model. In addition, the convolution or the matrix multiplication is less sensitive to quantization error than other operations of the neural network model. Accordingly, a neural processor including the processing units configured to process the convolution or the matrix multiplication with the quantized integer parameter, and a neural network model compiled specifically for the neural processor to accelerate inference operations, are provided together, thereby making it possible to provide an edge device capable of achieving a speeded-up operation speed with low power consumption.

[0536] An example of the present disclosure described from the perspective of a floating-point parameter can cause a result of a convolution or a matrix multiplication of an integer parameter to be input to an SFU of a neural processor, and the integer parameter to be converted into a floating-point parameter in a corresponding circuit within the SFU to process a specific operation of a neural network model. In particular, the specific operation of the neural network model is susceptible to quantization error of a quantized integer parameter. Accordingly, for an operation susceptible to quantization error, an SFU configured to selectively convert a quantized integer parameter output by the processing unit into a floating-point parameter for processing, and a neural network model compiled specifically for the neural processor to accelerate inference operations, are provided together, thereby making it possible to provide an edge device capable of achieving a speeded-up operation speed with low power consumption and substantially suppressing deterioration of inference accuracy due to quantization error.

[0537] The extraction section 300b-18 can convert the third neural network model into a format suitable for delivery to the neural processor 100a within the edge device 1000. The format may, for example, be machine code, binary code, or a model in the form of an Open Neural Network Exchange (ONNX). However, the extraction section 300b-18 of the present disclosure is not limited to a specific format and can be configured to convert the third neural network model into all formats compatible with the neural processor that runs the third neural network model.

[0538] Figure 14 A block diagram showing the structure of a neural network model performance evaluation system according to another example of the present disclosure.

[0539] Referring to Figure 14 The neural network model performance evaluation system 10000 according to another example of the present disclosure can include a user device 1000a, a neural network model optimization apparatus 2000a, and a server 3000a.

[0540] Figure 15The neural network model performance evaluation system 10000 according to another example of the disclosure is configured as follows, i.e., a specific neural network model is processed in the neural network model optimization device 2000a, and the performance evaluation result of the neural network model optimization device 2000a is provided to the user.

[0541] That is, according to another example of the disclosure, the neural network model optimization device 2000a according to an example of the disclosure can be provided online.

[0542] The user device 1000a can be a device owned by a user who wants to obtain the performance evaluation result information of the neural network model optimization device 2000a that will process the neural network model. The user device 1000a can include a smartphone, a tablet, a PC, a notebook computer, etc., which accesses the server 3000a and can provide a user interface for confirming information related to the neural network model. Among them, the user device 1000a can be an edge device.

[0543] In addition, the user device 1000a can include a neural processor 100a and can receive the neural network model optimized by the neural network model optimization device 2000a for use in the neural processor 100a of the user.

[0544] As a method by which the user device 1000a accesses the server 3000a, there can be an access method through a web service, an access method through an FTP server, an access method through a cloud server, or an access method through an application program inside the user device 1000a. However, the method by which the user device 1000a accesses the server 3000a is not limited thereto, and various well-known communication technologies can be used.

[0545] The user can transmit information of the neural network model to the server 3000a using various communication technologies. Specifically, the user uploads at least one specific neural network model and at least one specific evaluation dataset of the neural network model to the server 3000a through the user device 1000a in order to optimize the neural processor currently owned by the user or to evaluate the performance of another neural processor that the user has a purchase intention. The above-mentioned specific evaluation dataset means a dataset input to the neural network model optimization device 2000a in order to evaluate the performance of the neural network model optimization device 2000a.

[0546] The user device 1000a can receive and output the performance evaluation result of the neural network model optimization device 2000a on the neural network model from the neural network model optimization device 2000a.

[0547] For example, the user device 1000a can be any kind of terminal capable of uploading information of a neural network model to be evaluated in the neural network model performance evaluation system 10000 to the server 3000a.

[0548] For example, the user device 1000a can be any kind of terminal capable of uploading an evaluation dataset for evaluating the neural network model in the neural network model performance evaluation system 10000.

[0549] For example, the user device 1000a can be any kind of terminal capable of uploading a training dataset for retraining the neural network model in the neural network model performance evaluation system 10000.

[0550] That is, the user device 1000a can be referred to as a performance evaluation data transmission unit of a neural network model or a performance evaluation result reception unit of a neural network model.

[0551] To this end, the user device 1000a can include a processor 1120a, a display device 1140a, a user interface 1160a, a network interface 1180a, and a memory 1200a. The display device 1140a can display an option for selecting one or more NPUs. In addition, the display device 1140a can also display an option for compiling a neural network model. The memory 1200a can store a software module that can be executed to enable the processor 1120a to access the server 3000a, and in addition, the memory 1200a can store a neural network model and an evaluation dataset for transmission to the neural network model optimization device 2000a through the server 3000a. The user interface 1160a can include a keyboard and a mouse, and provide user input related to the user's selection of one or more neural processing units for processing a neural network model, and selection of a compilation option related to neural network model compilation. The network interface 1180a is a hardware component (e.g., a network card) that enables the user device 1000a to communicate with the server 3000a through a network.

[0552] The neural network model optimization device 2000a includes a neural processing unit for processing a neural network model received from the user device 1000a through the server 3000a. The neural network model optimization device 2000a can also compile and evaluate the neural network model. The neural network model optimization device 2000a grasps the performance of the processed neural network model and reports the performance result thereof to the user device 1000a through the server 3000a.

[0553] The neural network model optimization device 2000a can include a system configured of a general-purpose computer, a notebook computer, a cloud computer, a cloud server, etc., and execute various programs for determining information of a neural processor. The neural network model optimization device 2000a can obtain at least one specific neural network model for evaluating the performance of a neural processor and at least one specific evaluation dataset input in the neural network model from the server 3000a, compile and process the neural network model, and provide a performance evaluation result after the processing.

[0554] The server 3000a is a computing device that communicates with the user device 1000a in order to manage access to the neural network model optimization device 2000a. The server 3000a can include a processor 3120a, a network interface 3160a, and a memory 3180a. The network interface 3160a enables the server 3000a to communicate with the user device 1000a and the neural network model optimization device 2000a through a network. The memory 3180a stores instructions executable by the processor 3120a to perform one or more of the following operations: (i) manage user accounts; (ii) authenticate, grant permission when a user accesses in order to evaluate one or more neural processors; (iii) receive a user's selection of a neural network model, an evaluation dataset, a neural processor to be evaluated, and a user's selection of a compilation option; (iv) encrypt and store data received from a user; (v) transmit a neural network model and a user's processor selection information through a network; (vi) deliver a performance report on a selected neural processor and a recommendation for a neural processor to the user device 1000a through a network. In addition, the server 3000a can perform various services.

[0555] Further described, a neural network model, a training dataset, an evaluation dataset, etc. developed by a user belong to the intellectual property of the user, and thus require thorough security. Hereinafter, a neural network model, a training dataset, an evaluation dataset, etc. developed by a user can be referred to as user data. Therefore, in order to secure the user data uploaded to the performance evaluation system 10000, the performance evaluation system 10000 can be configured to not only perform user account login, data encryption (Data Encryption) to protect data itself, differential privacy (Differential Privacy), and data masking processing (Data Masking), but also perform access control (Access Control) and audit logging (Audit Logging) in order to control and review model access.

[0556] Data Encryption encrypts user data to protect the confidentiality of the data. Differential Privacy uses a statistical method that makes user data not sensitive to data including personal information. Data Masking obscures a part of the user data to hide sensitive information.

[0557] In addition, Access Control can be used to restrict accounts that can access user data, Audit Logging can be used to record which account is used to access user data, and Audit Logging can be used to keep a log of system and user data access, track who accesses the model and when, and detect abnormal activities.

[0558] Further, when uploading a training dataset and / or an evaluation dataset, a separate user data protection clause signing step can also be performed. Thus, the user's neural network model, training dataset, and / or evaluation dataset can be protected.

[0559] The following will be described with reference to Figure 15 The neural network model optimization apparatus 2000a will be described.

[0560] Figure 15 A block diagram of the structure of the neural network model optimization apparatus according to another example of the present disclosure.

[0561] Referring to Figure 16 The neural network model optimization apparatus 2000a can include a central processing unit (CPU) 2140a, an NPU farm (NPU FARM) 2180a including a plurality of neural processing units (NPU) 2200a, a graphic processing unit (GPU) 2300a, and a memory 2500a, each of which can communicate with each other through one or more communication buses or signal lines. The neural network model optimization apparatus 2000a can be operated by a specific operating system (OS). For example, the OS can use Microsoft Windows, MacOS, Linux (e.g., Ubuntu, Fedora, Debian, CentOS, Arch Linux), Unix, iOS, Android, etc.

[0562] The CPU 2140a can include one or more operation processors for running instructions stored in the memory 2500a. The memory 2500a can store various software modules including, but not limited to, the compiler 2100a, the storage module 2400a, and the reporting program 2600a.

[0563] On the other hand, the memory 2500a can include a volatile or non-volatile recording medium capable of storing various data, instructions, and information.

[0564] For example, the memory 2500a can include at least one type of storage medium of a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., an SD or XD memory, etc.), a random access memory, a static random access memory, a read-only memory, an electrically erasable programmable read-only memory, a programmable read-only memory, a network storage, a cloud, a blockchain database.

[0565] The CPU 2140a or the GPU 2300a within the neural network model optimization apparatus 2000a can load and run the compiler 2100a stored in the memory 2500a. The compiler 2100a can be either a semiconductor circuit or software stored in the memory 2500b and run by the CPU 2140b.

[0566] The compiler 2100a can convert a specific neural network model into machine code executable by the plurality of neural processors 2200a. That is, the compiler 2100a can generate machine code executable in the plurality of neural processors 2200a having different characteristics, respectively. Accordingly, the compiler 2100a can generate machine code to be executed in a neural processor 2200a selected from the plurality of neural processors 2200a. The machine code can also be referred to as a binary code.

[0567] The compiler 2100a can generate machine code of the neural network model for evaluating the performance of at least one neural processor 2200a selected for performance evaluation.

[0568] The compiler 2100a can be configured to provide various compilation options. The compilation options can be provided in the form of a UI in a screen of the user device 1000a so as to select various compilation options. The compiler 2100a can set a plurality of compilation options differently according to the neural processor selected for performance evaluation, and generate machine code of the optimized neural network model.

[0569] The plurality of compilation options can differ according to the types of the plurality of neural processors 2200a, and thus even the same neural network model can be compiled into different machine codes according to the types of the plurality of neural processors 2200a. That is, each machine code can be generated according to each selected compilation option.

[0570] The storage module 2400a can store various data used in the neural network model optimization device 2000a. That is, the storage module 2400a can store a neural network model compiled in a machine code form, one or more training data sets, one or more evaluation data sets, performance evaluation results, and output data from the plurality of neural processors 2200a.

[0571] The reporting procedure 2600a can process the compiled neural network model and report the performance evaluation results.

[0572] That is, the reporting procedure 2600a first determines whether the compiled neural network model can be processed by the plurality of neural processors 2200a.

[0573] When the compiled neural network model cannot be processed by the plurality of neural processors 2200a, the reporting procedure 2600a can report a specific layer in the plurality of layers of the neural network model that cannot be processed by the plurality of neural processors 2200a or a specific operation that cannot be processed.

[0574] When the compiled neural network model can be executed by a specific neural processor in the plurality of neural processors 2200a, the reporting procedure 2600a can report processing performance of the plurality of neural processors 2200a.

[0575] An index of the processing performance can be a temperature profile of the neural processor, power consumption (Watt), Trillion Operations Per Second per Watt (TOPS / W), Frame Per Second (FPS), Inference Per Second (IPS), and accuracy.

[0576] The temperature profile means temperature change data of the neural processor measured over time when the neural processor is executed.

[0577] The power consumption means power data measured when the neural processor is executed. The power consumption differs depending on the amount of operations of the neural network model developed by the user, and the power can be accurately measured only when the neural network model of the user is provided.

[0578] Trillion Operations Per Second per Watt (TOPS / W) is an index that measures the efficiency of AI hardware, and means the number of operations that can be performed per watt per second.

[0579] TOPS / W indicates how many operations can be performed by the unit power consumed by the hardware, and is an index that means the energy efficiency of the plurality of neural processors 2200a.

[0580] Inference Per Second (IPS) is an indicator representing the number of inference operations that can be performed by the plurality of neural processors 2200a per 1 second, and means the operation processing speed of the plurality of neural processors 2200a. IPS can also be referred to as Frame Per Second (FPS).

[0581] Accuracy is an indicator representing the proportion of correctly predicted samples in the whole, and means the inference accuracy of the plurality of neural processors 2200a. Further description, the accuracy of the plurality of neural processors 2200a and the inference accuracy of the graphics processor 2300a can be different from each other. This is because the parameters of the neural network model inferred in the graphics processor 2300a can be floating point, and the parameters of the neural network model inferred in the plurality of neural processors 2200a can be integer. In addition, various optimization algorithms can be selectively applied. Therefore, the parameters of the neural network model inferred in the plurality of neural processors 2200a will have a difference in the calculation value in various operations, and thus will have a different inference accuracy from the neural network model inferred in the graphics processor 2300a. The difference in inference accuracy will differ depending on the structure and parameter size characteristics of the neural network model, and in particular, the shorter the bit width length of the quantized parameter, the greater the tendency of the inference accuracy degradation caused by over-quantization. For example, the bit width of the quantization can be 2 bits to 16 bits. In addition, there is also a tendency for the inference accuracy degradation to increase caused by over-pruning.

[0582] On the other hand, the reporting program 2600a can analyze the processing performance of the neural network model compiled according to each of the plurality of compilation options, and can propose any one of the plurality of compilation options in reverse.

[0583] In addition, the reporting program 2600a can also recommend a specific type of neural processor 2200a based on the performance parameters of different neural processors 2200a.

[0584] The plurality of neural processors 2200a can be in the form of an NPU farm 2180a consisting of various NPU product groups of different performance and price on the market by a specific company. The NPU farm 2180a can provide for online evaluation of the performance of the neural network model developed by the user. The NPU farm can be provided in the form of a cloud NPU.

[0585] The plurality of neural processors 2200a can receive an evaluation dataset and input it into the compiled neural network model to perform performance evaluation.

[0586] The plurality of neural processors 2200a can include various neural processors.

[0587] More specifically, the plurality of neural processors 2200a can be classified by operation amount as a reference.

[0588] For example, the first neural processor can be a neural processor for intelligent CCTV. The first neural processor can have ultra-low power consumption, low-level inference processing performance (e.g., processing performance of 5 TOPS), extremely small semiconductor package size, extremely low price characteristics. The first neural processor can not support a specific neural network model including a specific operation and requiring a high memory bandwidth due to performance limitations. For example, the model name of the first neural processor can be "DX-V1", and the "DX-V1" neural processor can operate neural network models such as ResNet, Mobilenet v1 / v2, SSD, YOLOv5, YOLOv7, etc.

[0589] For example, the second neural processor can be a neural processor for image recognition, object sensing, object tracking of a robot. The second neural processor can have low power consumption, medium-level inference processing performance (e.g., processing performance of 16 TOPS), a relatively small semiconductor package size, low price characteristics. The second neural processor can not support a specific neural network model requiring a high memory bandwidth. For example, the model name of the second neural processor can be "DX-V2", and the "DX-V2" neural processor can operate neural network models such as ResNet, Mobilenet v1 / v2, SSD, YOLOv5, YOLOv7, etc.

[0590] For example, the third neural processor can be a neural processor for image recognition, object sensing, object tracking, generative AI service of an autonomous vehicle. The third neural processor has low power consumption, high-level inference processing performance (e.g., processing performance of 25 TOPS), a medium semiconductor package size, medium price characteristics. For example, the model name of the third neural processor can be "DX-M1", and the "DX-M1" neural processor can operate neural network models such as ResNet, MobileNet v1 / v2 / v3, SSD, EfficientNet, EfficientDet, YOLOv5, YOLOv7, YOLOv8, DeepLabv3, PIDNet, ViT, Generative adversarial network, Stable diffusion, etc.

[0591] For example, the fourth neural processor can be a neural processor for a CCTV control room, a control center, a large language model, a generative AI service. The fourth neural processor has low power consumption, high-level inference processing performance (for example, 400 TOPS of processing performance), a large semiconductor package size, and a high price characteristic. For example, the model name of the fourth neural processor can be "DX-H1", and the "DX-H1" neural processor can operate neural network models such as ResNet, Mobilenet v1 / v2, SSD, YOLOv5, YOLOv7, YOLOv8, DeepLabv3, PIDNet, ViT, Generative adversarial network, Stable diffusion, large LLM, etc.

[0592] That is, each neural processor can have different operation processing capabilities, different semiconductor chip sizes, different power consumption characteristics, etc.

[0593] However, the types of the plurality of neural processors 2200a are not limited thereto, and can be distinguished according to various classification criteria.

[0594] The graphics processor (GPU) 2300a is hardware that performs graphics and image processing and complex operation jobs. The graphics processor 2300a is illustrated as one in the drawing, but is not limited thereto, and can be a plurality of graphics processors connected through a cloud GPU, an NVLink, an NVSwitch, etc.

[0595] The graphics processor 2300a includes a plurality of cores and can perform a plurality of jobs in parallel through the plurality of cores. Accordingly, the graphics processor 2300a can perform large-scale data processing jobs such as scientific computing, deep learning.

[0596] In particular, the graphics processor 2300a can be used to train deep learning and machine learning models in large-scale data sets. Deep learning models have a large number of parameters, and training takes a long time, and the graphics processor 2300a can improve the training speed by processing such operations in parallel. When a user selects a specific neural processor among the plurality of neural processors 2200a and performs retraining of a neural network model through various compilation options, a graphics processor that can be used can be selected in the graphics processor 2300a, and the selected graphics processor can perform retraining of the neural network model according to each compilation option.

[0597] The plurality of neural processors 2200a and the graphics processor 2300a can be implemented in the form of an integrated chip (IC) such as an SoC (System on Chip) in which various operation devices are integrated, or a printed circuit board on which the integrated chip is mounted.

[0598] Figure 16 A block diagram illustrating a structure of a compiler of a neural network model optimization apparatus according to another example of the disclosure.

[0599] Referring to Figure 17 The compiler 2100a of the neural network model optimization apparatus can compile the neural network model into machine code based on a plurality of compilation options.

[0600] The compiler 2100a of the neural network model optimization apparatus can include an optimizer 2110a, a verifier 2120a, and a code generator 2130a.

[0601] The compiler 2100a can receive structure data of a neural processor selected from among a plurality of neural processors 2200a. The structure data of the neural processor can include a memory size of an NPU internal memory, a layer structure of the NPU internal memory, number information of processing units, information of special operation circuit units, etc. The compiler 2100a can determine a processing order of each layer based on the structure data of the neural processor and graph information of a neural network model to be compiled.

[0602] The optimizer 2110a can perform a task of optimizing a neural network model expressed in a directed acyclic graph (DAG) according to a neural processor selected from among a plurality of neural processors 2200a. A user can select at least one of various optimization options provided by the optimizer 2110a online through the user device 1000a.

[0603] For example, the optimizer 2110a can provide an option to convert an integer parameter to a specific bit width. The specific bit width can be one bit width in a range of 2 bits to 16 bits. Accordingly, the optimizer 2110a can convert a neural network model based on a floating point parameter to a neural network model based on an integer parameter. At this time, the plurality of neural processors 2200a are designed to process the integer parameter. For example, the optimizer 2110a can convert a neural network model based on a nonlinear trigonometric function operation to a neural network model based on a piecewise linear function approximation. At this time, the plurality of neural processors 2200a can be designed to process the piecewise linear function approximation operation. For example, the optimizer 2110a can apply various optimization algorithms to reduce the size of a weight value, a feature map, etc. of the neural network model. For example, the optimizer 2110a can improve an accuracy degradation problem of the neural network model optimized and improved by various retraining algorithms.

[0604] The verification module 2120a can perform a verification function to determine whether the plurality of neural processors 2200a can run the customer's neural network model. The verification module 2120a can analyze the structure of the optimized neural network model, check whether each layer operator is supported by hardware in the plurality of neural processors 2200a, and determine whether the neural network model is executable. If not, a separate error report file is generated and reported to the user.

[0605] The code generation module 2130a can optimize the neural network model determined to be executable by the verification module 2120a through the optimization module 2110a, and generate machine code for running in the selected neural processor among the plurality of neural processors 2200a for the optimized neural network model. The generated machine code is provided to the corresponding neural processor, so that performance evaluation of the plurality of neural processors 2200a can be performed.

[0606] For example, first machine code corresponding to the first neural network model can be generated for a first neural processor among the plurality of neural processors 2200a. Second machine code corresponding to the first neural network model can be generated for a second neural processor among the plurality of neural processors 2200a. Third machine code corresponding to the first neural network model can be generated for a third neural processor among the plurality of neural processors 2200a. Fourth machine code corresponding to the first neural network model can be generated for a fourth neural processor among the plurality of neural processors 2200a.

[0607] Figure 18a A block diagram illustrating a structure of an optimization module of a neural network model processing apparatus according to another example of the disclosure.

[0608] The optimization module (Optimizer) 2110a can optimize the neural network model based on a plurality of compilation options.

[0609] More specifically, the optimization module 2110a can set the compilation options based on hardware information of the neural processor 2200a.

[0610] Also, the optimization module 2110a can set the plurality of compilation options considering characteristics of parameters of the neural network model (e.g., size of weight values, size of feature maps, etc.) and inference accuracy degradation characteristics.

[0611] The optimization module 2110a can set a plurality of compilation options, which can be at least one of a pruning option, a quantization option, a model compression option, a knowledge distillation option, an outlier alleviation option, a parameter refinement option, and a retraining option.

[0612] The pruning option can provide a technique capable of reducing the amount of computation of the neural network model. The pruning algorithm can be configured to replace smaller values close to 0 with 0 among the weight values of all layers of the neural network model. The plurality of neural processing units 2200a can skip the multiplication operation related to the 0 weight value to improve the operation speed of the convolution and reduce power consumption, and can reduce the parameter size of the machine code of the neural network model to which the pruning option is applied. Further description, when a certain weight value parameter is replaced with 0 according to the pruning option, the pruning algorithm can provide substantially the same effect as cutting off the connection net of the neural network model having the corresponding weight value data. For example, the pruning option can provide a first pruning option based on size that removes the weight value having a small size and a second pruning option based on percentage that removes a certain proportion of the smallest weight value.

[0613] The quantization option can provide a technique capable of reducing the size of the parameters of the neural network model. The quantization algorithm can be configured to selectively reduce the number of bits of the weight values and the feature maps of each layer of the neural network model. When the number of bits of a certain feature map and a certain weight value is reduced according to the quantization option, the parameter size of the machine code of the neural network model to which the quantization option is applied can be reduced. For example, if the quantization option is applied, a 32-bit parameter representing a floating point can be converted into a 2-bit to 16-bit parameter representing an integer.

[0614] The model compression option can provide a technique of compressing the weight value parameters or the feature map parameters of the neural network model. The model compression technique can be implemented using a compression technique known in the art. Accordingly, the parameter size of the machine code of the neural network model to which the model compression option is applied can be reduced. The model compression option can be selectively provided to the neural processing units 2200a including a decompression decoder.

[0615] A knowledge distillation option can provide a technique of transferring knowledge obtained in a complex model (a teacher model) to a smaller and simpler model (a student model). In a knowledge distillation algorithm, the parameter size of the teacher model is generally larger than that of the student model, and has a higher accuracy. For example, in the retraining option described later, a neural network model trained with a floating point 32-bit parameter can be set as a teacher model, and a neural network model to which various optimization options are applied can be set as a student training model, and the accuracy of the student model can be improved through the knowledge distillation option. Among them, the student model can be a model in which at least one of the pruning option, the quantization option, the model compression option, and the retraining option is selected.

[0616] A parameter refinement option can be executed together with the quantization option. In order to reduce errors that can occur due to quantization and improve computational performance due to quantization while maintaining the accuracy of the neural network model, optimization can be performed on parameters required for the quantization process. According to the parameter refinement option, optimal values can be calculated for the scale value and the offset value used for floating point parameter quantization of the neural network model, respectively.

[0617] An outlier alleviation option can be executed together with the quantization option. In the input value and the weight value of the neural network model, outliers can be included according to actual data, which can be a cause of amplifying errors in the quantization process. In order to effectively quantize, outliers need to be appropriately compensated. According to the outlier alleviation option, outliers included in the input parameter and the weight value parameter can be adjusted using an adjustment value for adjusting outliers before MAC operation.

[0618] A retraining option is a technique that can compensate for a decrease in inference accuracy when various optimization options are applied. For example, when the quantization option, the pruning option, and the model compression option are applied, the accuracy of the neural network model inferred in the plurality of neural processing units 2200a can decrease. At this time, an option that can retrain the neural network model on which pruning, quantization, and / or model compression is performed online can be provided. The inference accuracy of the neural network model completed by retraining can be improved again.

[0619] In particular, the retraining option can include a transfer learning option, a pruning aware retraining option, a quantization aware retraining option, and a quantization aware self-distillation option, etc.

[0620] The quantization aware retraining (QAT) option is an option that integrates quantization in a training step of a neural network model to make the model reflect quantization errors and fine-tune weight values. The quantization aware retraining algorithm can include loss function, gradient calculation, and optimization algorithm modification. The quantization aware retraining option can compensate for quantization errors by performing fine-tuning that retrains in a direction that minimizes a loss caused by quantization after a trained neural network model is quantized by an artificial neural network.

[0621] The quantization aware self-distillation option is used to prevent underfitting problems and perform QAT during retraining, and in minimizing a difference (loss) between a predicted value as a result of model operation and a label value of training data, a difference (loss) between the predicted value and a result value of a quantization simulated model operated with respect to the same parameters can be reflected together and retraining can be performed. In an example, according to the quantization aware self-distillation option, when a difference between a predicted value of a pre-trained model using parameters expressed in 32-bit floating point and an actual result value is a first loss, and a difference between a predicted value of a quantization simulated model with respect to the same parameters and the predicted value of the pre-trained model is a second loss, the parameters can be updated during retraining of the pre-trained model so that the first loss is minimized. The parameters can be updated during retraining of the quantization simulated model so that the second loss is minimized.

[0622] When the QAT is also applied to the Pre-trained model pre-trained with data augmentation, underfitting occurs due to over-regularization, and thus accuracy is degraded. In order to prevent such a problem, pruning-aware self-distillation can be performed. According to the pruning-aware self-distillation, a difference between a prediction value of a quantization simulation using the same parameters and a prediction value of the pre-trained model can be reflected, and thus degradation of accuracy that can occur due to over-regularization can be prevented.

[0623] A pruning-aware retraining (PAT) option is an option of fine-tuning remaining weight values after identifying and removing less important weight values in a trained neural network model. A pruning criterion can use weight value size, activation value, sensitivity analysis, etc. The pruning-aware retraining option can reduce the size of the neural network model, improve inference speed, improve overfitting problems during retraining, etc.

[0624] A transfer learning option means an option in which a neural network model transfers knowledge learned in one job to other related jobs to learn. The transfer learning algorithm is effective in a case where there is not enough data at an early stage or a lot of computational resources are required to train a neural network model at an early stage.

[0625] Without limitation, the optimization module 2110a can provide an AI-based optimization option. The AI-based optimization algorithm is a method of generating a best lightweight neural network model by exploring a neural network model structure using an AI reinforcement learning method after various algorithms of a compilation option, or is a method of obtaining an optimal lightweight result by performing a lightweight process by an AI itself inside the optimization module 2110a, rather than a lightweight method based on a quantization algorithm, a pruning algorithm, a retraining algorithm, a model compression algorithm, and a model compression algorithm.

[0626] Figure 18a A block diagram of an interface for selecting a plurality of neural processors and a compilation option of a neural network model processing apparatus according to another example of the disclosure.

[0627] The user interface can be displayed in the display apparatus 1140a of the user device 1000a after the user accesses the server 3000a using the user device 1000a.

[0628] Display device 1140a displays two parts: an NPU selection section 5100a and a compilation options section 5200a. Users can select one or more NPUs in the NPU selection section 5100a to simulate neural network models using more than one evaluation dataset. The NPU selection section 5100a displays four types of NPUs: DX-M1, DX-H1, DX-V1, and DX-V2. Users can select the NPU to be used for online simulation for performance evaluation. Figure 18a As an example, the example shows a scenario where a DX-M1 is selected for testing and evaluation.

[0629] The compiler options section 5200 displays pre-defined options to facilitate user selection of compiler choices. (See also...) Figure 18b The compiler options section 5200a provides a first preset option, a second preset option, and a third preset option. However, this disclosure is not limited to the preset options described herein, and may provide a wider variety of preset options.

[0630] For ease of description, the following description uses preset quantization algorithms as an example. From a specific perspective, each preset option may be the most effective quantization preset option; however, this disclosure is not limited to this. Users can consider the characteristics of each preset option to select at least one preset option.

[0631] For example, the first preset option is to execute only a quantization algorithm that converts the 32-bit floating-point data of the trained neural network model into 8-bit integer data. However, this disclosure is not limited to 8 bits and can be set to a bit width between 2 bits and 16 bits, the bit width being limited to a specific bit width depending on the hardware architecture of the selected neural processor. That is, the first preset option is a quantization algorithm that runs after the neural network model has been trained, and therefore can be referred to as Post Training Quantization (PTQ) or quantization. The first preset option can use mathematical formulas 1 to 3. The first preset option typically completes quantization within a few minutes, thus having the advantage of rapid quantization execution. Therefore, it has the advantage that the results of power consumption, processing speed, etc. of the user-provided neural network model can be quickly confirmed among the user-selected neural processors in multiple neural processors 2200a.

[0632] For example, a first preset option consisting of a first quantization option can be provided to the user as a DXNN Lite option. When the first quantization option is used, no further training is required, and therefore the retraining step of the neural network model can be omitted.

[0633] For example, the second preset option can be an option of performing layer-wise retraining algorithm for the neural network model after performing quantization of converting the 32-bit floating point data of the neural network model into 8-bit integer data. However, the present disclosure is not limited to 8-bit, and can be set to have a bit width of between 2-bit and 16-bit, and the bit width can be limited to a specific bit width according to the hardware structure of the selected neural processor.

[0634] That is, the second preset option can be an option of further performing layer-wise retraining algorithm with the neural network model to which the first preset option is applied as an input model. Accordingly, the second preset option can be an option of an algorithm combination of various retraining options provided by the quantization algorithm and the optimization module 2110a.

[0635] That is, the second preset option calculates the quantization loss function after quantizing data corresponding to a part of the plurality of layers of the neural network model. Then, the quantization loss function is calculated after quantizing data corresponding to another part of the plurality of layers of the neural network model. The above operations are repeatedly performed, and quantization is performed only for a part of the plurality of layers of the neural network model for which the quantization loss function is minimized, to optimize the quantization algorithm. For the second preset option, layer-wise retraining can be performed in a direction in which the difference between the floating point data (e.g., floating point 32) and the integer data (e.g., integer 8) of the feature map is minimized, and thus has an advantage in that retraining can be performed even without a training data set. The second preset option generally completes quantization within several hours, and thus has an advantage in that quantization can be performed within a proper time. Accordingly, it has an advantage in that the accuracy of the neural network model provided by the user can be further improved in the neural processor selected by the user among the plurality of neural processors 2200a, compared to the first preset option.

[0636] For example, the second preset option constituted by the second quantization option can be provided to the user as a service name of the DXNN pro. In this case, for the second quantization option, a retraining step of the neural network model can be required because the layer-wise retraining algorithm for the neural network model is performed.

[0637] For example, the third preset option is an option of performing a quantization aware retraining (QAT) algorithm after performing a quantization algorithm, i.e., converting 32-bit data of a floating point (FP) representing a neural network model into 8-bit data of an integer (INT) representing the neural network model. However, the present disclosure is not limited to 8-bit, and can be set to have a bit width of between 2-bit and 16-bit, and the bit width can be limited to a specific bit width according to a hardware structure of a selected neural processor.

[0638] That is, the third preset option can be an option of further performing a quantization aware retraining algorithm with respect to a neural network model in which the first preset option has been performed as an input model. Accordingly, the third preset option can be an option of a combination of one algorithm among various retraining options provided by the quantization algorithm and the optimization module 2110a.

[0639] That is, in the third preset option, the quantization aware retraining algorithm performs fine-tunning in a direction of minimizing deterioration of inference accuracy caused by quantization after quantizing the trained neural network model. However, in order to perform retraining in a direction of minimizing deterioration of inference accuracy caused by quantization, a user needs to provide a training dataset of the neural network model. In addition, an evaluation dataset can be required in order to prevent overfitting during retraining. Specifically, the quantization aware retraining algorithm performs retraining in such a manner that machine codes of the quantized neural network model and the training dataset are input to a corresponding neural processor among the plurality of neural processors 2200a, and compensates for deterioration of inference accuracy caused by quantization error. The third preset option has an advantage of securing relatively higher inference accuracy than the first and second preset options, but can be performed in a case where accuracy is emphasized since quantization can be completed within several days.

[0640] For example, the third preset option constituted by the third quantization option can be provided to a user as a service name of the DXNN master. In the third quantization option, since retraining is performed based on inference accuracy of a neural network model, a retraining step of the neural network model can be required. In the process of retraining in a direction of minimizing loss caused by quantization, the user needs to provide a training dataset and / or an evaluation dataset of the neural network model for the quantization aware retraining algorithm of the third quantization option. The training dataset is data necessary for the quantization aware retraining. The evaluation dataset is data that can be selectively used in order to improve an overfitting problem during retraining.

[0641] Figure 18bA block diagram of an interface for performance evaluation and proposal of a plurality of neural processors of a neural network model processing apparatus according to another example of the present disclosure.

[0642] In Figure 19a to Figure 19d , results of simulation and evaluation using two different NPUs are shown. The upper left box represents the results using DX-M1 NPU, and the upper box represents the results using DX-H1 NPU. In the lower end box, the NPU selection recommended based on the performance parameters of the two different NPUs can be displayed.

[0643] Figure 19a A block diagram of the structure of one neural processor of a neural network model optimization apparatus according to another example of the present disclosure.

[0644] Specifically, in Figure 19b , the internal structure of a first neural processor in the plurality of neural processors is shown. Also, in Figure 19c , the internal structure of a second neural processor in the plurality of neural processors is shown. Also, in Figure 19d , the internal structure of a third neural processor in the plurality of neural processors is shown. Also, in Figure 19a , the internal structure of a third neural processor in the plurality of neural processors is shown.

[0645] Figure 19a The first neural processor 2200a shown is configured to include a processing unit array 2210a, an NPU internal memory 2220a, and an NPU controller 2230a.

[0646] For example, the first neural processor 2200a can be configured to include the processing unit array 2210a, the NPU internal memory 2220a configured to store a neural network model that can be inferred in the processing unit array 2210a, or to store at least part of the data of the neural network model, and the NPU controller 2230a configured to control the processing unit array 2210a and the NPU internal memory 2220a. The neural network model can be machine code compiled with various optimization options.

[0647] The NPU controller 2230a is configured to control the processing unit array 2210a operation and the reading and writing sequence of the NPU internal memory 2220a for inference operation of the first neural processor 2200a.

[0648] The NPU controller 2230a can be configured to control the processing unit array 2210a and the NPU internal memory 2220a according to machine code. The NPU controller 2230a is configured to control the processing unit array 2210a and the NPU internal memory 2220a according to the operation schedule defined by the machine code, step by step. Therefore, the neural processor can process the operations of each layer sequentially according to the structure of the neural network model. The NPU controller 2230a can obtain the memory address storing the feature maps and weight values ​​of the neural network model, or determine the memory address to be stored.

[0649] Processing unit array 2210a refers to a structure in which multiple processing units PE1 to PE12 are arranged in an array. Each processing unit may be configured to include a multiply and accumulate (MAC) arithmetic unit and / or an arithmetic logic unit (ALU). However, the examples in this disclosure are not limited thereto.

[0650] exist Figure 19a The example shows multiple processing units, but it can also be configured in parallel by multiple multipliers and adder trees instead of a MAC within a single processing unit. In this case, the processing unit array 2210a can also refer to at least one processing unit including multiple arithmetic units.

[0651] The processing unit array 2210a is configured to include multiple processing units PE1 to PE12. Figure 19b The multiple processing units PE1~PE12 shown are merely examples for ease of description, and the number of processing units PE1~PE12 is unlimited. The size or number of processing unit arrays 2210a can be determined based on the number of processing units PE1~PE12. The size of processing unit array 2210a can be implemented in the form of an N×M matrix, where N and M are integers greater than 0. Processing unit array 2210a may include N×M processing units. That is, there may be more than one processing unit.

[0652] The size of the processing unit array 2210a can be designed to take into account the characteristics of the neural network model. Furthermore, the number of processing units can be determined by considering the data size of the running neural network model, the required operating speed, and the required power consumption. The data size of the neural network model can be determined corresponding to the number of layers in the neural network model and the size of the weight values ​​in each layer.

[0653] Therefore, the size of the processing unit array 2210a of the first neural processor 2200a of another example of the present disclosure is not limited. The more the number of processing units of the processing unit array 2210a increases, the more the parallel computing capability of the running neural network model increases, but the manufacturing cost and the physical size also increase.

[0654] For example, as shown in FIG. 21A, the second neural processor 2200a-1 can include two processing unit arrays 2210a-1, 2210a-2. The two processing unit arrays 2210a-1, 2210a-2 can respectively include a plurality of processing units PE1 to PE12 and be grouped. Figure 19c

[0655] As another example, as shown in FIG. 21B, the third neural processor 2200a-2 can include four processing unit arrays 2210a-1, 2210a-2, 2210a-3, 2210a-4. The four processing unit arrays 2210a-1, 2210a-2, 2210a-3, 2210a-4 can respectively include a plurality of processing units PE1 to PE12 and be grouped. Figure 19d

[0656] As still another example, as shown in FIG. 21C, the fourth neural processor 2200a-3 can include eight first neural processors 2200a. Figure 20

[0657] The eight first neural processors 2200a respectively divide the operation of processing the neural network model. Therefore, the processing speed of the fourth neural processor 2200a-3 can be further improved.

[0658] Therefore, the fourth neural processor 2200a-3 can further include a control part that allocates the respective operations of the eight neural processors.

[0659] The description of the characteristics of the first to fourth neural processors and the processing model is the same as the foregoing.

[0660] Figure 21 A block diagram showing the structure of a plurality of neural processors of another example of the present disclosure.

[0661] The plurality of neural processors 2200a can include a plurality of kinds of neural processors. Also, at least one or more of the same kind of neural processor can be configured.

[0662] ​​​For example, the “DX-M1” neural processor can configure a plurality of and form a first group G1, the “DX-H1” neural processor can configure a plurality of and form a second group G2, the “DX-V1” neural processor can configure a plurality of and form a third group G3, and the “DX-V2” neural processor can configure a plurality of and form a fourth group G4. The groups of the plurality of kinds of neural processors can be referred to as an NPU FARM 2180a. The NPU FARM 2180a can be a cloud form of an NPU system configured to respond to performance evaluation requests of a plurality of users who access online in real time.

[0663] Also, the plurality of neural processors 2200a included in the first to fourth groups G1 to G4 can be used for performance evaluation all at once according to a user selection, or only a part of them can be used for performance evaluation.

[0664] User data that needs to be secured can be stored in the server 3000a or stored in the storage module 2400a of the neural network model optimization apparatus 2000a.

[0665] The at least one neural processor 2200a for operation can communicate with the server 3000a to receive at least one specific neural network model for evaluating the performance of the neural processor and at least one specific evaluation data set input in the neural network model. That is, the neural processor 2200a can receive user data required for performance evaluation.

[0666] A method of evaluating the performance of a neural network model using the above-described neural network model performance evaluation system will be described below.

[0667] Figure 21 A flowchart of a neural network model performance evaluation method for describing another example of the present disclosure.

[0668] Referring to Figure 20 The neural network model performance evaluation method S100 includes a neural processor kind selection step S110, a compilation option selection step S120, a neural network model upload step S130, a neural network model compilation and processing step S140, and a neural network model processing result reporting step S150.

[0669] In the neural processor kind selection step S110, a user selects a kind of neural processor to be subjected to performance evaluation. The kind of neural processor can differ according to a product group of a neural processor sold by a specific company.

[0670] As Figure 22As shown, the "DX-M1" neural processor can be configured in multiple and form a first group G1, the "DX-H1" neural processor can be configured in multiple and form a second group G2, the "DX-V1" neural processor can be configured in multiple and form a third group G3, and the "DX-V2" neural processor can be configured in multiple and form a fourth group G4.

[0671] The user selects the neural processor to be evaluated in performance among the "DX-M1" neural processor, the "DX-H1" neural processor, the "DX-V1" neural processor, and the "DX-V2" neural processor.

[0672] Moreover, the server 3000a can select at least one neural processor among the selected specific kind of neural processors to be used in the NPU FARM. At this time, the selected specific kind of neural processor is not limited to one kind, but can be multiple kinds.

[0673] Then, in the compiling option selection step S120, at least one of multiple compiling options of the neural network model to be processed by the selected at least one neural processor is selected.

[0674] More specifically, in the compiling option selection step S120, the compiling options can be set based on the hardware information of the neural processor 2200a.

[0675] Moreover, in the compiling option selection step, the multiple compiling options can be set according to the selection of the user. Among them, the description of the advantages and disadvantages of each compiling option can be displayed in the user device 1000a. Thus, the user can select and choose the multiple compiling options in the form required by the user. That is, the performance evaluation system 10000 can also provide the compiling options in a user customization manner rather than a preset option manner according to the detailed requirements of the user.

[0676] The above-mentioned compiling options can be at least one of a pruning algorithm, a quantization algorithm, a parameter refinement algorithm, an outlier alleviation algorithm, a model compression algorithm, a knowledge distillation algorithm, a retraining algorithm, and an AI based model optimization algorithm.

[0677] Alternatively, the compiling options can be configured to select one of the prepared preset options.

[0678] Then, in a neural network model uploading step S130, the selected at least one specific neural network model used for evaluating the performance of the neural processor is uploaded to the server 3000a. The neural network model uploading step S130 can be referred to as a user data uploading step.

[0679] Then, in a neural network model compiling and processing step S140, the input neural network model is compiled according to the selected compiling option, and the compiled machine code is input to the selected neural processor in the NPU FARM for processing.

[0680] In a neural network model processing result reporting step S150, the performance evaluation result of the neural processor processing the compiled neural network model can be reported. The performance evaluation result report can be stored in the user’s account or sent to the user’s email address. However, it is not limited thereto, and the performance evaluation result can be provided to the user in various ways. The performance evaluation result can also be disposed as user data, and the security policy applied to the user data is applied.

[0681] Further description, in the neural network model processing result reporting step S150, it is first determined whether the compiled neural network model can be processed by the plurality of neural processors 2200a.

[0682] When the compiled neural network model cannot be processed by the plurality of neural processors 2200a, in the neural network model processing result reporting step S150, the layer of the plurality of layers of the neural network model that cannot be processed by the plurality of neural processors 2200a can be reported.

[0683] Further, the layer of the plurality of layers of the neural network model that cannot be processed by the plurality of neural processors 2200a can be processed by the graphics processor 2300a.

[0684] When the compiled neural network model can be processed by the plurality of neural processors 2200a, in the neural network model processing result reporting step S150, the processing performance of the plurality of neural processors 2200a can be reported.

[0685] The indicators of the processing performance can be the temperature curve, power consumption (Watt), Trillion Operations Per Second per Watt (TOPS / W), Frame Per Second (FPS), Inference Per Second (IPS), and accuracy of the neural processor.

[0686] Wherein, when the user does not provide an evaluation dataset, the neural network model performance evaluation system 10000 can analyze the input data size of the neural network model, generate corresponding virtual data, and perform performance evaluation using the generated virtual data. For example, the size of the virtual data can be (224 x 224 x 3), (288 x 288 x 3), (380 x 380 x 3), (515 x 512 x 3), (640 x 640 x 3), etc., but the disclosure is not limited to the size of the virtual data. That is, even if a dataset that can evaluate inference performance is not provided, performance evaluation results such as power consumption, TOPS / W, FPS, IPS, etc. of the neural processor can be generated. However, at this time, inference accuracy evaluation results are not provided.

[0687] According to another example of the disclosure, the user can quickly learn in advance where the user's neural network model can be run in a specific NPU before purchasing the specific NPU.

[0688] According to another example of the disclosure, the user can quickly learn in advance how the user's neural network model is run in performance when driven in a specific NPU before purchasing the specific NPU.

[0689] According to another example of the disclosure, when each NPU is connected by a server by category, the user can evaluate online whether the user's neural network model is driven and obtains a result for each NPU that can be purchased.

[0690] Therefore, the performance evaluation system 10000 can provide the user with performance and price information of the neural processor required to implement the AI service developed by the user, helping the user to quickly make a purchase decision.

[0691] Figure 22 A flowchart for describing a neural network model performance evaluation method according to another example of the disclosure.

[0692] Referring to Figure 20 , the neural network model performance evaluation method S200 includes a neural processor category selection step S110, a compilation option selection step S120, a neural network model and dataset upload step S230, a neural network model compilation and processing step S140, and a neural network model processing result reporting step S150.

[0693] In the neural processor category selection step S110, the category of the neural processor in which the user will perform performance evaluation is selected. The category of the neural processor can differ according to the product group of the neural processor sold by a specific company.

[0694] As Figure 23As shown, the "DX-M1" neural processor can be configured in multiple and form a first group G1, the "DX-H1" neural processor can be configured in multiple and form a second group G2, the "DX-V1" neural processor can be configured in multiple and form a third group G3, and the "DX-V2" neural processor can be configured in multiple and form a fourth group G4.

[0695] The user can select a neural processor to be subjected to performance evaluation among the "DX-M1" neural processor, the "DX-H1" neural processor, the "DX-V1" neural processor, and the "DX-V2" neural processor.

[0696] Moreover, the server 3000a can select at least one neural processor among the selected specific kind of neural processors to be used in the NPU FARM. At this time, the selected specific kind of neural processor is not limited to one kind, but can be multiple kinds.

[0697] Then, in the compilation option selection step S120, at least one of multiple compilation options of the neural network model to be processed by the selected at least one neural processor is selected.

[0698] More specifically, in the compilation option selection step S120, the compilation options can be set based on the hardware information of the neural processor 2200a.

[0699] Moreover, in the compilation option selection step, the multiple compilation options can be set according to the selection of the user. Among them, the description of the advantages and disadvantages of each compilation option can be displayed in the user device 1000a. Thus, the user can select and choose the multiple compilation options in the form required by the user. That is, the performance evaluation system 10000 can also provide the compilation options in a user customization manner rather than a preset option manner according to the detailed requirements of the user.

[0700] The above-mentioned compilation options can be at least one of a pruning algorithm, a quantization algorithm, a parameter refinement algorithm, an outlier alleviation algorithm, a model compression algorithm, a knowledge distillation algorithm, a retraining algorithm, and an AI based model optimization algorithm. Alternatively, the compilation options can be configured to select one of the prepared preset options.

[0701] Then, in a neural network model and data set upload step S230, at least one specific neural network model used to evaluate the performance of the selected neural processor and at least one specific evaluation data set input in the neural network model are uploaded to the server 3000a. The specific evaluation data set described above means an evaluation data set input in the neural network model optimization device 2000a for evaluating the performance of the neural network model optimization device 2000a. Among them, the neural network model and data set upload step S230 can be referred to as a user data upload step.

[0702] Then, in a neural network model compilation and processing step S140, the input neural network model is compiled according to the selected compilation option, and the compiled machine code and evaluation data set are input to the selected neural processor in the NPU FARM for processing.

[0703] In a neural network model processing result reporting step S150, the performance evaluation result of the neural processor processing the compiled neural network model can be reported. The performance evaluation result report can be stored in the us...

Claims

1.A neural network model conversion method, comprising: receiving a first neural network model including one or more functions; converting the one or more functions and generating a second neural network model in a directed acyclic graph form including one or more graph modules; obtaining maximum and minimum values of a parameter input in the one or more graph modules and calculating one or more scale values; updating the parameter based on the one or more scale values; and generating a third neural network model in a machine code form executable in a specific neural processor including the updated parameter. 2.The neural network model conversion method of claim 1, wherein the third neural network model is generated based on hardware information of the specific neural processor. 3.The neural network model conversion method of claim 1, wherein the step of updating the parameter is performed based on one or more selected compilation options. 4.The neural network model conversion method of claim 1, wherein the step of updating the parameter is performed by selecting one or more of a quantization option, an outlier mitigation option, a parameter refinement option, a layer-wise training option, a quantization-aware retraining option, a quantization-aware self-distillation option, and a pruning option. 5.The neural network model conversion method of claim 1, wherein the step of updating the parameter is performed according to one or more of a quantization algorithm, an outlier mitigation algorithm, a parameter refinement algorithm, a layer-wise training algorithm, a quantization-aware retraining algorithm, a quantization-aware self-distillation algorithm, and a pruning algorithm. 6.The neural network model conversion method of claim 1, wherein the third neural network model is compiled to cause multiplication and accumulation operations included in the third neural network model to be processed in a processing unit of the specific neural processor. 7.The neural network model conversion method of claim 1, wherein the compiled third neural network model includes multiplication and accumulation operations including at least one of a convolution operation, a general matrix multiplication operation, and a matrix multiplication operation. 8.The neural network model conversion method of claim 1, wherein a weight value parameter of the third neural network model input to a processing unit of the specific neural processor is compiled to an integer form. 9.The neural network model conversion method of claim 1, wherein an input parameter of the third neural network model input to a processing unit of the specific neural processor is compiled to an integer form. 10.The neural network model conversion method of claim 1, wherein the third neural network model is compiled to cause activation function operations included in the third neural network model to be processed in an activation function circuit of the specific neural processor. 11.The neural network model conversion method of claim 1, wherein an output parameter of the third neural network model of a processing unit of the specific neural processor is in an integer form, and the third neural network model is compiled to cause the output parameter to be dequantized by a dequantization circuit of the specific neural processor. 12.The neural network model conversion method of claim 1, wherein, an output parameter of an activation function circuit of the specific neural processor of the third neural network model is in a floating point form, and the third neural network model is compiled such that the output parameter is quantized by a quantization circuit of the specific neural processor. 13.The neural network model conversion method of claim 1, wherein, the updated parameter of the third neural network model includes one or more weight value parameters. 14.The neural network model conversion method of claim 1, wherein, the third neural network model includes the one or more scale values for a dequantization operation of a dequantization circuit of the specific neural processor. 15.A neural processor configured to run inference dedicated machine code including weight value parameters of a neural network model and scale values corresponding to the weight value parameters, the neural processor comprising: a first circuit for on-chip memory configured to store at least a portion of data of the machine code; a second circuit for one or more processing units configured to process the weight value parameters and input parameters and generate output parameters; a third circuit for an activation function unit configured to apply an activation function to the output parameters and generate activation parameters; and a fourth circuit for a control unit configured to control the first to third circuits according to the machine code. 16.The neural processor of claim 15, wherein, the weight value parameters included in the machine code are updated according to one or more of a quantization algorithm, an outlier mitigation algorithm, a parameter refinement algorithm, a layer-wise training algorithm, a quantization-aware retraining algorithm, a quantization-aware self-distillation algorithm, and a pruning algorithm. 17.The neural processor of claim 15, further comprising: a dequantization circuit configured to dequantize the output parameters of the second circuit according to the machine code. 18.The neural processor of claim 15, further comprising: a quantization circuit configured to quantize the activation parameters of the third circuit according to the machine code. 19.The neural processor of claim 15, wherein, the second circuit is configured to process integer parameters, and the third circuit is configured to process floating point parameters. 20.A storage medium recording instructions which, when executed by one or more processors, cause the one or more processors to perform: a step of receiving a first neural network model including one or more functions; a step of converting the one or more functions and generating a second neural network model in a directed acyclic graph form including one or more graph modules; a step of obtaining maximum and minimum values of parameters input in the one or more graph modules and calculating one or more scale values; a step of updating the parameters based on the one or more scale values; and a step of generating a third neural network model including one or more functions corresponding to the updated parameters. The step of generating a third neural network model in a machine code form executable in the specific neural processor, including the updated parameters. The step of generating a third neural network model in a machine code form executable in the specific neural processor, including the updated parameters.

Images