Neural processing unit comprising dedicated circuitry for activation function computation and method therefor

Abstract

A function approximation unit according to the examples of the present disclosure is provided. The function approximation unit may include: a programmable function approximation (PAF) circuit configured to directly compute a piecewise function according to a pre-stored computation coefficient; one or more dedicated function circuits configured to process at least one specific function among a reciprocal, a reciprocal square root, and a negative exponential function through a fixed hard-wired computation pipeline; and control logic configured to selectively activate one of the PAF circuit or the dedicated function circuit according to the type of non-linear function to be processed.

Description

Technology Field

[0001] The present disclosure relates to a neural processing unit and a method comprising a dedicated circuit for activation function operations. Background Technology

[0002] Humans possess intelligence capable of recognition, generation, classification, inference, prediction, and control / decision-making. Artificial intelligence (AI) refers to the artificial imitation of human intelligence.

[0003] The human brain is composed of countless nerve cells called neurons, and each neuron is connected to hundreds to thousands of other neurons through connection points called synapses. A neural network (ANN) model is a system that models the operating principles of biological neurons and the relationships between them in order to mimic human intelligence. In other words, a neural network is a system that connects nodes, which mimic neurons, in a layered structure.

[0004] As neural network technology advances, the applications of neural network inference services utilizing big data-based training are becoming increasingly diverse. These inference services train neural networks on vast datasets, enabling them to infer from diverse and complex data through the resulting learned models. However, as the functionality and accuracy required for neural network inference services continue to increase, the demands for neural network model parameter size, computational complexity, and memory bandwidth are growing exponentially. Consequently, performance requirements for processors and memory capable of supporting these inference tasks are also rising. To address these demands, neural network inference services are increasingly being deployed on cloud computing-based servers that provide efficient processing capabilities for large-scale data operations.

[0005] Generative AI models are increasingly being developed to operate in diverse environments. These models include various neural network models, such as language models for conversational systems, stable diffusion models that generate images from text input, and transformer models that predict and decide future activities based on a series of events in specific situations. The computational cost of generating responses to input queries based on these generative AI models is very high. For example, the commercialization of Large Language Models (LMs) among generative AI models has begun in earnest.

[0006] Generative AI models are very diverse, and new models are continuously being released into the market. Representative generative AI models include, for example, GPT, LLaMA, DeepSeek, Mistral, DBRX, Qwen, Nova, Jamba, PaLM, PanGu, OPT, Gemini, Falcon, BLOOM, CLIP, RoBERTa, ALBERT, ELECTRA, LXNet, BERT, BART, CTRL, DALL·E, Stable Diffusion, Midjourney, Imagen, DeepFloyd, Runway Gen, DreamBooth, StyleGAN, BigGAN, Sora, Pika Labs, Stable Video Diffusion, Imagen Video, Make-A-Video, Phenaki, Dreamix, VideoPoet, Text2Video-Zero, CogVideo, etc.

[0007] However, providing generative AI services requires high-performance General-Purpose Graphics Processing Units (GPGPUs). Yet, the total power consumption of GPGPUs operating globally to process these LLMs has reached a level comparable to the total power consumption of entire nations, and this excessive energy demand is posing a serious problem for the commercialization of LLMs. Building GPGPU-based data centers to provide LLM services requires carefully evaluating power plant capacity to determine whether the necessary energy can be supplied, and the difficulty in meeting the energy demands of these data centers in many urban areas can serve as a major barrier to the widespread deployment of large-scale AI models. The problem to be solved

[0008] Neural networks are classified into 'single-layer neural networks' and 'multi-layer neural networks' depending on the number of layers. A typical multi-layer neural network consists of an input layer, a hidden layer, and an output layer. (1) The input layer is a layer that accepts input values. The number of input layers is equal to the number of input variables. (2) The hidden layer is located between the input layer and the output layer and is a layer that receives signals from the input layer, extracts features, and transmits them to the output layer. (3) The output layer is a layer that receives signals from the hidden layer and outputs them to the outside.

[0009] Deep neural networks (DNNs) with an increased number of hidden layers were introduced to achieve higher levels of artificial intelligence. Among DNNs, the Transformer artificial neural network is a DNN based on attention technology. Transformers utilize numerous matrix multiplication operations. Transformers can obtain an attention score (Q, K, V), which is an operation value, using parameters such as input values ​​and queries (Q), keys (K), and values ​​(V). Transformers can process various inference operations based on the operation value (e.g., attention score (Q, K, V)). Transformers are demonstrating excellent performance in the field of generative artificial intelligence.

[0010] Transformers are utilized in various large-scale language models. For example, a language model takes a text-based query as input and generates a response. The response generated by the language model can be produced by inputting tokens generated as part of the response back into the language model. The cost of generating a response in such a language model is proportional to the number of parameters. Therefore, using a language model with a large number of parameters results in high computational costs for generating a response. Conversely, reducing the number of parameters can lower computational costs and improve response speed, but it may lead to a degradation in the quality of the generated response.

[0011] Furthermore, the inference accuracy of a neural network model can vary depending on the characteristics of the activation function implemented in the neural processing unit. In other words, the performance and efficiency of the neural network model are determined by the hardware implementation characteristics of the activation function processing circuitry within the neural processing unit. Additionally, neural network models that process activation functions with complex mathematical formulas can be processed using hardware accelerators. If a dedicated processor for neural network models is implemented in hardware, it may require a significant amount of chip area (i.e., the number of logic gates). Moreover, such chips may consume a substantial amount of power.

[0012] Most complex functions intended to be solved in neural network models are nonlinear. To address this problem, activation functions are mostly nonlinear functions.

[0013] Depending on the non-linearity characteristics of the activation function applied to at least one neural network model processed by the neural processing unit, the performance and efficiency of the neural network model processed in hardware may vary.

[0014] Activation functions used in neural network models, particularly non-linear functions involving logarithmic or exponential operations, pose a problem by significantly increasing the complexity of digital logic design when implemented in hardware. This complicates the structure of hardware arithmetic units, directly causing increased chip power consumption and reduced computational processing speed.

[0015] To address this problem, there is an approach to implement specific activation functions using hard-wired dedicated circuits, but this faces the following fundamental limitations.

[0016] First, there is a lack of flexibility. Since the hard-wired approach can only handle predefined, fixed functions, it is impossible to process activation functions that emerge or are modified due to advancements in AI technology independently without hardware changes. Consequently, these functions are replaced by inefficient software processing or act as a serious constraint that makes the application of new AI models difficult.

[0017] Second, resource inefficiency. There is a disadvantage in that the number of gates in the dedicated circuit increases exponentially depending on the computational complexity of the activation function to be processed, thereby raising the chip area and manufacturing costs.

[0018] Against this backdrop, the inventor of the present disclosure conceived an idea for a neural processing structure that can maximize hardware resource efficiency and minimize power consumption by hard-wired functions to selectively perform integer-based operations in specific sections and floating-point-based vector operations in other sections, depending on the characteristics of neural network operations. Through this, the inventor developed a new neural processing unit structure that can overcome the limitations of existing ASICs and dramatically improve the Power, Performance, Area (PPA) ratio.

[0019] However, neural processing units using conventional activation function approximation techniques, including the above approach, still faced the following obvious technical limitations.

[0020] First, there is the issue of reduced inference accuracy and inefficiency of hardware resources. Conventional polynomial approximation or lookup table (LUT) methods required excessive computation or massive hardware resources to ensure inference accuracy. In particular, when the data range of non-linear operations is wide and dynamically changing, such as in transformer-based models, numerous LUT entries were required to handle this using the LUT method. This caused significant area overhead during hardware implementation, which became a direct cause of increased chip manufacturing costs and reduced power efficiency.

[0021] Second, there is a structural bottleneck that slows down computation speed. Conventional transcendental function approximation relied on complex pre- and post-processing steps that mixed floating-point (FP) and integer (INT) representations. It is evident that this software instruction-based sequential processing method inevitably increases computation cycles, which reduces function throughput per unit of time and is a fundamental bottleneck that hinders the overall computational efficiency of the neural network.

[0022] Third, there is the problem of a lack of flexibility and scalability that cannot accommodate new models. As mentioned earlier, existing hard-wired processors can only handle predefined, fixed activation functions, presenting a clear limitation in that they cannot keep up with the advancement of new AI technologies.

[0023] In particular, there have recently been attempts to reduce the size of LUTs (e.g., Range-Invariant LUTs) by decomposing the range of input data into a scaling part and an invariant part normalized to a specific range, and approximating only the invariant part with a lookup table (LUT). However, this approach follows the following obvious limitations. Since the core of the technology still relies on the LUT lookup method, while it may partially improve memory efficiency by reducing the number of LUT entries, it requires multi-stage sequential processing steps such as input decomposition, LUT referencing, and scaling, which is the fundamental cause of computational latency. In other words, this is merely an incremental improvement on existing LUT methods and is far from achieving high-speed processing with a single pipeline without complex instruction control.

[0024] In order to overcome the complex limitations of the prior art described above, the inventor of the present disclosure has devised the following innovative solution.

[0025] First, to address the problem of lack of flexibility, it was recognized that a programmable methodology capable of approximating any form of activation function and an optimized hardware design to drive it are essential.

[0026] Furthermore, to achieve both accuracy and efficiency simultaneously, it was recognized that the most efficient approach to minimize the overall approximation error while reducing hardware complexity is to divide the function into multiple regions based on the morphological characteristics of the activation function to be approximated, and to individually program approximation parameters optimized for the characteristics of each region (e.g., linear or non-linear intervals).

[0027] Ultimately, in order to fundamentally solve the problem of reduced computational speed, the inventor of the present disclosure has developed a neural processing unit equipped with a function approximation unit capable of processing functions at high speed without complex instruction control by implementing each step of function approximation (e.g., input value preprocessing, approximation operation, result postprocessing) as an independent dedicated circuit instead of relying on software instructions and connecting them in a pipeline structure.

[0028] However, the problems of the present disclosure are not limited to those mentioned above, and other problems not mentioned will be clearly understood by a person skilled in the art from the description below. means of solving the problem

[0029] According to one embodiment of the present disclosure, a function approximation unit may be provided. The function approximation unit may include: a programmable function approximation (PAF) circuit configured to directly compute a piecewise function according to a pre-stored computation coefficient; one or more dedicated function circuits configured to process at least one specific function among a reciprocal, a reciprocal square root, and a negative exponential function through a fixed hard-wired computation pipeline; and control logic configured to selectively activate one of the PAF circuit or the dedicated function circuit according to the type of non-linear function to be processed.

[0030] According to one embodiment of the present disclosure, the PAF circuit may be configured to perform a linear function operation or a polynomial function operation having a degree of 1 or higher as a function of each interval using the operation coefficients.

[0031] According to one embodiment of the present disclosure, the dedicated function circuit may be configured to process the floating-point representation of an input value by decomposing it into an exponent and a mantissa. The PAF circuit may be configured to be used as a common operation unit that performs approximation operations on a specific normalized range of the decomposed mantissa.

[0032] According to one embodiment of the present disclosure, the computation coefficient used by the PAF circuit may be a constant value that is pre-optimized through artificial neural network learning to approximate the specific function within the specific normalized range.

[0033] According to one embodiment of the present disclosure, the control logic may be configured to perform power gating or clock gating to block power consumption of an inactive circuit.

[0034] According to one embodiment of the present disclosure, the function approximation unit may be configured to load the operation coefficient from an on-chip shared memory of the neural processing unit or from a register included in the function approximation unit.

[0035] According to one embodiment of the present disclosure, the neural processing unit includes a special function unit, and the function approximation unit may be configured to be included in the special function unit.

[0036] According to one embodiment of the present disclosure, a programmable function approximation circuit may be provided. The programmable function approximation circuit may include: a plurality of comparators configured to process an input value (X) in a floating-point format including a sign bit, an exponent, and a mantissa, and to determine the segment to which the input value (X) belongs by comparing a plurality of segment boundary values ​​(SB) with the input value (X); a selector configured to select a set of programmable operation coefficients corresponding to the determined segment from a plurality of stored sets of coefficients; and a hard-wired operation unit configured to directly perform a predefined arithmetic operation using the selected set of operation coefficients and the input value (X). The hard-wired operation unit may include a multiplier and an adder. The arithmetic operation may be a segment-wise linear function or a segment-wise polynomial function having a degree of one or more.

[0037] According to one embodiment of the present disclosure, the circuit may be configured to dynamically change the degree of the polynomial function by deactivating the operation of some circuits of the arithmetic unit through an enable signal.

[0038] According to one embodiment of the present disclosure, the floating-point format may be any one of Brain Floating-Point 16 (BF16), TensorFloat-32 (TF32), 8-bit floating-point (FP8), or Block Floating-Point (BFP).

[0039] According to one embodiment of the present disclosure, the set of programmable operation coefficients may be configured to be optimized in advance to approximate a specific interval of the objective function through artificial neural network learning, and to be stored as a constant value that is not changed at the time of execution.

[0040] According to one embodiment of the present disclosure, the artificial neural network learning may be configured to perform hardware-software joint optimization by taking into account the number of comparators included in the circuit.

[0041] According to one embodiment of the present disclosure, a neural processing unit may be provided. The neural processing unit may include a control unit that controls the operation of the neural processing unit; and a function approximation unit that receives a single instruction from the control unit and operates. The function approximation unit may include a hard-wired pipeline comprising, in response to the single instruction, a preprocessing circuit that preprocesses an input value, an approximation operation circuit that performs an approximation operation on the preprocessed value, and a postprocessing circuit that postprocesses the result of the approximation operation. The pipeline may be configured to autonomously perform multi-stage operations after receiving the single instruction without additional instruction intervention from the control unit and output a final approximation value.

[0042] According to one embodiment of the present disclosure, the approximation operation circuit is a programmable function approximation circuit.

[0043] According to one embodiment of the present disclosure, the function approximation unit may be configured to perform the multi-stage operation by selectively activating one of the programmable function approximation circuit of claim 1 or the dedicated function circuit of claim 1, depending on the target function specified by the single instruction.

[0044] According to one embodiment of the present disclosure, the preprocessing circuit may be configured to decompose the floating-point representation of an input value into an exponent part and a mantissa part, and the postprocessing circuit may be configured to combine the output of the approximation operation circuit with the exponent part.

[0045] According to one embodiment of the present disclosure, the function approximation unit may be included within a Vector Core. The single instruction may be configured to simultaneously trigger autonomous pipeline operations for each element of the vector data.

[0046] According to one embodiment of the present disclosure, the function approximation unit may be included within a scalar core. The single instruction may be configured to trigger autonomous pipeline operations on scalar data.

[0047] According to one embodiment of the present disclosure, the neural processing unit may further include means for storing computation coefficients optimized through offline learning in advance in coefficient registers within the pipeline before executing the single instruction.

[0048] Specific details of other examples are included in the detailed descriptions and drawings. Effects of the invention

[0049] According to the present disclosure, the following effects can be achieved.

[0050] First, computational performance and processing speed can be dramatically improved. The function approximation unit (FuncApx) of the present disclosure is implemented such that each step for complex transcendental function operations, such as reciprocals and the reciprocal of a square root, is used as an independent dedicated circuit. The function approximation unit connects each step for transcendental function operations as an independent dedicated circuit in a pipeline structure. In this way, the present disclosure eliminates overhead by executing multiple instructions sequentially, thereby enabling high-speed processing of complex function operations within a few clock cycles, which has the effect of reducing the total latency of neural network inference and maximizing throughput.

[0051] Second, it is possible to maximize the efficiency of hardware resources and minimize power consumption. By employing a programming method that divides activation functions into multiple regions based on their characteristics and applies optimized approximation parameters to each region, the use of unnecessarily complex computational circuits can be minimized, allowing for high accuracy to be achieved with simpler logic. This reduces the number of hardware gates, thereby decreasing chip area, and significantly lowers leakage current and dynamic power consumption by selectively activating only the necessary computational circuits. Consequently, it is possible to implement a neural processing unit with a significantly improved Power-to-Performance (PPA) ratio.

[0052] Third, high inference accuracy can be guaranteed. The present disclosure separates and processes the exponent and mantissa of an input value, and by normalizing the input range of the mantissa, which is the subject of the approximation operation, to a specific interval with a small amount of function change, approximation errors can be effectively reduced even with limited hardware resources. Accordingly, there is an effect of preventing a decrease in the inference accuracy of the neural network model.

[0053] Fourth, flexibility and scalability for future AI models can be secured.

[0054] The function approximation unit of the present disclosure is designed to be programmable, allowing for flexible adaptation to various activation functions that may emerge or be modified in the future simply by applying new programmable parameters without redesigning the hardware. This extends the lifespan of the neural processing unit and provides the effect of enhancing adaptability to continuous technological advancements.

[0055] The effects according to the present disclosure are not limited to those exemplified above, and a wider variety of effects are included within the present disclosure. Brief explanation of the drawing

[0056] FIG. 1 is a block diagram illustrating a computing system for processing a language model that can be applied to the examples of the present disclosure. FIG. 2 is a flowchart illustrating a method in which a neural processing unit applicable to the examples of the present disclosure generates output data from input data using an artificial intelligence model. FIG. 3 is a block diagram showing a neural processing unit that can be applied to the examples of the present disclosure. FIG. 4 is a block diagram illustrating a processing element that can be applied to examples of the present disclosure. FIG. 5 is a block diagram showing a vector core of a special function unit that can be applied to the examples of the present disclosure. FIG. 6 is a block diagram showing a scalar core of a special function unit that can be applied to the examples of the present disclosure. FIG. 7 is a diagram schematically showing the energy consumption per unit operation of a neural processing unit that can be applied to the examples of the present disclosure. FIG. 8 is a flowchart illustrating a method for controlling a neural processing unit that can be applied to examples of the present disclosure. FIG. 9 is a block diagram showing a system-on-chip according to a first embodiment of the first example of the present disclosure. FIG. 10 is a block diagram showing a system-on-chip according to a second embodiment of the first example of the present disclosure. FIG. 11 is a block diagram showing a system-on-chip according to a third embodiment of the first example of the present disclosure. FIG. 12 is a block diagram showing a system-on-chip according to a fourth embodiment of the first example of the present disclosure. FIG. 13 is a block diagram showing a system-on-chip according to the fifth embodiment of the first example of the present disclosure. FIG. 14 is a block diagram showing a system-on-chip according to the sixth embodiment of the first example of the present disclosure. FIG. 15 is a block diagram showing a system-on-chip according to the seventh embodiment of the first example of the present disclosure. FIG. 16 is a block diagram showing a system-on-chip according to the eighth embodiment of the first example of the present disclosure. FIG. 17 is a drawing showing an apparatus according to the first embodiment of the second example of the present disclosure. FIG. 18 is a drawing showing an apparatus according to a second embodiment of a second example of the present disclosure. FIG. 19 is a diagram showing a tree for explaining speculative decoding according to the third example of the present disclosure. FIG. 20 is a diagram schematically illustrating a speculative decoding operation according to a third example of the present disclosure. FIG. 21 is a flowchart illustrating an activation function programming method according to the first embodiment of the fourth example of the present disclosure. FIG. 22 is a graph showing the process of approximating an activation function by an activation function programming method according to the first embodiment of the fourth example of the present disclosure. FIG. 23 is a graph showing various cases of segmenting an activation function into a plurality of segments by an activation function programming method according to the first embodiment of the fourth example of the present disclosure. FIG. 24 is a diagram showing a function approximation circuit of a neural processing unit configured to process a programmed activation function according to the second embodiment of the fourth example of the present disclosure. FIG. 25 is a diagram showing a function approximation circuit of a neural processing unit configured to process a programmed activation function according to the third embodiment of the fourth example of the present disclosure. FIG. 26 is a drawing showing an example of a number system processed in a special function unit of a neural processing unit that can be applied to the examples of the present disclosure. FIG. 27 is a block diagram illustrating a function approximation part according to the fourth embodiment of the fourth example of the present disclosure. FIG. 28 is a block diagram illustrating an inverse function processing circuit according to the fifth embodiment of the fourth example of the present disclosure. FIG. 29 is a drawing illustrating the detailed configuration of an inverse function processing circuit according to the fifth embodiment of the fourth example of the present disclosure. FIG. 30 is a block diagram illustrating a circuit for processing the inverse of a square root function according to the sixth embodiment of the fourth example of the present disclosure. FIG. 31 is a diagram illustrating the detailed configuration of a circuit for processing the inverse of a square root function according to the sixth embodiment of the fourth example of the present disclosure. FIG. 32 is a block diagram illustrating a negative exponential function processing circuit according to the seventh embodiment of the fourth example of the present disclosure. FIG. 33 is a drawing illustrating the detailed configuration of a negative exponential function processing circuit according to the seventh embodiment of the fourth example of the present disclosure. Specific details for implementing the invention

[0057] Specific structural or step descriptions regarding the examples according to the concept of the present disclosure disclosed in this disclosure are provided merely for the purpose of explaining the examples according to the concept of the present disclosure.

[0058] Embodiments according to the concept of the present disclosure may be implemented in various forms. The present disclosure shall not be interpreted as being limited to the embodiments described herein.

[0059] Embodiments according to the concept of the present disclosure may be subject to various modifications. The present disclosure may take various forms. Accordingly, specific examples are illustrated in the drawings and described in detail in the present disclosure. However, this is not intended to limit the examples according to the concept of the present disclosure to specific forms of disclosure. Accordingly, all modifications, equivalents, and substitutions that fall within the spirit and scope of the present disclosure should be understood to be included in the present disclosure.

[0060] Terms such as first and / or second may be used to describe various components. However, the present disclosure should not be limited by such terms.

[0061] The above terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of rights according to the concept of the present disclosure, the first component may be named the second component, and similarly, the second component may be named the first component.

[0062] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. On the other hand, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

[0063] Other expressions describing the relationship between components, such as "between" and "exactly between," or "adjacent to" and "directly adjacent to," should be interpreted in the same way.

[0064] In this document, expressions such as “A or B,” “at least one of A or / and B,” or “one or more of A or / and B” may include all possible combinations of items listed together. For example, “A or B,” “at least one of A and B,” or “at least one of A or B” may refer to cases including (1) at least one A, (2) at least one B, or (3) both at least one A and at least one B.

[0065] Expressions such as "first," "second," "first," or "second" used in this disclosure may modify various components regardless of order and / or importance. Such expressions are used only to distinguish one component from another and do not limit said components. For example, a first user device and a second user device may represent different user devices regardless of order or importance. For example, without departing from the scope of rights set forth in this document, a first component may be named a second component, and similarly, a second component may be renamed a first component.

[0066] The terms used in this disclosure are used merely to describe specific embodiments and are not intended to limit the scope of other examples.

[0067] Singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as generally understood by a person of ordinary knowledge in the technical field described in this document.

[0068] Terms used in this disclosure that are defined in general dictionaries may be interpreted as having the same or similar meaning as they have in the context of the relevant technology. Terms used in this disclosure are not to be interpreted in an ideal or overly formal sense unless explicitly defined in this document. In some cases, even terms defined in this document are not to be interpreted to exclude the embodiments of this document.

[0069] The terms used in this disclosure are used merely to describe specific embodiments and are not intended to limit this disclosure.

[0070] The singular expression includes the plural expression unless the context clearly indicates otherwise. In this disclosure, terms such as “comprising” or “having” are intended to specify the existence of the described features, numbers, steps, actions, components, parts, or combinations thereof. Accordingly, it should be understood that the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof is not excluded in advance.

[0071] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which this disclosure pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology. Unless explicitly defined in this disclosure, they are not to be interpreted in an ideal or overly formal sense.

[0072] The features of each of the various embodiments of the present disclosure may be combined or combined with one another, either partially or wholly. As will be fully understood by those skilled in the art, the various embodiments of the present disclosure are technically capable of various interlocking and operation. Each of the embodiments of the present disclosure may be implemented independently of one another or may be implemented together in an interlocking relationship.

[0073] In describing each example, technical details that are well known in the technical field to which this disclosure belongs and are not directly related to this disclosure are omitted. This is intended to convey the essence of this disclosure more clearly without obscuring it by omitting unnecessary explanations.

[0074] In the present disclosure, 'unit' may refer to a hardware module or a hardware circuit. For example, 'unit' may be a hardware circuit for performing a specific function. However, 'unit' does not necessarily mean a dedicated circuit physically separated from other circuits to perform only that function. 'Unit' may be configured based on multiple circuits, and these circuits may exist at different locations and perform a specific function by being interconnected under a control signal. For example, 'Unit A' and 'Unit B' may include at least some common electronic circuits.

[0075] In this disclosure, the term "artificial intelligence model" may include not only generative AI models but also traditional AI models that perform judgment, classification, prediction, etc. Artificial intelligence models may include language models, models that interpret images or videos, and any model that generates output data from input data using an artificial neural network may qualify as an artificial intelligence model. For example, an artificial intelligence model may generate output text from input text. An artificial intelligence model may classify input data or generate prediction data from input data. An artificial intelligence model may search for or track objects of interest from images or videos. An artificial intelligence model may perform situational judgments by analyzing images or videos. An artificial intelligence model may generate an image / video corresponding to the input text based on the input text. An artificial intelligence model may generate output text describing the input image / video from the input image / video. Input data and output data may each be text, voice signals, images, videos, etc., and are not limited to any specific type.

[0076] An artificial intelligence model can be implemented as an artificial neural network. Therefore, as a representative embodiment, processing or computation regarding an 'artificial intelligence model' refers to processing or computation of an artificial neural network, and the parameters of the artificial intelligence model may include, but are not limited to, the weights of the artificial neural network. For example, it should be understood that processing or computation of an artificial intelligence model may include mathematical operations that are not included in the neural network structure, such as softmax.

[0077] 'Transformer' is an artificial intelligence model implemented based on an attention mechanism and can be used in language models (Large Language Model, LLM and Small LLM). Language models include BERT (Bidirectional Encoder Representations from Transformers), GPT (Generative Pre-trained Transformer), RoBERTa (Robustly Optimized BERT Pretraining Approach), ALBERT (A Lite BERT), ELECTRA (Efficiently Learning an Encoder that Classifies Token Replacements Accurately), Transformer-XL (Transformer with Extra Long Context), and XLNet (which combines the advantages of GPT and BERT). Combined model), BART (Bidirectional and Auto-Regressive Transformers), CTRL (Conditional Transformer Language), T5 (Text-to-Text Transfer Transformer), LaMDA (Language Model for Dialogue Applications), Gopher (DeepMind's LLM), InstructGPT (Fine-tuned model based on GPT-3), PanGu (Huawei's Chinese model), PaLM (Pathways Language Model), OPT 175B(Open Pretrained Transformer 175B), BLOOM(BigScience Large Examples include the Open-science Open-access Multilingual Model and HyperCLOVA (Naver's massive Korean model), and new language models that may emerge in the future may also be included.

[0078] Transformers can be utilized not only in natural language processing (NLP) but also in computer vision, and representative transformer-based image models include ViT (Vision Transformer) and Swin Transformer (Sliding Window-based Transformer for Vision).

[0079] Therefore, the Transformer is one of the core techniques of various artificial intelligence models utilizing attention mechanisms. For example, a Transformer-based language model can output 'yes you are an expert' when 'npu is an ai accelerator' is input.

[0080] According to examples of the present disclosure, devices, methods, systems, and computer-readable media that enable generative artificial intelligence services to operate on the device itself (also referred to as on-device) may be provided.

[0081] Generally, a transformer-based generative AI model receives a query as input and generates a response. For example, a language model receives a query in text form and generates tokens. Subsequently, the query and the generated tokens are input back into the language model to generate the next token. This token generation operation can be repeated multiple times until a response to the query is completed. Tokens may correspond to words or parts of words. The response generated by the language model may include a series of additional tokens generated by passing the tokens generated as part of the response back into the language model.

[0082] Below, several embodiments will be described clearly and in detail with reference to the accompanying drawings so that those skilled in the art to which the present invention pertains (hereinafter, people skilled in the art) can easily practice the present invention.

[0083] First, FIGS. 1 and 2 illustrate the process of inferring a Transformer-based generative artificial intelligence language model on an NPU. The computing system according to the present disclosure operates on an edge device and infers input text data through embedding, attention, feedforward, softmax, etc., to output output tokens. This process consists of numerous matrix multiplications and non-linear operations, and the disclosed NPU enables high-speed inference through the optimization of these operations.

[0084] FIG. 1 is a block diagram illustrating a computing system for processing a language model that can be applied to the examples of the present disclosure.

[0085] Referring to FIG. 1, a computing system (1) can receive input data and generate output data. The computing system (1) may include a Neural Processing Unit (NPU) (100). Additionally, the computing system (1) may include additional computational units such as a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and memory in addition to the Neural Processing Unit (100), and embedding transformation for the input text may be performed by the Central Processing Unit or the Graphics Processing Unit, etc. The transformed embedding vector is transmitted to the Neural Processing Unit (100) and may be utilized by the Neural Processing Unit (100) for subsequent inference operations. The Neural Processing Unit (100) may perform inference operations for a Transformer-based generative artificial intelligence model. The Neural Processing Unit (100) may be referred to as an artificial intelligence accelerator, an artificial intelligence hardware accelerator, etc., but is not limited thereto. Here, the computing system (1) may be an example of a device (10000) and may include a smartphone, a robot, an ADAS system, etc.

[0086] The neural processing unit (100) can generate output data from input data. According to one embodiment, the input data may be text data. The output data may be text data or image / video data. Text data consists of at least one token. For example, when text containing five tokens 'npu', 'is', 'an', 'AI', and 'accelerator' is input to the NPU, the neural processing unit (100) can perform a transformer-based artificial neural network operation to output text containing six tokens 'yes', 'i', 'am', 'the', 'AI', and 'accelerator'.

[0087] Transformer-based artificial neural network computation can be performed on-device. That is, the computing system (1) can be included and operated on an edge device such as a smartphone, camera, CCTV, robot, or drone, and does not require wired or wireless communication with the outside of the computing system (1) for artificial neural network computation.

[0088] Transformer-based artificial neural network operations can be performed based on an attention mechanism. Under the attention mechanism, input data in text form is converted into embedding vectors through an embedding process, and these embedding vectors can be transformed into Query, Key, and Value vectors using the vector-matrix multiplication (MatMul) with the Query weight matrix, Key weight matrix, and Value weight matrix. The Query weight matrix, Key weight matrix, and Value weight matrix are each pre-trained values. Subsequently, the similarity between the Query (Q) and the Key (K) is calculated, and an attention score can be obtained by assigning a value (V) to the calculated similarity.

[0089] FIG. 2 is a flowchart illustrating a method in which a neural processing unit applicable to the examples of the present disclosure generates output data from input data using an artificial intelligence model.

[0090] Referring to FIG. 2, the neural processing unit (100) can obtain an attention score for input data (S11).

[0091] The input data may include multiple tokens, and each token has a predetermined dimension (d model It can be expressed as an embedding vector of ). The number of dimensions of the embedding vector (d model ) can be, for example, 128, 256, 1024, 2048, or 4096, etc. For convenience of explanation below, d model The explanation assumes 4096. In this embodiment, one token in the computing system (1) can be represented as an embedding vector of 1 x 4096 dimensions.

[0092] For example, if a user inputs “npu is an AI accelerator” consisting of 5 tokens into the computing system (1), the computing system (1) can convert the input text into an embedding vector of size 5 x 4096 (i.e., 5 x 1 x 4096), and the neural processing unit (100) can receive the embedding vector. The neural processing unit (100) can obtain a query vector Q by performing a vector-matrix product of the query weight matrix on the embedding vector. The neural processing unit (100) can obtain a key vector K by performing a vector-matrix product of the key weight matrix on the embedding vector. Then, the neural processing unit (100) can obtain similarity through the inner product between the query vector Q and the key vector K. When the query vector Q and the key vector K are multiplied by an inner product, a scalar value is obtained, and consequently, a vector QK of the same size (1 x 5) as the number of tokens in the input data is obtained. T is obtained. QK Trepresents the similarity between the current token and key k (each of the 5 tokens). However, the vector QK of the obtained size T Since the elements of can be numerically very large, QK for numerical stability T The square root of the magnitude of the key vector K ( A normalized final similarity can be obtained by dividing by ) and then applying a softmax operation. <Mathematical Formula 1> below is a mathematical representation of the softmax-based similarity calculation. k can be one of 1, 2, ..., N (N is the number of tokens included in the input data, in this embodiment N = 5).

[0093]

[0094] The softmax operation is an operation that converts multiple numbers into probabilities between 0 and 1. <Mathematical Formula 2> below represents the vector [z1, z2, ... , z n Represents a formula that performs a softmax operation value for ].

[0095]

[0096] However, z i If there is a value that is too large, numerical instability may occur. To resolve this, the maximum value z max By using the negative exponential trick, numerically more stable calculations can be made possible. <Mathematical Equation 3> below substitutes z from <Mathematical Equation 2> max This is a softmax calculation formula modified using .

[0097]

[0098] Through <Equation 1>, similarity can be obtained as a 1x5 vector of size [0.007, 0.993, 0, 0, 0], for example. Similarity is a value in which the similarity between the current query (e.g., 'npu') and other tokens is expressed as a probability.

[0099] According to one embodiment, the neural processing unit (100) can assign a value to the similarity. The value vector is similar to the method used to calculate the key vector. The embedding vector can be converted into a value vector through a vector-matrix multiplication with a pre-learned value weight matrix. An attention score can be obtained by performing a vector-matrix multiplication of the value vector V with the similarity obtained through <Equation 1>. Consequently, an attention score of size 5X 4096, which is the same dimension as the input data, can be obtained by performing an operation such as <Equation 4> below.

[0100]

[0101] That is, the neural processing unit (100) repeatedly performs the attention operation of <Equation 4> to generate output data. In this process, it can be seen that the neural processing unit (100) repeatedly performs high-dimensional calculations including operations such as vector-matrix multiplication, division, and multiplication. According to one embodiment, the neural processing unit (100) may additionally perform residual linking and layer normalization. Residual linking and layer normalization are intended to compensate for the possibility of gradient vanishing that may occur in the differentiation-based learning method of deep learning, and can minimize information loss by adding input values ​​to the output values ​​that have passed through the layers of the artificial neural network. After the attention score is calculated for residual linking, the embedding vector of the input data can be added to the attention score. Layer normalization can mitigate covariant shift by adjusting values ​​based on the mean and variance of the output value (i.e., the result of residual linking) in natural language processing.

[0102] Next, the neural processing unit (100) can process the attention score through a feed-forward neural network (S12).

[0103] A feed-forward neural network is designed to predict non-linear phenomena and enables the inference of contextual meanings, such as phrases or words, by recombining relational information of tokens for which attention scores have been calculated. As an example, a feed-forward neural network may use a fully connected hidden layer with two layers having 2,048 nodes, and the output may be a 4,096-dimensional vector identical to the input. The feed-forward neural network is pre-trained, and vector-matrix multiplication operations may be repeatedly performed for inference. Here, the input is represented as a 4,096-dimensional vector as the result of residual connection and normalization in step S11, and the residual connection and normalization of step S11 may also be additionally performed on the output of the hidden layer composed of two layers.

[0104] Next, the neural processing unit (100) can generate a final output token (S13).

[0105] To this end, the output vector generated in step S12 must be expanded to the total number of tokens in the set that the language model can process (e.g., 30,000). For example, the output vector from step S12 can be expanded into a logit vector of size 1 x 30,000 by performing a matrix multiplication operation with the final output weight matrix (e.g., 4,096 x 30,000). The size of the final output weight matrix for the above final output is d. modelAs X SIZE_DIC, SIZE_DIC represents the total number of tokens that can be output from a language model undergoing inference operations in the neural processing unit (100) (30,000 in this embodiment). A softmax operation is applied to the calculated logit vector of size 1 X 30,000 so that all 30,000 tokens are normalized to probability values ​​between 0 and 1, and the token having the highest probability value among the 30,000 probability values ​​can be selected as the next output token to be generated. For example, if the input to the neural processing unit (100) is 'npu is an AI accelerator', the output token is determined to be 'yes', and if it is 'npu is an AI accelerator yes you are an', the output token can be determined to be 'expert'.

[0106] As described above with reference to FIGS. 1 and 2, the neural processing unit repeatedly performs numerous matrix multiplication operations (MatMul), vector operations, scalar operations, etc., for the inference operations of a transformer-based artificial neural network model, and the neural processing unit needs to be optimized for these operations. Below, the structure and operation of the neural processing unit (100) will be described.

[0107] FIG. 3 is a block diagram showing a neural processing unit that can be applied to the examples of the present disclosure.

[0108] Hereinafter, the neural processing unit (100) will be described in detail with reference to FIG. 3. The neural processing unit (100) is a dedicated processor specialized for Deep Neural Network (DNN) computations, separate from processors such as a central processing unit or a graphics processing unit. The neural processing unit (100) may be referred to as a Tensor Processing Unit (TPU), an AI accelerator, a Neural Signal Processor (NSP), a Neural Network Processor (NNP), etc.

[0109] The neural processing unit (100) may be a semiconductor implemented as an electrical / electronic circuit. That is, each component of the neural processing unit (100) may be a semiconductor circuit composed of numerous electronic elements (e.g., transistors, capacitors, electrical wiring, etc.). Therefore, some of these may be difficult to identify and distinguish with the naked eye and may be identified only by operation.

[0110] This neural processing unit (100) is intended to accelerate artificial neural network computations and is also called an artificial intelligence hardware accelerator. The artificial neural network can output computations of Transformer-based language models. However, it is not limited to this and can also process vision-based models such as YOLO or ResNet. It is mounted on a device (10000) and can output various information based on artificial intelligence. In addition to the neural processing unit (100), the device (10000) may include a central processing unit, a graphics processing unit, an application processor (AP), a microcontroller unit (MCU), etc. The device (10000) may include a microphone, a camera, a touchpad, a keyboard, etc. to receive input data, and may include a monitor, a speaker, etc. to output output data. The device (10000) may include, but is not limited to, a smartphone, a virtual assistant terminal located in a home or office, home appliances, a VCU or ADAS system of a car, a camera such as a CCTV, a robot, smart mobility, etc.

[0111] The neural processing unit (100) according to the examples of the present disclosure is specialized in accelerating the processing of an attention algorithm of a transformer-based neural network model at low power, and this will be explained in detail later. However, the neural processing unit (100) of the present disclosure is not limited to a transformer neural network and can process various neural network models such as a Convolutional Neural Network (CNN).

[0112] A neural processing unit (100) applicable to the examples of the present disclosure may include a controller (CTL) (110), a processing core (P_CORE) (120), a direct memory access unit (DMA) (130), an internal memory (I_MEM) (140), and a special function unit (SFU) (150).

[0113] The processing core (120), vector core (V_CORE) (153), and scalar core (S_CORE) (154) of the neural processing unit (100) may be computational circuits configured to efficiently process the attention layer of the neural network model. The attention layer can obtain an attention score (Q, K, V), which is a computational value, using parameters such as input values ​​and a query (Q), a key (K), and a value (V).

[0114] The processing core (120), vector core (153), and scalar core (154) are designed to process operations such as matrix multiplication, root mean square normalization, and softmax for attention operations. The tensor data form required for the above operations may be one of a multidimensional matrix form, a two-dimensional matrix form, a one-dimensional array form, or a scalar value, and one of the processing core (120), vector core (153), and scalar core (154) may be selected for the operation depending on the data form. Accordingly, the neural processing unit (100) can rapidly accelerate the operation of a generative artificial intelligence model with low power consumption by appropriately adopting a hardware module appropriate for the operation.

[0115] That is, the processing core (120) is capable of processing mathematical operations required for artificial neural network inference operations and may include an Arithmetic Logic Unit (ALU), a Multiply-Accumulate Unit (MAC), an Adder Tree, etc.

[0116] The controller (110) may be electrically connected to the processing core (120), the direct memory access unit (130), the internal memory (140), and the special function unit (150). The controller (110) may be configured to control operations related to neural network operations of the neural processing unit (100). The controller (110) may control each circuit included in the neural processing unit (100) to process inference operations of the neural network model by executable code generated by a compiler (not shown). Here, the compiler (not shown) may be a semiconductor circuit or a software module operating in a separate device from the neural processing unit (100).

[0117] A compiler (not shown) can obtain executable code that can be executed on a neural processing unit (100) by compiling a specific neural network model. That is, the compiler (not shown) can generate executable code that can be executed exclusively on a specific neural processing unit having specific performance. The executable code may also be referred to as machine code or binary code.

[0118] The controller (110) can control the processing core (120), direct memory access unit (130), internal memory (140), and special function unit (150), etc., based on various instructions and operation schedules included in the execution code. The compiler (not shown) can be provided with hardware characteristics of the neural processing unit (100) (e.g., the computational performance of the processing core (120), the computational performance of the vector core (153), the computational performance of the scalar core (154), the processing performance of the number system conversion unit (quantization unit (QUANT) (151), dequantization unit (DEQUANT) (152)), the capacity of the internal memory (140), and the main memory (not shown) bandwidth of the corresponding bus (not shown), etc.). In addition, the compiler (not shown) can be provided with structural information of the neural network model to be processed by the neural processing unit (100), information of algorithms included in the neural network model, and information on the size of parameters, etc. Then, the compiler (not shown) can generate execution code to control the read / write order of data required for neural network operations, the processing order of neural network operations, and whether each component of the neural processing unit (100) operates. The controller (110) receives the execution code from the compiler and can control the neural processing unit (100) based on the execution code. Each execution code can be generated corresponding to each neural network model, and the neural processing unit (100) can be configured to process at least one execution code.

[0119] The processing core (120) may include electronic circuits specialized for matrix multiplication, which accounts for the largest amount of computation among neural network computations. For example, the processing core (120) may be composed of multiple processing elements. For example, the processing core (120) may be designed as an array structure of M x N processing elements (where M and N are integers) or an adder tree structure, but is not limited thereto. The processing core (120) processes data in parallel based on multiple processing elements (PE) (not shown), and parameters (corresponding to input data) e.g. It can be configured to process matrix multiplication operations of weights, kernels, queries (Q), keys (K), values ​​(V), etc.

[0120] To this end, processing elements (not shown) may each include a MAC (Multiply and Accumulate) operator. Thus, the processing core (120) can efficiently process matrix multiplication or convolution operations. To elaborate, the processing core (120) may be a circuit designed to simultaneously process tensor data of size M x N. Here, the number of processing cores (120) may be at least one. The processing core (120) may exchange data with the vector core (153), scalar core (154), internal memory (140), and main memory (not shown) of the special function unit (150). The processing core (120) may be configured to receive integer parameters as input and output integer parameters. The input integer parameters may include input parameters and weights. The input parameters may be configured to have a first bit width, and the weight parameters may be configured to have a second bit width. The output integer parameter may be configured to have a third bit width, and the third bit width may be configured to be larger than the first bit width or the second bit width.

[0121] As the number of processing elements (not shown) included in the processing core (120) increases, the number of transistors in the ASIC (Application Specific Integrated Circuit) increases, and the manufacturing cost may increase. If the circuit of the processing core (120) is designed to process integer parameters rather than floating-point numbers, the number of transistors in the ASIC can be significantly reduced, and the power consumption of the neural processing unit (100) can be significantly reduced.

[0122] The bit width of integer parameters that can be processed by the processing core (120) can be determined based on the hardware design of the processing element (not shown). This is described later in FIG. 4. The number of processing elements (not shown) may range from hundreds to tens of thousands. To elaborate, the number of transistors included in a processing element designed to process floating-point parameters is about four times greater than the number of transistors included in a processing element designed to process integer parameters. Therefore, if the circuit of the processing core (120) of the neural processing unit (100) according to the examples of the present disclosure is designed to exclude floating-point parameter operations and only process integer parameters, the number of transistors can be reduced by about 75% compared to the floating-point operation circuit. In this embodiment, the number of processing elements (not shown) included in the processing core (120) can be easily increased in the same chip area compared to floating-point, and there is an effect of improving computation processing performance with low power while reducing the size of the semiconductor chip.

[0123] The special function unit (150) may refer to a collection of various circuit sections configured to process various neural network operations that are not processed by the processing core (120). That is, it is a circuit capable of accelerating special operations that are difficult to perform in the ALU. The special function unit (150) can perform operations such as exponentiation, logarithmic functions (log, ln), square roots (sqrt), reciprocals and inverse square roots (reciprocal, 1 / sqrt(x)), trigonometric functions (Sin, Cos, Tan, ArcSin, ArcCos), and quantization (INT8 / FP16 conversion). For example, the special function unit (150) may include a quantization section (151), an inverse quantization section (152), a vector core (153), and a scalar core (154). The quantization section (151) and the inverse quantization section (152) may be collectively referred to as a number system conversion section, but are not limited thereto.

[0124] The quantization unit (151) of the special function unit (150) can convert floating point (FP) data into integer (INT) data, and the inverse quantization unit (152) can inversely convert integer data into floating point data.

[0125] The vector core (153) of the special function unit (150) may refer to a set of circuits that process at least some of the operations of the artificial intelligence model, excluding matrix multiplication processed by the processing core (120). The vector core (153) may be designed to include a plurality of specialized vector operation circuits. For example, the vector core (153) may include a vector maximum value operation unit (V_MAX) (1531), a vector multiplication unit (V_MUL) (1532), a vector addition unit (V_ADD) (1533), a MAC operation unit (V_MAC) (1534), a mask unit (V_MASK) (1535), a rounding clipping unit (RND_CLIP) (1536), and a vector function approximation unit (V_FunApx) (1537). The operation of each module of the vector core (153) will be described later with reference to FIG. 5.

[0126] The scalar core (154) of the special function unit (150) may refer to a set of circuits that process at least some of the remaining operations among the operations of the artificial intelligence model, excluding those processed by the processing core (120) and the vector core (153). The scalar core (154) may be designed to include a plurality of specialized scalar operation circuits. For example, the scalar core (154) may include a scalar maximum value operation unit (S_MAX) (1541), a scalar multiplication unit (S_MUL) (1542), a scalar addition unit (S_ADD) (1543), a scalar summation unit (S_SUM) (1544), and a scalar function approximation unit (S_FunApx) (1545). The operation of each of the modules of the scalar core (154) will be described later with reference to FIG. 6.

[0127] The direct memory access unit (130) can access at least one memory electrically connected to the neural processing unit (100) to perform data read / write operations. The at least one memory may include at least one of a main memory (not shown), a shared memory (not shown), a storage device (not shown), etc. The neural processing unit (100) can transmit various data related to artificial intelligence operations to at least one memory (not shown) or read data from at least one memory (not shown) through the direct memory access unit (130). The direct memory access unit (130) may be configured to perform operations such as setting the address of the internal memory (140), generating read / write commands, and controlling. The direct memory access unit (140) may be configured to control at least one memory by being connected to at least one communication bus.

[0128] Internal memory (140) is memory placed in the on-chip area of ​​the neural processing unit (100). Internal memory (140) may include memory for caching or storing data processed in the on-chip area and / or a register file for storing instructions for running a compiled neural network model. The register file may also be configured to be included in the controller (110). For example, when a memory (not shown) connected to a bus is provided, internal memory (140) may read and store data required for computation from said memory. Internal memory (140) may include SRAM, a register file, a DRAM, a resistive RAM, a magneto-resistive RAM, a phase-change RAM, a ferroelectric RAM, or a flash memory. Internal memory (140) may be composed of at least one memory unit. Internal memory (140) may be composed of a homogeneous memory unit or a heterogeneous memory unit. The capacity of the internal memory (140) may be several megabytes to tens of megabytes, but is not limited thereto.

[0129] The quantization unit (151) of the special function unit (150) may be configured to include a quantization circuit. The inverse quantization unit (152) may be configured to include an inverse quantization circuit. The quantization unit (151) and the inverse quantization unit (152) may be referred to as a number system conversion unit.

[0130] According to one embodiment, the input and output of the processing core (120) may be in the form of integers. The input and output of the vector core (153) and the scalar core (154) may be in the form of floating-point numbers. The quantization unit (151) and the dequantization unit (152) can convert data into the data form (number system) required by each hardware during data transmission between the processing core (120), the vector core (153), and the scalar core (154). That is, the quantization circuit of the quantization unit (151) can convert floating-point parameters into integer parameters. The dequantization circuit of the dequantization unit (152) can convert integer parameters into floating-point parameters.

[0131] According to the examples of the present disclosure, floating-point numbers are not limited to IEEE standard 754, and Brain Floating Point, Dynamic Floating-Point, Variable Precision Floating-Point (VPFP), Custom Floating-Point (CFP), or Flexible Floating-Point (FFP) may be applied to improve computational efficiency, reduce memory usage, reduce power consumption, etc. of the neural processing unit (100). VPFP is a floating-point format that allows the bit width of the exponent and mantissa to be dynamically set or adjusted. Using VPFP allows for the optimization of power and performance by allowing flexible precision levels based on specific calculations. Using a dynamic floating-point format, which is used interchangeably with variable precision, allows for the immediate adjustment of mantissa and exponent sizes during runtime, enabling real-time response to data characteristics. CFP represents an application-specific format where the bit allocation for the mantissa and exponent is adjusted to meet the requirements of a specific use case. This may be useful for neural processing unit (100) implementations that select an accurate bit width to optimize hardware resources and data accuracy. FFP refers to a floating-point format that allows for adjustable bit allocation between the exponent and mantissa. FFP may be implemented in a neural processing unit (100) to provide an optimal balance between range and precision. The bit width of the floating-point number according to the examples of the present disclosure may be between 4 bits and 32 bits. According to the examples of the present disclosure, the bit width of the integer may be between 4 bits and 32 bits.

[0132] The neural processing unit (100) disclosed in the present disclosure may provide at least one of power gating, clock gating, and register retention functions that enable the neural processing unit (100) to operate at low power.

[0133] Power gating is a technology that eliminates leakage power by completely cutting off power to unused circuit blocks. The circuit blocks to which power gating is applied are provided with power switches. Accordingly, specific circuit blocks of the neural processing unit (100) can be disconnected from the power supply using the power switches. Power gating can eliminate both dynamic power and leakage power, thereby enabling the device to operate in a low-power mode.

[0134] Clock gating is a technology that reduces dynamic power consumption by blocking clock signals from unused circuit blocks. Circuit blocks to which clock gating is applied can selectively block clock signals input to specific blocks using clock control logic. When clock gating is applied, power is continuously supplied to the Neural Processing Unit (NPU), allowing the circuit's state to be maintained. Since clock gating does not turn off the power, operation can resume immediately upon reactivating the clock, thereby reducing dynamic power consumption by preventing unnecessary switching activities.

[0135] Register retention is a technology designed to allow a specific register of the neural processing unit (100) to maintain its state even when the power is turned off. Registers to which register retention is applied can retain data stored in the register using only low power in sleep mode. Therefore, even if the register is switched to power-saving mode, important register values ​​are retained, so initialization is not required upon return. Register retention can provide low-power operation and can retain data using extremely little power even when the power of the neural processing unit (100) is turned off.

[0136] The hardware operation between each component of the neural processing unit (100) of FIG. 3 will be explained in detail below based on FIG. 8.

[0137] FIG. 4 is a block diagram illustrating a processing element that can be applied to examples of the present disclosure.

[0138] Referring to FIG. 4, the processing element (121) may be configured to include a multiplier (1211), an adder (1212), and an accumulator (1213). The processing element (121) of FIG. 4 may be one of a plurality of processing elements included in the processing core (120) of FIG. 3, and the processing element (121) may refer to each processing element in the following description.

[0139] The multiplier (1211) multiplies the input (N)-bit data and (M)-bit data. The operation value of the multiplier (1211) is output as (N+M)-bit data. Here, N and M are integers greater than 0. The first input unit may be configured to receive (N)-bit data. The second input unit may be configured to receive (M)-bit data. For example, the first input unit may be configured to receive input parameters and the second input unit may be configured to receive weight parameters.

[0140] The bit width of the parameters input to the first input and second input sections of the processing element (121) can be determined when the neural processing unit (100) compiles the current neural network model being processed to obtain executable code. That is, the bit width of the input parameters and the bit width of the weight parameters of the neural network model can be determined at the compilation stage by the compiler. For example, the bit widths of the input parameters and the weight parameters can be quantized equally, and the bit widths can be 32 bits, 16 bits, 8 bits, 4 bits, etc., but the present disclosure is not limited thereto. For example, the bit width of the input parameters and the bit width of the weight parameters can be determined differently from each other, and the parameters can be quantized based on their respective bit widths. For example, the bit width of the input parameters and the bit width of the weight parameters can be quantized to 16 bits and 8 bits, respectively. For example, the bit width of the input parameters and the bit width of the weight parameters can be quantized to 8 bits and 4 bits, respectively. That is, the bit widths of the parameters input to each input of the processing element (121) may differ from one another. The processing element (121) may include quantization information of the data input to each input in the execution code.

[0141] The accumulator (1213) uses the adder (1212) for (L) loops to accumulate the operation value of the multiplier (1211) and the operation value of the accumulator (1213). Accordingly, the bit width of the data in the output and input sections of the accumulator (1213) can be output as (N+M+log2(L)) bits. Here, L is an integer greater than 0. When accumulation is finished, an initialization signal (INTIALIZATION RESET) is input so that the data stored inside the accumulator (1213) can be initialized to 0. The accumulator (1213) of the processing element (121) is configured to maintain the accumulated value when zero skipping is enabled. Based on the maximum value that can be accumulated in the accumulator (1213), the output data (X) bit can be set to a bit width such that no overflow of the output data (X) bit occurs. For example, (X) bit can be 16 bits to 64 bits.

[0142] To elaborate, the quantization unit (151) of the special function unit (150) can convert an integer parameter output from the processing core (120) into a floating-point number and transmit it to the vector core (153), scalar core (154), and / or internal memory (140). The quantization unit (151) can be controlled under a control signal from the controller (110).

[0143] The controller (110) can restrict the operation of the multiplier (1211) so that it does not perform operations (e.g., zero-skipping operation) based on the fact that when 0 is input to one of the first input and second input sections of the multiplier (1211), the result of the operation becomes 0 even without performing the operation. For example, when 0 is input to one of the first input and second input sections of the multiplier (1211) of the processing element (121), the multiplier (1211) can operate in a zero-skipping manner. For zero-skipping, each processing element (121) included in the processing core (120) can be enabled or disabled. The controller (110) can provide an enable or disable signal (CLOCK-GATING CONTROL) to each processing element (121) on a clock basis. When the processing element (121) is disabled, the multiplier (1211) is disabled. Accordingly, the power consumed in the operation of the multiplier (1211) can be reduced. An example of the power consumption of the multiplier (1211) can be shown in FIG. 7.

[0144] A processing element (121) may be designed to receive a control signal (CLOCK-GATING CONTROL) from a controller (110) for zero-skipping operation control (i.e., activation or deactivation). Specifically, a multiplier (1211) of the processing element (121) may be designed to receive a respective control signal (CLOCK-GATING CONTROL) for zero-skipping operation control from the controller (110). Alternatively, an adder (1212) of the processing element (121) may be modified to receive a control signal (CLOCK-GATING CONTROL) for zero-skipping operation control from the controller (110). Alternatively, each multiplier (1211) and adder (1212) of the processing element (121) may be modified to receive a respective control signal (CLOCK-GATING CONTROL) for zero-skipping operation control simultaneously from the controller (110).

[0145] The controller (110) can optimize the operation path by pre-reflecting and integrating a multiplication operation (FpMul) based on the normalization coefficients included in the weights of the neural network model at compile time before the softmax operation is performed, and omitting the multiplication operation at runtime. This corresponds to a static optimization method for reducing computational efficiency and power consumption by eliminating unnecessary runtime multiplication operations.

[0146] Additionally, the controller (110) may selectively control whether to perform a multiplication operation between a query vector and a key vector depending on the type of operation request (e.g., attention operation, feedforward operation, etc.) or the timing of the operation (e.g., decoding phase, learning phase, etc.). This configuration may be implemented by controlling the clock supply of a processing element (121) including a multiplier (1211) or by bypassing the operation path itself to skip the operation.

[0147] Such control may be processed within the neural processing unit (100) or may be performed through an integrated controller in a system-on-chip (SoC) architecture that includes multiple computation units. In this case, the controller (110) may be configured to dynamically control whether computation is enabled for each computation unit by including clock gating or power control signals for each unit.

[0148] FIG. 5 is a block diagram showing a vector core of a special function unit that can be applied to the examples of the present disclosure.

[0149] The vector core (153) may refer to a set of circuits that process some of the operations of the artificial intelligence model, excluding matrix multiplication, which is processed by the processing core (120). The vector core (153) differs from the processing core (120) in that it is composed of floating-point arithmetic circuits. That is, the vector core (153) can be configured to receive floating-point parameters as input and output floating-point parameters. The vector core (153) can be designed to be pipelined with the processing core (120) and the scalar core (154) for computational efficiency with the processing core (120) and the scalar core (154). To elaborate, the vector core (153) is configured to exchange data with the processing core (120), the scalar core (154), the internal memory (140), and the main memory (not shown). To elaborate, the vector core (153) can be specialized in efficiently processing some of the attention operations of the transformer-based neural network model. That is, the vector core (153) may include a plurality of specialized vector operation circuits.

[0150] For example, the vector core (153) may be designed to include a vector maximum value operation unit (V_MAX) (1531), a vector multiplication unit (V_MUL) (1532), a vector addition unit (V_ADD) (1533), a MAC operation unit (V_MAC) (1534), a mask unit (V_MASK) (1535), a rounding clipping unit (RND_CLIP) (1536), and a vector function approximation unit (V_FuncApx) (1537). The vector maximum value operation unit (1531) is configured to include a circuit that processes a maximum value operation of the input tensor data. The vector multiplication unit (1532) is configured to include a circuit that processes a multiplication operation of the input tensor data. The vector addition unit (1533) is configured to include a circuit that processes an addition operation of the input tensor data. The MAC operation unit (1534) is configured to include a circuit that processes Multiply And Accumulate (MAC) operations on the input tensor data. The mask unit (1535) is configured to include a circuit that processes masking operations on the input tensor data. The rounding clipping unit (1536) is configured to include a circuit that processes rounding and clipping operations on the input tensor data. The vector function approximation unit (1537) is configured to include a circuit that processes approximation operations of various functions and can be applied to exponential function operations or vector-unit weighted sum operations performed during softmax operations. In particular, the vector core (153) can be configured to generate an input vector for softmax operations by multiplying the inner product result between a query vector and a key vector by a normalization factor, create a stabilized softmax distribution by subtracting the maximum value from each element of the generated input vector, and output a final attention result by multiplying the softmax operation result by a value vector. Here, the normalization factor may be a constant value calculated at compile time and stored in memory.This operation flow can be implemented around circuits such as a vector maximum value operation unit (1531), a vector multiplication unit (1532), a vector addition unit (1533), and a vector function approximation unit (1537), and can be configured in a pipelined and parallel processing manner to consider operation efficiency. If necessary, it can be further configured to include an auxiliary operation flow before and after the softmax operation by including a MAC operation unit (1534) for Multiply-And-Accumulate operation, a mask unit (1535) for masking processing, or a rounding clipping unit (1536) for output precision correction.

[0151] Below, the vector function approximation part (1537) will be described in more detail.

[0152] For example, the vector core (153) may be a circuit designed to simultaneously process N x 1 tensor data. The vector core (153) may be designed to receive N x 1 tensor data as input and output N x 1 tensor data. Here, M of the N x 1 tensor data received by the vector core (153) may be equal to M, which is the number of rows of M x N tensor data received by the processing core (120). That is, a pipeline circuit design can be implemented by matching the size of a specific dimension of the tensor data of the processing core (120) and the vector core (153) respectively. To elaborate, the vector core (153) may include a register file for storing one-dimensional array data. Therefore, the vector core (153) can improve computational processing efficiency by processing the tensor data output from the processing core (120) in units of a specific dimension. As described above, the vector core (153) is configured to process various function operations using a floating-point arithmetic unit. Therefore, if the vector core (153) is designed to process the same tensor size as the processing core (120), the semiconductor chip size and power consumption of the neural processing unit (100) may increase. However, since the vector core (153) is designed to process data in the form of a one-dimensional array, the increase in the semiconductor chip size of the neural processing unit (100) can be reduced, power consumption can be reduced, and the bottleneck of neural network computation can be reduced by being pipelined with the processing core (120). In some examples, the vector core (153) may be designed to receive tensor data of size L x 1 as input. Here, L may be a power of 2 value within a specific range based on N. For example, when N is 64, L may be 8, 16, 32, 64, 128, or 256.

[0153] FIG. 6 is a block diagram showing a scalar core of a special function unit that can be applied to the examples of the present disclosure.

[0154] The scalar core (154) may refer to a set of circuits that process some of the operations of the neural network model, excluding the operations processed by the processing core (120) and the vector core (153). The scalar core (154) differs from the processing core (120) in that it is composed of floating-point arithmetic circuits. That is, the scalar core (154) may be configured to receive floating-point parameters as input and output floating-point parameters. The scalar core (154) may be designed to be pipelined with the processing core (120) and the vector core (154) for computational efficiency with the processing core (120) and the vector core (153). To elaborate, the scalar core (154) is configured to exchange data with the processing core (120), the vector core (153), the internal memory (140), and the main memory (not shown). To elaborate, the scalar core (154) can be specialized to efficiently process part of the attention operation of a transformer-based neural network model. That is, the scalar core (154) may include a plurality of specialized scalar operation circuits.

[0155] For example, the scalar core (154) may be designed to include a scalar maximum value operation unit (1541), a scalar multiplication unit (1542), a scalar addition unit (S_ADD) (1543), a scalar summation unit (S_SUM) (1544), and a scalar function approximation unit (S_FuncApx) (1545). The scalar maximum value operation unit (1541) is configured to include a circuit that processes a maximum value operation of input tensor data. The scalar multiplication unit (1542) is configured to include a circuit that processes a multiplication operation of input tensor data. The scalar addition unit (1543) is configured to include a circuit that processes an addition operation of input tensor data. The scalar summation unit (1544) is configured to include a circuit that accumulates a plurality of input scalar values ​​or performs an aggregation operation according to a certain range or condition. The scalar function approximation unit (1545) is configured to include circuits that process approximation operations of various functions. These function approximation operations can be applied to the calculation of correction coefficients, such as the approximation of exponential functions used in softmax operations and the reciprocal operation of the total sum. In other words, the scalar core (154) can be configured to calculate the maximum value among the results of the product operation between the normalized query vector and the key vector, calculate the total sum of the exponential function, and then calculate the reciprocal to calculate the correction coefficient for the softmax operation.

[0156] Below, the scalar function approximation part (1545) will be described in more detail.

[0157] For example, the scalar core (154) may be a circuit designed to simultaneously process tensor data of size 1 x N. The scalar core (154) may be designed to receive tensor data of size 1 x N as input and output tensor data of size 1 x 1. The scalar core (154) may be designed to receive tensor data of size 1 x N as input and output scalar data of size 1 x 1. Here, the 1 x N size of the tensor data received by the scalar core (154) may be the same as the M x 1 size received by the vector core (153). That is, a pipeline circuit design can be implemented by matching the size of a specific dimension of the tensor data of the vector core (153) and the scalar core (154) respectively. To elaborate, the scalar core (154) may include a register file for storing array data in a one-dimensional form. Accordingly, the scalar core (154) can improve computational processing efficiency by processing tensor data output from the vector core (153) in units of specific dimensions. As described above, the scalar core (154) is configured to process various function operations as a floating-point arithmetic unit. Therefore, if the scalar core (154) is designed to process the same tensor size as the processing core (120), the semiconductor chip size and power consumption of the neural processing unit (100) may increase. However, since the scalar core (154) is designed to process data in a scalar form, the increase in the semiconductor chip size of the neural processing unit (100) can be reduced, power consumption can be reduced, and bottlenecks in neural network operations can be reduced by being pipelined with the vector core (153). In some examples, the scalar core (154) may be designed to receive tensor data of size L x 1. Here, L can be a power of 2 value within a specific range based on N. For example, when N is 64, L can be 8, 16, 32, 64, 128, 256.

[0158] In some embodiments, the vector core (153) shown in FIG. 5 and the scalar core (154) shown in FIG. 6 may be transformed into a single integrated 'vector-scalar operation unit' with increased integration density by sharing functionally overlapping circuits. This integrated architecture can maximize the efficiency of hardware resources by sharing operation circuits of similar functions, going beyond simply physically placing the two units adjacent to each other.

[0159] Referring again to FIG. 5, the vector core (153) is designed to perform parallel operations on each element of a vector composed of multiple data elements (e.g., N). This can be conceptually understood as a structure in which N independent scalar operation lanes are arranged in parallel. On the other hand, the scalar core (154) of FIG. 6 processes a single data (1x1 scalar). Therefore, based on the structural similarity, the present embodiment may be configured to perform scalar operations by selectively activating only some lanes of the vector operation circuit according to a control signal of the control unit (110).

[0160] According to the configuration described above, the integrated vector-scalar operation unit can be implemented as an optional resource sharing architecture in which circuits are shared so that existing vector operators replace scalar operators with redundant functions, and only scalar operations with specialized functions, such as reduction, are maintained separately. This allows for a significant improvement in power-to-performance ratio (PPA) by providing all functions identically while reducing the total number of gates and chip area of ​​the special function unit (SFU, 150).

[0161] FIG. 7 is a diagram schematically showing the energy consumption per unit operation of a neural processing unit that can be applied to the examples of the present disclosure.

[0162] The power consumption of the processing core (120), vector core (153), and scalar core (154) of the neural processing unit (100) will be explained below with reference to FIG. 7.

[0163] FIG. 7 schematically illustrates the energy consumed per unit operation of various arithmetic units of a neural processing unit (100). For example, energy consumption can be described by dividing it into addition operations and multiplication operations. However, the energy consumed per unit operation may vary depending on the semiconductor chip's foundry process technology (e.g., 2nm technology, 5nm technology, 7nm technology, 14nm technology, 28nm technology, etc.) or the technological capability of a specific foundry company.

[0164] The processing core (120) may be designed to perform operations on integer parameters of a specific bit width. "8b INT Add" in FIG. 7 refers to an 8-bit integer addition operation of the adder (1212) of the processing element (121). An 8-bit integer addition operation may consume 0.03 pj of energy. "16b INT Add" refers to a 16-bit integer addition operation of the adder (1212) of the processing element (121). A 16-bit integer addition operation may consume 0.05 pj of energy. "32b INT Add" refers to a 32-bit integer addition operation of the adder (1212) of the processing element (121). A 32-bit integer addition operation may consume 0.1 pj of energy. "8b INT Mult" refers to an 8-bit integer multiplication operation of the multiplier (1211) of the processing element (121). An 8-bit integer multiplication operation can consume 0.2 pj of energy. "32b INT Mult" refers to a 32-bit integer multiplication operation of the multiplier (1211) of the processing element (121). A 32-bit integer multiplication operation can consume 3.1 pj of energy.

[0165] To elaborate, when the processing core (120) is composed of thousands to tens of thousands of processing elements designed as integer arithmetic units, the power consumption of the neural processing unit (100) can be significantly reduced compared to when the processing core is composed of the same number of processing elements designed as floating-point arithmetic units. Generally, since most of the computation in a generative artificial intelligence model is processed in the processing core (120), the design of an arithmetic unit with a specific integer bit width is important for the low-power design of the neural processing unit (100).

[0166] The vector core (153) and scalar core (154) of the special function unit (150) may be designed to perform operations on floating-point parameters. "16b FP Add" refers to a 16-bit floating-point addition operation of the adder of the special function unit (150). A 16-bit floating-point addition operation may consume 0.4 pj of energy. "32b FP Add" refers to a 32-bit floating-point addition operation of the adder of the special function unit (150). A 32-bit floating-point addition operation may consume 0.9 pj of energy. "16b FP Mult" refers to a 16-bit floating-point multiplication operation of the multiplier of the special function unit (150). A 16-bit floating-point multiplication operation may consume 1.1 pj of energy. "32b FP Mult" refers to a 32-bit floating-point multiplication operation of the multiplier of the special function unit (150). A 32-bit floating-point multiplication operation can consume 3.7 pj of energy. To elaborate, if the special function unit (150) is designed with floating-point arithmetic units, designing the number of arithmetic units of the special function unit (150) to be relatively smaller than the number of processing elements (121) of the processing core (120) is advantageous in terms of power consumption of the neural processing unit (100) and miniaturization of the semiconductor chip size. Therefore, the number of floating-point arithmetic units of the special function unit (150) can be designed to be smaller than the number of integer arithmetic units of the processing core (120).

[0167] According to one embodiment, the computational efficiency of the neural processing unit (100) can be improved by implementing a data pipeline circuit design of a processing core (120), a vector core (153), and a scalar core (154) so ​​that the array size of a specific dimension of tensor data processed by the neural processing unit (100) is compatible with each other among hardware components.

[0168] FIG. 8 is a flowchart illustrating a method for controlling a neural processing unit that can be applied to examples of the present disclosure.

[0169] Referring to FIG. 8, the neural processing unit (100) can perform integer operations based on input data through the processing core (120) in response to an input neural network operation request (S110). Here, the integer operations may include, for example, operations such as vector-matrix multiplication (MatMul), vector accumulation, or integer-based activation functions on quantized tensor data, and these operations can be performed in parallel in a plurality of processing elements arranged inside the processing core (120) of FIG. 3 described above. These operations are mainly performed in parallel in units of a plurality of processing elements inside the processing core (120), and by processing the input data and neural network weights in an integer format (INT8, INT16, etc.), high speed and low power characteristics can be satisfied simultaneously.

[0170] Next, the neural processing unit (100) can perform floating-point based special operations on the integer operation result performed by the processing core (120) in step S110 through the special function unit (150) (S120).

[0171] At this time, the special function unit (150) may be configured to perform non-linear function approximation operations (e.g., activation function operations (ReLU, GELU, tanh, etc.), normalization, softmax, or other non-linear function approximation operations) on the integer operation result converted to a floating-point number.

[0172] At this time, the special function unit (150) is configured to include a quantization unit (151), an inverse quantization unit (152), a vector core (153), and a scalar core (154), as shown in FIG. 3. The inverse quantization unit (152) inversely quantizes integer data into floating-point formats such as FP16 and BFLOAT, and the converted data is input into non-linear function operations. Additionally, the vector core (153) performs operations such as softmax, normalization, ReLU, and GELU in a high-speed parallel manner and is suitable for multi-dimensional tensor data. On the other hand, the scalar core (154) is used for conditional branching, precise function approximation, and single-value-based calculations, and may include an FSM-based control circuit for complex logic processing. These operations are suitable for high-precision calculations or non-linear activation processing that are difficult to implement with integer-based operations, and contribute to ensuring the accuracy of the entire operation flow.

[0173] Next, the neural processing unit (100) checks the operation status of each integer operation by step S110 and special operation by step S120 through the controller (110), and can control the operation of the processing core (120) or special function unit (150) according to the operation status (S130). At this time, as shown in FIG. 3, the controller (110) communicates with each module and can control the operation clock of each unit by considering the operation load, operation priority, operation condition flag, etc., or readjust the order of operations if necessary. For example, if the input data consists entirely of zeros, power consumption can be minimized by performing clock gating or power gating by omitting the corresponding integer operation or floating-point operation. In addition, operation scheduling can be optimized based on the decoding information of the execution instruction to prevent resource conflicts between operation modules or pipeline bottlenecks.

[0174] In this way, the control flow of FIG. 8 operates in close coordination with each hardware component of FIG. 3 and is designed with a structure that can maximize power efficiency along with improved computational performance.

[0175] FIG. 9 is a block diagram illustrating a system-on-chip according to a first embodiment of the first example of the present disclosure.

[0176] Referring to FIG. 9, an apparatus (10000) according to a first embodiment of a first example of the present disclosure is described. The apparatus (10000) includes a circuit board (BD), a memory (2000), and a system-on-chip (1000). The memory (2000) and the system-on-chip (1000) may be placed on a circuit board (not shown). The system-on-chip (1000) is configured to include a neural processing unit (100), a central processing unit (200), a first bus (BUS_1) (310), and a second bus (BUS_2) (320). The memory (2000) is configured to be electrically connected to the first bus (310). Each example and each embodiment of the present disclosure may be modified and combined with other examples and other embodiments of the present disclosure.

[0177] The system-on-chip (1000) may refer to a system manufactured from a single semiconductor substrate. The system-on-chip (1000) may be configured to further include a package (not shown) that protects the semiconductor substrate.

[0178] The neural processing unit (100) may be configured to communicate with the memory (2000). A first bus (310) may be provided between the neural processing unit (100) and the memory (2000). For example, the first bus (310) may be an AXI (Advanced Extensible Interface) bus. However, the examples of the present disclosure are not limited thereto. The first bus (310) may be configured to support a read and write address / control interface between the memory (2000) and the neural processing unit (100). The first bus (310) may be configured as an independent data communication channel to support high-speed data transmission for transaction processing in addition to memory control commands. The first bus (310) may be configured as a dedicated bus for transmitting input parameters, output parameters, and weight parameters of a neural network model, and it is desirable to appropriately design the bandwidth of the first bus (310) considering the parameter sizes of the neural network models to be processed.

[0179] The neural processing unit (100) may be configured to communicate with the central processing unit (200). A second bus (320) may be provided between the neural processing unit (100) and the central processing unit (200). For example, the second bus (320) may be an Advanced High-Performance Bus (AHB). However, the examples of the present disclosure are not limited thereto. The second bus (320) may be provided for efficient communication between the neural processing unit (100), the central processing unit (200), and peripheral devices of the system-on-chip (1000). The second bus (320) may provide a master-slave architecture to improve data throughput. The second bus (320) may be configured to support burst transmission and pipeline computation operations of the system-on-chip (1000). The second bus (320) may be configured to provide an on-the-fly function. The second bus (320) can be configured so that the master of the second bus (320) can execute new commands on the fly without waiting while the central processing unit (200) or the neural processing unit (100) reads or writes data in real time. Thus, real-time data processing of the neural processing unit (100) can be optimized, and at least one core of the neural processing unit (100) can be configured to use the bus independently. Thus, the second bus (320) can support the system-on-chip (1000) in performing various commands in real time.

[0180] To elaborate, the first bus (310) provides a higher memory bandwidth than the second bus (320), and the second bus (320) provides an on-the-fly function, thereby allowing the central processing unit (200) to dynamically control the neural processing unit (100). Accordingly, the system-on-chip (1000) can dynamically process various input queries to the generative neural network model.

[0181] Refer to FIGS. 3 to 7 for a description of the neural processing unit (100). Accordingly, a redundant description of the neural processing unit (100) is omitted.

[0182] The central processing unit (200) may be configured to control the neural processing unit (100) based on the execution code of the artificial intelligence model. The execution code may be generated by a compiler (not shown). The central processing unit (200) may control the neural processing unit (100) by directly setting the register values ​​of each circuit of the neural processing unit (100) based on the execution code. Here, the execution code may include each register value. Additionally, the central processing unit (200) may offload and process operations that are not processed by the neural processing unit (100). In this case, the neural network model may be compiled to offload specific operations of the neural network model to the central processing unit (200), and an execution code may be obtained.

[0183] The memory (2000) may include, but is not limited to, a main memory located outside the system-on-chip (1000). The memory (2000) may be electrically connected to the system-on-chip (1000) via a first bus (310). The memory (2000) of the device (10000) according to the first embodiment of the first example of the present disclosure may include at least one memory such as SRAM, LPDDR RAM, Resistive RAM, Magneto-resistive RAM, Phase-change RAM, Ferroelectric RAM, Flash Memory, HBM, etc. The memory (2000) may be composed of at least one memory unit (e.g., a bank, etc.). The memory (2000) may be composed of homogeneous memory or heterogeneous memory. It is preferable that the capacity of the memory (2000) be greater than the total size of the weight parameters of the neural network model. In this case, all of the weight parameters of the neural network model can be loaded into the memory (2000) at once and reside there. However, if the storage capacity of the memory (2000) is insufficient, only some of the weight parameters can be loaded, making it difficult to process the inference operations of the neural network model quickly in real time. For example, the capacity of the memory (2000) may be one of 4 GBytes or 8 GBytes. That is, the capacity of the memory (2000) may be 4 GBytes to 8 GBytes, and may be determined by considering the size of the parameters of the neural network model to be run on the system-on-chip (1000).

[0184] That is, the main memory (2000) is provided as dedicated memory for the neural processing unit (100) and can be configured to sufficiently store and access at high speed all essential data, including all learned weights of the neural network model as well as key activation values ​​generated during the inference operation and used in the next operation. While conventional large-scale models had to rely on external slow memory or servers for some of the weights or activation values, the present disclosure keeps all computation data resident within the on-device main memory, thereby minimizing data movement bottlenecks and eliminating external dependencies. This establishes a foundation for completing and executing the entire inference process of the neural network model on the device itself (server-independent operation) without exchanging data with an external server, which provides significant advantages in terms of real-time responsiveness and data privacy. This memory structure supports the neural processing unit (100) in processing even complex models stably without delay. As a result, a true on-device AI environment can be realized.

[0185] The bandwidth of the first bus (310) can be determined according to the processing performance of the neural processing unit (100) (e.g., Tera Operations Per Second (TOPS) of the neural processing unit (100)). For example, a bandwidth of 2 GB / s to 4 GB / s of memory (2000) per 1 TOPS may be required. For example, 10 TOPS of neural processing units (100) may have the memory (2000) configured to provide a bus bandwidth of 20 GB / s to 40 GB / s. The bandwidth of the first bus (310) can be determined according to the operating frequency of the memory (2000) and the number of communication channels. For example, to configure a memory bandwidth of 40 GB / s with LPDDR5, the following formula may be used.

[0186]

[0187] To elaborate, LPDDR5 generally supports a maximum data rate of 6400 Mbps per pin. The bus width of an LPDDR channel is generally 16 bits (2 bytes). However, the number of channels cannot be a decimal point and is an integer. However, the examples of the present disclosure are not limited to the bandwidth of the first bus (310).

[0188] The device (10000) according to the first embodiment of the first example of the present disclosure may be configured to provide a low-power function for use on a device.

[0189] The low-power mode of the device (10000) according to the first embodiment of the first example of the present disclosure may be provided, for example, a sleep mode, a deep sleep mode, a retention mode, etc. In some examples, the low-power mode may be referred to as a standby mode, a hibernation mode, an idle mode, etc.

[0190] A sleep mode can be implemented, for example, by applying clock gating to at least one of the processing core (120), vector core (153), and scalar core (154) of the neural processing unit (100).

[0191] Deep sleep mode can be implemented, for example, by applying power gating to at least one of the processing core (120), vector core (153), scalar core (154), or memory (2000) of the neural processing unit (100).

[0192] The retention mode can be implemented by supplying a minimum retention voltage to the controller (110) or memory (2000) of the neural processing unit (100).

[0193] For example, the neural processing unit (100) of the device (10000) according to the first embodiment of the first example of the present disclosure may be configured to provide at least one of power gating, clock gating, and register retention functions. Further details, when the device (10000) enters sleep mode, the neural processing unit (100) may activate the clock gating function. Further details, when the device (10000) enters deep sleep mode, the neural processing unit (100) may activate the power gating function. Further details, when the device (10000) enters retention mode, the supply voltage input to the neural processing unit (100) may be reduced or the operating frequency may be reduced.

[0194] For example, the memory (MEM) of the device (10000) according to the first embodiment of the first example of the present disclosure may provide at least one of deep sleep, retention, and maintain parameters functions. To elaborate, when the device (10000) enters deep sleep mode, the memory (2000) may be power-gated, and in the case of a volatile memory device, data may be lost. To elaborate, when the device (10000) enters retention mode, the supply voltage input to the memory (2000) may be reduced or the operating frequency may be reduced, and in the case of a volatile memory device, data may be maintained. To elaborate, when the device (10000) enters parameter maintenance mode, the supply voltage input to the memory (2000) may be reduced or the operating frequency may be reduced, and in the case of a DRAM, the refresh period of the memory cell may be increased, and in the case of a volatile memory device, data may be maintained.

[0195] A low-power mode may be activated when no input query for speculative decoding is entered into the device (10000) for a certain period of time or longer. The certain period may be provided, for example, based on a timer, counter value, etc. The low-power mode may be activated based on a specific threshold time and may be deactivated when an input query processed by speculative decoding is entered into the device (10000). The state in which the low-power mode is deactivated may be referred to as a wake-up state.

[0196] The device (10000) according to the first embodiment of the first example of the present disclosure may be a device capable of operating in a battery environment. Accordingly, the neural processing unit (100) may be designed to operate at battery voltage.

[0197] A device (10000) according to the first embodiment of the first example of the present disclosure may be configured to provide at least one low-power mode while simultaneously providing speculative decoding in an on-device environment. The device (10000) may enter a low-power mode based on preset conditions, thereby reducing the power consumption of the device (10000).

[0198] The system-on-chip (1000) according to the first embodiment of the first example of the present disclosure has the effect of rapidly accelerating inference operations of a generative neural network model at low power through a pipeline circuit structure of a processing core (120), a vector core (153), and a scalar core (154).

[0199] FIG. 10 is a block diagram illustrating a system-on-chip according to a second embodiment of the first example of the present disclosure.

[0200] Referring to FIG. 10, an apparatus (10000) according to a second embodiment of the first example of the present disclosure is described. The apparatus (10000) includes a circuit board (BD), a plurality of main memories (M_MEM_1,..., M_MEM_N) (2000-1,..., 2000-N), and a system-on-chip (1000). In describing the apparatus (10000) according to the second embodiment of the first example of the present disclosure, any content that overlaps with the description with reference to FIG. 3 through FIG. 9 may be omitted for convenience of explanation. Each example and each embodiment of the present disclosure may be modified and implemented by combining with other examples and other embodiments of the present disclosure.

[0201] The system-on-chip (1000) may refer to a system manufactured from a single semiconductor substrate. The system-on-chip (1000) may be configured to further include a package (not shown) that protects the semiconductor substrate.

[0202] A system-on-chip (1000) according to a second embodiment of the first example of the present disclosure is configured to include a neural processing unit (100), a central processing unit (200), a first bus (310), and a second bus (320). A plurality of main memories (2000-1, …, 2000-N) are configured to be electrically connected to the first bus (310). A plurality of main memories (2000-1, …, 2000-N) are configured to include a plurality of semiconductor chips.

[0203] A first bus (310) may be provided between the neural processing unit (100) and a plurality of main memories (2000-1, …, 2000-N). A second bus (320) may be provided between the neural processing unit (100) and the central processing unit (200).

[0204] The operation of the neural processing unit (100) has been described in detail with reference to FIGS. 3 through 9, so a detailed description of the operation of the neural processing unit (100) is omitted. The operation of the central processing unit (200) has been described in detail with reference to FIG. 9, so a detailed description of the operation of the central processing unit (200) is omitted.

[0205] A plurality of main memories (2000-1, …, 2000-N) may be composed of a plurality of memories disposed outside the system-on-chip (1000). However, the present disclosure is not limited thereto. A plurality of main memories (2000-1, …, 2000-N) may each be electrically connected to the system-on-chip (1000) through a first bus (310). Each of the plurality of main memories (2000-1, …, 2000-N) of the device (10000) according to the second embodiment of the first example of the present disclosure may include SRAM, LPDDR RAM, Resistive RAM, Magneto-resistive RAM, Phase-change RAM, Ferroelectric RAM, Flash Memory, or HBM, etc. A plurality of main memories (2000-1, …, 2000-N) may be composed of homogeneous memory or heterogeneous memory.

[0206] It is desirable that the total capacity of the multiple main memories (2000-1, …, 2000-N) be greater than the total capacity occupied by the weight parameters of the artificial intelligence model. In this case, the weight parameters of the neural network model can be loaded and reside in the multiple main memories (2000-1, …, 2000-N) at once. If the storage capacity of the multiple main memories (2000-1, …, 2000-N) is insufficient, it may be difficult to process the inference operations of the neural network model quickly in real time. For example, the total capacity of the multiple main memories (2000-1, …, 2000-N) may be one of 8 GByte, 16 GByte, 32 GByte, 64 GByte, or 128 GByte. That is, the total capacity of the multiple main memories (2000-1, …, 2000-N) may be 8 GByte to 128 GByte, and may be determined by considering the parameter size of at least one artificial intelligence model to be run on the system-on-chip (1000). Accordingly, the device (10000) can reduce the manufacturing cost of the system-on-chip (1000) while easily increasing the capacity of the main memory by placing the multiple main memories for loading and storing the parameters of the artificial intelligence model outside the system-on-chip (1000). If the multiple main memories are placed inside a package protecting the system-on-chip (1000), the size of the package may increase and the manufacturing cost may increase, but the criteria of a specific form factor (e.g., M.2 2230, M.2 2242) may be satisfied.

[0207] The bandwidth of the first bus (310) may be determined according to the processing performance of the neural processing unit (100) (e.g., Tera Operations Per Second (TOPS) of the neural processing unit (100)). For example, a plurality of main memories (2000-1, …, 2000-N) may be configured to provide a bus bandwidth of 60 GB / s to 120 GB / s for a neural processing unit (N100) of 30 TOPS. The bandwidth of the first bus (310) may be determined according to the operating frequency of the plurality of main memories (2000-1, …, 2000-N) and the number of communication channels. However, the examples of the present disclosure are not limited to the bandwidth of the first bus (310).

[0208] The device (10000) according to the second embodiment of the first example of the present disclosure provides a plurality of main memories (2000-1, …, 2000-N), and the total capacity of the main memories may be a capacity capable of storing all parameters of at least one neural network model at once. In addition, according to the configuration of the second embodiment of the first example, the memory capacity can be expanded to correspond to the size of a generative neural network model in which the size of the weight parameters is substantial. Thus, the system-on-chip (1000) can rapidly process the neural network model in real time. To elaborate, the plurality of main memories (2000-1, …, 2000-N) of the device (10000) according to the embodiments of the present disclosure are configured as dedicated memories of the system-on-chip (1000). If the parameters of the artificial intelligence model processed by the neural processing unit (100) are not all stored in multiple main memories (2000-1, …, 2000-N), in particular, if all weight parameters and attention scores cannot be stored in multiple main memories (2000-1, …, 2000-N), the parameters must be stored in a separate storage device, and a decrease in computational speed due to large-capacity memory operations is caused.

[0209] FIG. 11 is a block diagram illustrating a system-on-chip according to a third embodiment of the first example of the present disclosure.

[0210] With reference to FIG. 11, an apparatus (10000) according to a third embodiment of the first example of the present disclosure is described. The apparatus (10000) includes a circuit board (BD), a main memory (2000), and a system-on-chip (1000). In describing the system-on-chip (1000) according to the third embodiment of the first example of the present disclosure, any redundancy in the descriptions above with reference to FIG. 3 through FIG. 10 may be omitted for convenience of explanation. Each example and each embodiment of the present disclosure may be modified and implemented by combining with other examples and other embodiments of the present disclosure.

[0211] The system-on-chip (1000) may refer to a system manufactured from a single semiconductor substrate. The system-on-chip (1000) may be configured to further include a package (not shown) that protects the semiconductor substrate.

[0212] A system-on-chip (1000) according to the third embodiment of the first example of the present disclosure is configured to include a neural processing unit (100), a central processing unit (200), a first bus (310), a second bus (320), a third bus (330), and a shared memory (S_MEM) (400). The shared memory (400) is configured as an on-chip memory of the system-on-chip (1000). The main memory (2000) is configured to be electrically connected to the first bus (310). The main memory (2000) is configured to include at least one semiconductor chip. The shared memory (400) is configured to be electrically connected to the third bus (330).

[0213] A first bus (310) may be located between the neural processing unit (100) and the main memory (2000). A second bus (320) may be located between the neural processing unit (100) and the central processing unit (200). A third bus (330) may be located between the neural processing unit (100) and the shared memory (400). For example, the third bus (330) may be an AXI (Advanced Extensible Interface) bus. However, the examples of the present disclosure are not limited thereto. The third bus (330) may be configured to support a read and write address / control interface between the shared memory (400) and the neural processing unit (100). The third bus (330) may be configured as an independent data communication channel to support high-speed data transmission for transaction processing in addition to memory control commands. The third bus (330) may be configured as a dedicated bus for transmitting input parameters, output parameters, and weight parameters of the neural network model. Here, the first bus (310) and the third bus (330) are configured to be electrically connected. Accordingly, the shared memory (400) and the main memory (2000) can transmit tensor data through the first bus (310) and the third bus (330).

[0214] The operation of the neural processing unit (100) has been described in detail with reference to FIGS. 3 to 10, so a detailed description is omitted. The operation of the central processing unit (200) has been described in detail with reference to FIGS. 9 to 10, so a detailed description is omitted.

[0215] The main memory (2000) may consist of at least one memory placed outside the system-on-chip (1000). The main memory (2000) may consist of homogeneous memory or heterogeneous memory. The capacity of the main memory (2000) may be determined by considering the parameter size of at least one neural network model to be run on the system-on-chip (1000). The bandwidth of the first bus (310) may be determined based on the processing performance of the neural processing unit (100).

[0216] The shared memory (400) may be composed of at least one memory placed inside the system-on-chip (1000). The shared memory (400) of the system-on-chip (1000) according to the third embodiment of the first example of the present disclosure may include at least one memory such as SRAM, LPDDR RAM, Resistive RAM, Magneto-resistive RAM, Phase-change RAM, Ferroelectric RAM, Flash Memory, HBM, etc. The shared memory (400) may be composed of homogeneous memory or heterogeneous memory. As a representative embodiment, the shared memory (400) may be implemented as SRAM. The capacity of the shared memory (400) is designed to be smaller than the capacity of the main memory (2000). If the capacity of the shared memory (400) increases, the semiconductor manufacturing cost may increase rapidly. Therefore, the capacity of the shared memory (400) may be configured to be 4 MByte to 128 MByte. Preferably, the capacity of the shared memory (400) may be configured to be 16 MByte to 64 MByte. The capacity of the shared memory (400) may be larger than the capacity of the internal memory (140) of the neural processing unit (100).

[0217] The operation of the shared memory (400) and the main memory (2000) will be explained with reference to FIG. 7. “32b S_MEM Read” in FIG. 9 refers to a 32-bit data read operation of the main memory (2000). Here, the main memory (2000) may be an LPDDR RAM. In this case, the 32-bit data read operation of the main memory (2000) may consume 640 pj of energy. “32b S_MEM Read” in FIG. 9 refers to a 32-bit data read operation of the shared memory (400). Here, the shared memory (400) may be an SRAM. In this case, the 32-bit data read operation of the shared memory (400) may consume 5 pj of energy. That is, when the neural processing unit (100) uses a shared memory (400) implemented in SRAM, the energy consumption of the device (10000) can be significantly reduced compared to when only the main memory (2000) is used. However, since the manufacturing cost of the shared memory (400) is high, it is difficult to implement it in a large capacity. Therefore, when a compiler (not shown) generates execution code for a neural network model to be processed by the system-on-chip (1000), it can set reusable input parameters, output parameters, and weight parameters to be stored preferentially in the shared memory (400) rather than in the main memory (2000). That is, the compiler (not shown) can determine reusable parameters during a series of neural network model operations and provide a function to store them in the shared memory (400) for reuse.

[0218] A system-on-chip (1000) according to the third embodiment of the first example of the present disclosure provides a shared memory (400), and by executing an execution code configured to store reusable parameters in the shared memory (400), the power consumption of the system-on-chip (1000) can be reduced.

[0219] FIG. 12 is a block diagram illustrating a system-on-chip according to a fourth embodiment of the first example of the present disclosure.

[0220] Referring to FIG. 12, an apparatus (10000) according to a fourth embodiment of the first example of the present disclosure is described. The apparatus (10000) includes a circuit board (BD), a main memory (2000), and a system-on-chip (1000). In describing the system-on-chip (1000) according to the fourth embodiment of the first example of the present disclosure, descriptions that are redundant in FIG. 3 through FIG. 11 may be omitted for convenience of explanation. Each example and each embodiment of the present disclosure may be modified and implemented by combining with other examples and other embodiments of the present disclosure.

[0221] The system-on-chip (1000) may refer to a system manufactured from a single semiconductor substrate. The system-on-chip (1000) may be configured to further include a package (not shown) that protects the semiconductor substrate.

[0222] A system-on-chip (1000) according to the fourth embodiment of the first example of the present disclosure is configured to include a neural processing unit (100), a central processing unit (200), a first bus (310), a second bus (320), a third bus (330), and a plurality of shared memories (400-1, …, 400-N). The plurality of shared memories (400-1, …, 400-N) are composed of on-chip memories of the system-on-chip (1000). A main memory (2000) is configured to be electrically connected to the first bus (310). The main memory (2000) is configured to include at least one semiconductor chip. The plurality of shared memories (400-1, …, 400-N) are configured to be electrically connected to the third bus (330).

[0223] A first bus (310) may be located between the neural processing unit (100) and the main memory (2000). A second bus (320) may be located between the neural processing unit (100) and the central processing unit (200). A third bus (330) may be located between the neural processing unit (100) and a plurality of shared memories (400-1, …, 400-N). Here, the first bus (310) and the third bus (330) are configured to be electrically connected.

[0224] Since the operation of the neural processing unit (100) has been described with reference to FIGS. 3 to 11, a redundant description of the operation of the neural processing unit (100) is omitted. Since the operation of the central processing unit (200) has been described with reference to FIGS. 9 to 11, a redundant description of the central processing unit (200) is omitted.

[0225] The main memory (2000) may consist of at least one memory placed outside the system-on-chip (1000). The main memory (2000) may consist of homogeneous memory or heterogeneous memory. The capacity of the main memory (2000) may be determined by considering the parameter size of at least one neural network model to be run on the system-on-chip (1000). The bandwidth of the first bus (310) may be determined according to the processing performance of the neural processing unit (100).

[0226] A plurality of shared memories (400-1, …, 400-N) may be composed of a plurality of memories disposed within the system-on-chip (1000). Each shared memory may be configured to operate independently of one another. A plurality of shared memories (400-1, …, 400-N) of the system-on-chip (1000) according to the fourth embodiment of the first example of the present disclosure may include at least one memory among SRAM, LPDDR RAM, Resistive RAM, Magneto-resistive RAM, Phase-change RAM, Ferroelectric RAM, Flash Memory, HBM, etc. A plurality of shared memories (400-1, …, 400-N) may be composed of homogeneous memory or heterogeneous memory. Here, a plurality of shared memories (400-1, …, 400-N) is described as an example implemented as SRAM. The capacity of the multiple shared memories (400-1, …, 400-N) is designed to be smaller than the capacity of the main memory (2000). If the capacity of the multiple shared memories (400-1, …, 400-N) increases, the semiconductor manufacturing cost may increase rapidly. Therefore, the total capacity of the multiple shared memories (400-1, …, 400-N) can be configured to be 4 MByte to 128 MByte. Preferably, the total capacity of the multiple shared memories (400-1, …, 400-N) can be configured to be 16 MByte to 64 MByte. The capacity of the multiple shared memories (400-1, …, 400-N) can be configured to be larger than the capacity of the internal memory (140) of the neural processing unit (100).

[0227] When the neural processing unit (100) uses multiple shared memories (400-1, …, 400-N) implemented as SRAM, the energy consumption of the device (10000) can be significantly reduced compared to when only the main memory (2000) is used. Accordingly, when the compiler (not shown) generates executable code for a neural network model to be processed by the system-on-chip (1000), it can set a data reuse command to preferentially store reusable input parameters, output parameters, and weight parameters in multiple shared memories (400-1, …, 400-N) located inside the system-on-chip (1000) rather than in the main memory (2000).

[0228] A system-on-chip (1000) according to the fourth embodiment of the first example of the present disclosure provides a plurality of shared memories (400-1, …, 400-N) and can reduce the power consumption of the system-on-chip (1000) by executing an execution code configured to preferentially store reusable parameters in the plurality of shared memories (400-1, …, 400-N). In addition, by providing a plurality of independent shared memories for each domain (e.g., weight domain, attention score domain, etc.), memory operations of each domain (e.g., weight, input parameter, output parameter) of the neural network model parameters can be easily processed.

[0229] A system-on-chip (1000) according to the fourth embodiment of the first example of the present disclosure provides a plurality of shared memories (400-1, …, 400-N) and can reduce the power consumption of the system-on-chip (1000) by executing an execution code configured to preferentially store reusable parameters in the plurality of shared memories (400-1, …, 400-N). In addition, by providing a plurality of independent shared memories, memory operations of each domain of the parameters of the neural network model (e.g., weights, input parameters, output parameters) can be easily processed.

[0230] FIG. 13 is a block diagram illustrating a system-on-chip according to the fifth embodiment of the first example of the present disclosure.

[0231] With reference to FIG. 13, an apparatus (10000) according to the fifth embodiment of the first example of the present disclosure is described. The apparatus (10000) includes a circuit board (BD), a plurality of main memories (2000-1, …, 2000-N), and a system-on-chip (1000). In describing the system-on-chip (1000) according to the fifth embodiment of the first example of the present disclosure, descriptions that are redundant in FIG. 3 through FIG. 12 may be omitted for convenience of explanation. Each example and each embodiment of the present disclosure may be modified and implemented by combining with other examples and other embodiments of the present disclosure.

[0232] The system-on-chip (1000) may refer to a system manufactured from a single semiconductor substrate. The system-on-chip (1000) may be configured to further include a package (not shown) that protects the semiconductor substrate.

[0233] A system-on-chip (1000) according to the fifth embodiment of the first example of the present disclosure is configured to include a neural processing unit (100), a central processing unit (200), a first bus (310), a second bus (320), a third bus (330), and a plurality of shared memories (400-1, …, 400-N). The plurality of shared memories (400-1, …, 400-N) are composed of on-chip memories of the system-on-chip (1000). A plurality of main memories (2000-1, …, 2000-N) are configured to be electrically connected to the first bus (310). A plurality of main memories (2000-1, …, 2000-N) are configured to include at least one semiconductor chip. A plurality of shared memories (400-1, …, 400-N) are configured to be electrically connected to the third bus (330).

[0234] A first bus (310) may be provided between the neural processing unit (100) and a plurality of main memories (2000-1, ..., 2000-N). A second bus (320) may be provided between the neural processing unit (100) and the central processing unit (200). A third bus (330) may be provided between the neural processing unit (100) and a plurality of shared memories (400-1, ..., 400-N). Here, the first bus (310) and the third bus (330) are configured to be electrically connected.

[0235] Refer to FIGS. 3 to FIGS. 12 for a description of the neural processing unit (100). Accordingly, a redundant description of the neural processing unit (100) is omitted. Refer to FIGS. 9 to FIGS. 12 for a description of the central processing unit (200). Accordingly, a redundant description of the central processing unit (200) is omitted. Refer to FIG. 10 for a description of the plurality of main memories (2000-1, ..., 2000-N). Accordingly, a redundant description of the plurality of main memories (2000-1, ..., 2000-N) is omitted. Refer to FIG. 12 for a description of the plurality of shared memories (400-1, ..., 400-N). Accordingly, a redundant description of the plurality of shared memories (400-1, ..., 400-N) is omitted.

[0236] Multiple main memories (2000-1, ..., 2000-N) may be composed of multiple memories placed outside the system-on-chip (1000). The total capacity of the multiple main memories (2000-1, ..., 2000-N) may be determined by considering the parameter size of at least one neural network model to be run on the system-on-chip (1000). The total capacity of the multiple main memories (2000-1, ..., 2000-N) may be configured to be larger than the total capacity of the multiple shared memories (400-1, ..., 400-N). Thus, by providing multiple main memories outside the system-on-chip (1000), the manufacturing cost of the system-on-chip (1000) can be reduced.

[0237] Multiple shared memories (400-1, …, 400-N) may be composed of multiple memories placed inside the system-on-chip (1000). The total capacity of the multiple shared memories (400-1, …, 400-N) may be configured to be larger than the capacity of the internal memory (140) of the neural processing unit (100). Accordingly, by placing reusable parameters in the multiple shared memories (400-1, …, 400-N), the power consumption of the device (10000) can be reduced by reducing the amount of data read or transmitted from the multiple main memories (2000-1, …, 2000-N).

[0238] When a compiler (not shown) generates executable code for a neural network model to be processed by a system-on-chip (1000), it can be configured to store reusable input parameters, output parameters, and weight parameters in multiple shared memories (400-1, …, 400-N) preferentially over multiple main memories (2000-1, …, 2000-N).

[0239] The device (10000) according to the fifth embodiment of the first example of the present disclosure can improve data reuse while supporting a generative neural network model with a large number of parameters, with an expandable main memory and an expandable shared memory, by combining the features of the second embodiment of the first example of the present disclosure and the fourth embodiment of the first example of the present disclosure.

[0240] FIG. 14 is a block diagram illustrating a system-on-chip according to the sixth embodiment of the first example of the present disclosure.

[0241] Referring to FIG. 14, an apparatus (10000) according to the sixth embodiment of the first example of the present disclosure is described. The apparatus (10000) includes a circuit board (BD), a memory (2000), and a system-on-chip (1000). In describing the system-on-chip (1000) according to the sixth embodiment of the first example of the present disclosure, descriptions that are redundant in FIG. 3 through FIG. 13 may be omitted for convenience of explanation. Each example and each embodiment of the present disclosure may be modified and implemented by combining with other examples and other embodiments of the present disclosure.

[0242] The system-on-chip (1000) may refer to a system manufactured from a single semiconductor substrate. The system-on-chip (1000) may be configured to further include a package (not shown) that protects the semiconductor substrate.

[0243] A system-on-chip (1000) according to the sixth embodiment of the first example of the present disclosure is configured to include a plurality of neural processing units (NPU_1, ..., NPU_N) (100-1, ..., 100-N), a central processing unit (200), a first bus (310), and a second bus (320). A memory (2000) is configured to be electrically connected to the first bus (310).

[0244] Each of the plurality of neural processing units (100-1, ..., 100-N) may be configured to communicate with each other with the memory (2000). A first bus (310) may be provided between the plurality of neural processing units (100-1, ..., 100-N) and the memory (2000).

[0245] Each of the plurality of neural processing units (100-1, ..., 100-N) may be configured to communicate with each other with the central processing unit (200). A second bus (320) may be provided between the plurality of neural processing units (100-1, ..., 100-N) and the central processing unit (200).

[0246] Refer to FIGS. 3 to FIGS. 13 for a description of the multiple neural processing units (100-1, ..., 100-N). Accordingly, a redundant description of the neural processing unit (100) is omitted.

[0247] The processing performance (TOPS) of each of the multiple neural processing units (100-1, ..., 100-N) may be the same or different from one another. Each of the multiple neural processing units (100-1, ..., 100-N) may be configured to operate independently. Each of the multiple neural processing units (100-1, ..., 100-N) may be configured to process the computation of a specific neural network model in parallel.

[0248] Refer to FIGS. 9 to 13 for a description of the central processing unit (200). Accordingly, a redundant description of the central processing unit (200) is omitted.

[0249] The central processing unit (200) may be configured to control each of the plurality of neural processing units (100-1, ..., 100-N) based on the execution code of the neural network model. Here, a compiler (not shown) may be configured to generate respective execution codes corresponding to each of the plurality of neural processing units (100-1, ..., 100-N). The central processing unit (200) may be configured to control the plurality of neural processing units (100-1, ..., 100-N) by directly setting the register values ​​of the circuits of each of the plurality of neural processing units (100-1, ..., 100-N) based on each execution code.

[0250] The memory (2000) may be composed of a main memory placed outside the system-on-chip (1000). The memory (2000) may be composed of homogeneous memory or heterogeneous memory. The capacity of the memory (2000) may be determined by considering the parameter size of at least one neural network model to be run on the system-on-chip (1000). The bandwidth of the first bus (310) may be determined according to the processing performance of a plurality of neural processing units (100-1, ..., 100-N).

[0251] The bandwidth of the first bus (310) can be determined according to the processing performance of a plurality of neural processing units (100-1, ..., 100-N) (e.g., Tera Operations Per Second (TOPS) of the neural processing unit (100)). For example, a bandwidth of 2 GB / s to 4 GB / s of memory (2000) per TOPS may be required. For example, if there are two neural processing units (100) of 10 TOPS, the memory (2000) may be configured to provide a bus bandwidth of 40 GB / s to 80 GB / s. The bandwidth of the first bus (310) can be determined according to the operating frequency of the memory (2000) and the number of communication channels.

[0252] The system-on-chip (1000) according to the sixth embodiment of the first example of the present disclosure provides a plurality of neural processing units (100-1, ..., 100-N), thereby improving the parallel processing performance of the neural network model and having the effect of generating a response to an input query faster.

[0253] FIG. 15 is a block diagram illustrating a system-on-chip according to the seventh embodiment of the first example of the present disclosure.

[0254] Referring to FIG. 15, an apparatus (10000) according to the seventh embodiment of the first example of the present disclosure is described. The apparatus (10000) includes a circuit board (BD), a plurality of main memories (2000-1, ..., 2000-N), and a system-on-chip (1000). In describing the system-on-chip (1000) according to the seventh embodiment of the first example of the present disclosure, descriptions that are redundant in FIG. 3 through FIG. 14 may be omitted for convenience of explanation. Each example and each embodiment of the present disclosure may be modified and implemented by combining with other examples and other embodiments of the present disclosure.

[0255] The system-on-chip (1000) may refer to a system manufactured from a single semiconductor substrate. The system-on-chip (1000) may be configured to further include a package (not shown) that protects the semiconductor substrate.

[0256] A system-on-chip (1000) according to the seventh embodiment of the first example of the present disclosure is configured to include a plurality of neural processing units (100-1, ..., 100-N), a central processing unit (200), a first bus (310), a second bus (320), a third bus (330), and a plurality of shared memories (400-1, ..., 400-N). The plurality of shared memories (400-1, ..., 400-N) are composed of on-chip memories of the system-on-chip (1000). A plurality of main memories (2000-1, ..., 2000-N) are configured to be electrically connected to the first bus (310). A plurality of main memories (2000-1, ..., 2000-N) are configured to include at least one semiconductor chip. A plurality of shared memories (400-1, ..., 400-N) are configured to be electrically connected to the third bus (330).

[0257] A first bus (310) may be provided between the neural processing unit (100) and a plurality of main memories (2000-1, ..., 2000-N). A second bus (320) may be provided between the neural processing unit (100) and the central processing unit (200). A third bus (330) may be provided between the neural processing unit (100) and a plurality of shared memories (400-1, ..., 400-N). Here, the first bus (310) and the third bus (330) are configured to be electrically connected.

[0258] Refer to FIGS. 3 to FIGS. 14 for a description of the neural processing unit (100). Accordingly, a redundant description of the neural processing unit (100) is omitted. Refer to FIGS. 9 to FIGS. 14 for a description of the central processing unit (200). Accordingly, a redundant description of the central processing unit (200) is omitted. Refer to FIG. 10 for a description of the plurality of main memories (2000-1, ..., 2000-N). Accordingly, a redundant description of the plurality of main memories (2000-1, ..., 2000-N) is omitted. Refer to FIG. 12 for a description of the plurality of shared memories (400-1, ..., 400-N). Accordingly, a redundant description of the plurality of shared memories (400-1, ..., 400-N) is omitted. Refer to FIG. 14 for a description of the plurality of neural processing units (100-1, ..., 100-N). Accordingly, redundant descriptions of multiple neural processing units (100-1, ..., 100-N) are omitted.

[0259] Multiple main memories (2000-1, ..., 2000-N) may be composed of multiple memories placed outside the system-on-chip (1000). The total capacity of the multiple main memories (2000-1, ..., 2000-N) may be determined by considering the parameter size of at least one neural network model to be run on the system-on-chip (1000). The total capacity of the multiple main memories (2000-1, ..., 2000-N) may be configured to be larger than the total capacity of the multiple shared memories (400-1, ..., 400-N). Thus, by providing multiple main memories outside the system-on-chip (1000), the manufacturing cost of the system-on-chip (1000) can be reduced.

[0260] Multiple shared memories (400-1, ..., 400-N) may be composed of multiple memories placed inside the system-on-chip (1000). The total capacity of the multiple shared memories (400-1, ..., 400-N) may be configured to be larger than the capacity of the internal memory (140) of the neural processing unit (100). Accordingly, by placing reusable parameters in the multiple shared memories (400-1, ..., 400-N), the power consumption of the device (10000) can be reduced by reducing the amount of data read or transmitted from the multiple main memories (2000-1, ..., 2000-N).

[0261] When a compiler (not shown) generates executable code for a neural network model to be processed by a system-on-chip (1000), it can be configured to store reusable input parameters, output parameters, and weight parameters in multiple shared memories (400-1, ..., 400-N) preferentially over multiple main memories (2000-1, ..., 2000-N).

[0262] The apparatus (10000) according to the seventh embodiment of the first example of the present disclosure provides a plurality of neural processing units, an expandable main memory, and an expandable shared memory by combining the features of the fifth embodiment of the first example and the sixth embodiment of the first example of the present disclosure. Accordingly, data reuse can be improved while the plurality of neural processing units support the parallel processing of a generative neural network model with a large number of parameters.

[0263] FIG. 16 is a block diagram illustrating a system-on-chip according to the eighth embodiment of the first example of the present disclosure.

[0264] A system-on-chip (1000) according to the eighth embodiment of the first example of the present disclosure is described with reference to FIG. 16. In describing the system-on-chip (1000) according to the eighth embodiment of the first example of the present disclosure, descriptions that are redundant in FIG. 3 to FIG. 15 may be omitted for convenience of explanation. Each example and each embodiment of the present disclosure may be modified and implemented by combining with other examples and other embodiments of the present disclosure.

[0265] The system-on-chip (1000) may refer to a system manufactured from a single semiconductor substrate. The system-on-chip (1000) may be configured to further include a package (not shown) that protects the semiconductor substrate.

[0266] A system-on-chip (1000) according to the eighth embodiment of the first example of the present disclosure is configured to include a neural processing unit (100), a central processing unit (200), a first bus (310), and a second bus (320). Herein, the system-on-chip (1000) according to the eighth embodiment of the first example of the present disclosure is configured to further include a digital signal processing unit (500). A memory (2000) is configured to be electrically connected to the first bus (310). The memory (2000) is configured to include at least one semiconductor chip.

[0267] A first bus (310) may be provided between the neural processing unit (100) and the memory (2000). A first bus (310) may be provided between the digital signal processing unit (500) and the memory (2000). A second bus (320) may be provided between the neural processing unit (100) and the central processing unit (200). A second bus (320) may be provided between the digital signal processing unit (500) and the central processing unit (200).

[0268] Refer to FIGS. 3 to 15 for a description of the neural processing unit (100). Accordingly, a redundant description of the neural processing unit (100) is omitted. Refer to FIGS. 9 to 15 for a description of the central processing unit (200). Accordingly, a redundant description of the central processing unit (200) is omitted. Refer to FIGS. 9 to 15 for a description of the memory (2000). Accordingly, a redundant description of the memory (2000) is omitted.

[0269] The digital signal processing unit (500) may be configured to process at least a portion of the computation of a neural network model that is inefficient to process in the neural processing unit (100). The digital signal processing unit (500) includes at least one vector processor (not shown) and at least one scalar processor (not shown).

[0270] A system-on-chip (1000) according to the eighth embodiment of the first example of the present disclosure further provides a digital signal processing unit (500) to offload specific operations of a neural network model that are inefficient to process in the neural processing unit (100). Thus, the system-on-chip (1000) can rapidly process the neural network model in real time.

[0271] In some examples, a dedicated bus or high-speed bus interface operating separately from the system bus may be further included to improve the transfer speed of parameters or computational data between the shared memory(s) and the neural processing unit (100). Here, the system bus is a common data transmission path within the system-on-chip (1000) including the second bus (320), the third bus (330), etc., and may be configured to support communication between modules such as the neural processing unit, the central processing unit, and the on-chip memory. Meanwhile, the dedicated bus may perform the role of connecting model parameters of a specific domain stored in the shared memory in parallel with a plurality of neural processing units to alleviate data transmission bottlenecks and enable high-speed access. This separate bus configuration is designed to reduce the load on the main bus (e.g., the first bus (310)) while simultaneously optimizing the computational path accessing the shared memory.

[0272] Various system-on-chip (SoC) embodiments according to the first example of the present disclosure (see FIGS. 9 through 16) provide a hardware foundation for implementing a scalable on-device AI acceleration platform with high parallel processing capabilities and flexibility, and such a platform may include the following key features optionally or in combination:

[0273] First, according to various examples, a multi-NPU core architecture may be provided that maximizes parallel processing capabilities by integrating multiple NPU cores, each having optimized power, performance, and area (PPA) (each NPU includes an integer processing core (P_CORE) and a special function unit (SFU) for handling floating-point (FP) operations), onto a single chip. (e.g., see the 6th embodiment of the first example (Fig. 14) and the 7th embodiment (Fig. 15).)

[0274] Second, a hierarchical / distributed memory system may be provided, comprising multiple independent on-chip shared memory (SRAM) banks allocated to each NPU core or group of NPU cores to enhance data locality, and a multi-channel main memory (external DRAM) system accessible to all of these NPU cores and providing high total bandwidth. (e.g., see the third, fourth, fifth, and seventh embodiments of the first example (Figs. 11, 12, 13, 15)). This efficiently manages complex data flows and minimizes memory bottlenecks.

[0275] Third, the special function unit (SFU) within the NPU core may include a programmable piecewise affine function (PAF) unit. (e.g., see the configuration of the SFU (150) within the NPU (100) of FIG. 3 and the fourth example (Figs. 24, 25, 27)). This allows for software-based support of various current and future activation functions and non-linear operations without separate hardware changes, thereby ensuring the flexibility and future scalability of the system.

[0276] Fourth, in a system configuration according to the first example of the present disclosure (e.g., FIGS. 9 to 16), a digital signal processing device (DSP, 500) may be optionally integrated to offload data processing before and after neural network computation (e.g., sensor data preprocessing, postprocessing) or specific computations that are inefficient to process in the NPU (Fig. 16 illustrates one example of such DSP integration). This improves the overall efficiency of the system by allowing the NPU to focus on core AI computations.

[0277] The organic combination of the aforementioned components creates powerful synergy. Multiple NPU cores and a distributed / hierarchical memory structure provide extreme parallelism and high data feeding capabilities, enabling the simultaneous or very rapid processing of complex or multiple AI models in an on-device environment. The addition of programmable SFUs provides flexibility not limited to specific models or computations, as well as adaptability to new future AI algorithms, within the on-device environment. The optional integration of DSPs further expands the scope of application for this platform.

[0278] In summary, the scalable on-device artificial intelligence (AI) acceleration system according to the present disclosure may include: one or more neural processing unit (NPU) cores, each comprising an integer arithmetic processing core and a floating-point special function unit (SFU); one or more on-chip shared memories (optional) connected to each of the NPU cores or a group of NPU cores to provide data locality; a high-bandwidth main memory interface shared by the NPU cores for storing large amounts of data; and an integrated controller that distributes AI operations to the NPU cores and (optional) the DSP, and controls data flow through the on-chip shared memory and the main memory interface. In this case, the floating-point special function unit (SFU) may include a programmable function approximation (PAF) circuit that approximates various non-linear functions according to input parameters. Additionally, a digital signal processing unit (DSP) that performs specific signal processing operations independently of the operations of the NPU cores may be optionally provided. This enables the parallel and flexible acceleration of various AI models and workloads in an on-device environment, making it possible to realize high-performance AI applications.

[0279] Various system-on-chip (SoC) embodiments according to the first example of the present disclosure (see FIGS. 9 through 16) can create integrated synergy effects beyond simple parallel arrangement of hardware by combining with the unique architecture of the neural processing unit (100) described above. This is a key feature of the present disclosure that solves technical challenges in implementing high-performance on-device AI and can be embodied as follows.

[0280] For example, intelligent linkage between a hierarchical / distributed memory system and a hybrid function approximation unit can be implemented by the example described above. As illustrated in FIGS. 11, 13, 15, etc., a system according to the first example of the present disclosure may have a hierarchical memory structure composed of a large-capacity external main memory (2000) and a high-speed on-chip shared memory (400). The effect of such a memory system may be enhanced when it operates in conjunction with a hybrid structure of a special function unit (150) that combines a programmable method and a dedicated hardware method to process various non-linear functions. Specifically, the control unit (110) or the compiler can optimize data placement so that the massive neural network weights are stored in the external main memory (2000), while key parameters, such as programmable operation coefficients that the function approximator (FuncApx) repeatedly uses to approximate activation functions like GELU and SiLU, reside in the high-speed on-chip shared memory (400) or the internal memory of the neural processing unit (NPU). Since the coefficients are loaded directly from the on-chip memory without needing to access the external memory for every activation function operation, memory bottlenecks in the system can be significantly alleviated and power consumption associated with data movement can be reduced.

[0281] For example, a computation pipeline through NPU-DSP heterogeneous computing can be implemented by the example described above.

[0282] As illustrated in FIG. 16, the system of the present invention can implement a highly specialized heterogeneous computing architecture by integrating a digital signal processing device (DSP, 500) together with a neural processing unit (100). This can improve processing efficiency by breaking down the entire AI inference process into the following three-stage specialized pipeline.

[0283] The three-stage specialization pipeline may include a signal pre / post-processing pipeline, a large integer matrix operation pipeline, and a high-precision non-linear function operation pipeline.

[0284] The signal pre / post-processing pipeline can be a pipeline dedicated to the DSP.

[0285] For example, the signal pre / post-processing pipeline can be configured so that the DSP (500) is dedicated to traditional signal processing operations such as sensor data filtering and Fourier transform (FFT).

[0286] The large integer matrix operation pipeline can be a pipeline dedicated to P_CORE.

[0287] The large-scale integer matrix operation pipeline can be configured so that a low-power, integer operation-based processing core (P_CORE, 120) handles matrix multiplication (MatMul), which accounts for most of the neural network operation volume.

[0288] A high-precision non-linear function computation pipeline can be a pipeline dedicated to SFU / FuncApx.

[0289] A high-precision nonlinear function operation pipeline can be configured such that a function approximation unit (FuncApx) within a special function unit (SFU, 150) that performs autonomous pipeline operations with a single instruction processes complex nonlinear functions requiring floating-point (FP) precision, such as softmax, negative exponents, and reciprocals, at ultra-high speed. This division of roles can optimize the overall processing performance and energy efficiency of the system by allocating hardware optimized for each operation.

[0290] FIG. 17 illustrates a device (10000) according to the first embodiment of the second example of the present disclosure.

[0291] The device (10000) illustrated in FIG. 17 may be a top view of the device (10000), but is not limited thereto. The dimensions of each of the hardware components described with reference to FIG. 17 may refer to width (X-axis length) and depth (Y-axis length).

[0292] An apparatus (10000) according to the first embodiment of the second example of the present disclosure includes a circuit board (9000) and a package (5000). The circuit board (9000) may further include a slot (9100) (e.g., an M.2 slot). An apparatus (10000) according to the first embodiment of the second example of the present disclosure may correspond to an apparatus (10000) according to the embodiments of the first example of the present disclosure. An apparatus (9000) according to the first embodiment of the second example of the present disclosure may correspond to an apparatus (BD) according to the embodiments of the first example of the present disclosure. A package (5000) according to the first embodiment of the second example of the present disclosure may correspond to a package (not shown) in the embodiments of the first example of the present disclosure. Redundant content described with reference to FIGS. 3 through 16 is omitted.

[0293] The device (10000) according to the first embodiment of the second example of the present disclosure may be composed of a system-on-chip (SoC). A system-on-chip (SoC) may refer to a semiconductor package that integrates at least one neural processing unit (100) and various components of an electronic system. A system-on-chip (SoC) may integrate digital circuits, analog circuits, mixed-signal and radio frequency processing circuits into a single package. For example, in addition to the neural processing unit (NPU), the system-on-chip (SoC) may further include at least one of a central processing unit (CPU), a digital signal processor (DSP), an image signal processor (ISP), and a graphics processing unit. A system-on-chip (SoC) may include at least one memory (MEM). A system-on-chip (SoC) may include a high-speed data bus for efficient communication between various circuits included in the system-on-chip (SoC). A system-on-chip (SoC) may include at least one interface, such as PCIe, USB, I2C, SPI, UART, and GPIO, for connection with external devices and sensors. The system-on-chip (SoC) may include an on-chip power management device that regulates voltage and power distribution of the semiconductor package. The system-on-chip (SoC) may include a communication interface that integrates wired or wireless communication protocols, such as Ethernet, Wi-Fi, Bluetooth, and cellular connectivity, for data transmission.

[0294] Referring to FIG. 17, the device (10000) is an example implemented in an M.2 form factor. In the first embodiment of the second example of the present disclosure, the width of the circuit board (9000) may be 22 mm and the depth may be 42 mm, 60 mm, 80 mm, or 110 mm, but is not limited thereto.

[0295] The M.2 board is a next-generation interface standard for high-speed data transmission that offers high speed and efficiency. Designed for a compact size, it is being adopted in various devices such as laptops, desktops, and mini PCs. In particular, small form factors like M.2 make it easy to apply on-device.

[0296] A package (5000) mounted on a device (10000) may include a system-on-chip (SoC) and at least one memory (MEM). The system-on-chip (SoC) according to the first embodiment of the second example of the present disclosure may correspond to the system-on-chip (1000) according to the embodiments of the first example of the present disclosure. The memory (MEM) of FIG. 17 may correspond to the memory (MEM) or main memory (M_MEM) described above with reference to FIG. 9 through 16.

[0297] The physical size of the system-on-chip (SoC) may be, for example, 4.50 mm x 4.85 mm. However, the present disclosure is not limited thereto. Considering the minimum separation distance required between the system-on-chip (SoC) and the memories (MEMs), the physical size of each memory (MEM) may be limited to 6.43 mm x 5.30 mm. However, the present disclosure is not limited to the size or dimensions of each component.

[0298] Meanwhile, the maximum number of memory (MEM) within the package (5000) can be determined by considering the processing capability of the system-on-chip (SoC) (e.g., TOPS) or the size of a package of a specific size or smaller that satisfies the size of a specific form factor applied to the system-on-chip (SoC).

[0299] According to one embodiment, the process node of the semiconductor manufacturing process of the memory (MEM) may be lower than the process node of the semiconductor manufacturing process of the system-on-chip (SoC). For example, the memory (MEM) may be manufactured using a 14nm semiconductor manufacturing process, and the system-on-chip (SoC) may be manufactured using a 7nm semiconductor manufacturing process. For example, the memory (MEM) may be manufactured using a 5nm semiconductor manufacturing process, and the system-on-chip (SoC) may be manufactured using a 7nm semiconductor manufacturing process. For example, the memory (MEM) may be manufactured using a 2nm semiconductor manufacturing process, and the system-on-chip (SoC) may be manufactured using a 5nm semiconductor manufacturing process. According to the above configuration, the size of the system-on-chip (SoC) can be reduced to fit the specifications of a specific form factor (e.g., M.2). In particular, for on-device applications, a large size of the system-on-chip (SoC) can be a barrier to commercialization.

[0300] Referring to FIG. 17, the package (5000) may include up to four memory (MEMs). Each memory (MEM) may be LPDDR5 (Low Power Double Data Rate 5). By increasing the number of bus channels between the system-on-chip (SoC) and the memory (MEMs) to correspond to the number of memory (MEMs), the memory bandwidth of the first bus (310) of the first exemplary embodiments of the present disclosure may be increased. For example, if the system-on-chip (SoC) of the package (5000) and the four memory (MEMs) are placed closest to each other, the size of the package (5000) may be 21 mm x 31 mm. In this embodiment, the package (5000) may be mounted on a device (10000) having a width of 22 mm and a depth of 42 mm.

[0301] However, the package (5000) of FIG. 17 cannot be mounted on a circuit board (9000) having a depth of 30 mm. Therefore, FIG. 18 describes an embodiment in which a package (5000) including a system-on-chip (SoC) and four memories (MEMs) can be mounted on a circuit board (9000) having a depth of 30 mm.

[0302] The above package (5000) may include at least one semiconductor chip to which a fan-in semiconductor package, a fan-out semiconductor package, a package-on-package (PoP), and / or a 2.5D package method is applied. Additionally, the technical features of different examples of the present disclosure may be combined and modified.

[0303] FIG. 18 illustrates a device (10000) according to a second embodiment of a second example of the present disclosure.

[0304] The device (10000) illustrated in FIG. 18 may be a top view of the device (10000), but is not limited thereto. The dimensions of each of the hardware components described with reference to FIG. 18 may refer to width (X-axis length) and depth (Y-axis length).

[0305] An apparatus (10000) according to the second embodiment of the second example of the present disclosure includes a circuit board (9000) and a package (5100). The circuit board (9000) may further include a slot (9100) (e.g., an M.2 slot). An apparatus (10000) according to the second embodiment of the second example of the present disclosure may correspond to an apparatus (10000) according to the embodiments of the first example of the present disclosure. An apparatus (9000) according to the second embodiment of the second example of the present disclosure may correspond to an apparatus (BD) according to the embodiments of the first example of the present disclosure. An apparatus (5100) according to the second embodiment of the second example of the present disclosure may correspond to an apparatus (not shown) according to the embodiments of the first example of the present disclosure. Duplicate descriptions of the above contents with reference to FIGS. 3 to 17 are omitted. In addition, in describing the second embodiment of the second example of the present disclosure, descriptions that overlap with the first embodiment of the second example may be omitted for the convenience of explanation.

[0306] Referring to FIG. 18, the device (10000) may be implemented in an M.2 form factor. In the second embodiment of the second example of the present disclosure, the width of the circuit board (9000) may be 22 mm and the depth may be 30 mm, but is not limited thereto.

[0307] A package (5100) mounted on a device (10000) may include a system-on-chip (SoC) and at least one memory (MEM). A system-on-chip (SoC) according to a second embodiment of the second example of the present disclosure may correspond to a system-on-chip (1000) according to embodiments of the first example of the present disclosure. The memory (MEM) of FIG. 18 may correspond to the memory (MEM) or main memory (M_MEM) described above with reference to FIG. 10 through FIG. 17.

[0308] The width of the circuit board (9000) illustrated in FIG. 18 may be 22 mm and the depth may be 30 mm. In order to mount the package (5100) within such a constrained size, the arrangement of the system-on-chip (SoC) and multiple memory (MEM) within the package (5100) must be improved.

[0309] The physical size of the above system-on-chip (SoC) may be 4.50 mm x 4.85 mm, and the physical size of each memory (MEM) may be 6.43 mm x 5.30 mm.

[0310] According to one embodiment, the process node of the semiconductor manufacturing process of the memory (MEM) may be lower than the process node of the semiconductor manufacturing process of the system-on-chip (SoC). For example, the memory (MEM) may be manufactured using a 14nm semiconductor manufacturing process, and the system-on-chip (SoC) may be manufactured using a 7nm semiconductor manufacturing process. For example, the memory (MEM) may be manufactured using a 5nm semiconductor manufacturing process, and the system-on-chip (SoC) may be manufactured using a 7nm semiconductor manufacturing process. For example, the memory (MEM) may be manufactured using a 2nm semiconductor manufacturing process, and the system-on-chip (SoC) may be manufactured using a 5nm semiconductor manufacturing process. According to the above configuration, the size of the system-on-chip (SoC) can be reduced to fit the specifications of a specific form factor (e.g., M.2).

[0311] Referring to FIG. 18, the system-on-chip (SoC) and the plurality of memory (MEM) can be arranged diagonally within the package (5100). In this case, the size of the package (5100) can be reduced to 21 mm x 21 mm. To elaborate, the configuration of the plurality of semiconductor chips arranged within the package (5100) can be referred to as a “rotated multi-chip substrate layout.” The package (5100) of the rotated multi-chip substrate layout has the effect of reducing the size of the package (5100) due to the arrangement of each semiconductor chip rotated at a specific angle. That is, the X-axis of the system-on-chip (SoC) and the X-axis of the substrate (5100) may not be parallel. The Y-axis of the system-on-chip (SoC) and the Y-axis of the substrate (5100) may not be parallel. The X-axis of the memory (MEM) and the X-axis of the substrate (5100) may not be parallel. The Y-axis of the memory (MEM) and the Y-axis of the substrate (5100) may not be parallel.

[0312] Meanwhile, the number of memory (MEM) in the package (5100) may be changed by taking into account the processing capability of the system-on-chip (SoC) (e.g., TOPS), the size of a specific form factor, and the size that can be placed within a package smaller than the specific form factor.

[0313] The above package (5100) may be configured to include at least one semiconductor chip to which a fan-in semiconductor package, a fan-out semiconductor package, a package-on-package (PoP), and / or a 2.5D package method is applied. Additionally, the technical features of different examples of the present disclosure may be combined and modified.

[0314] That is, the package (5100) may include a semiconductor package substrate, a system-on-chip (SoC) disposed on the semiconductor package substrate and rotated at a first angle relative to one side of the semiconductor package substrate with respect to a plane, and one or more memories (MEMs) disposed on the semiconductor package substrate adjacent to the system-on-chip (SoC) and rotated at a second angle relative to one side of the semiconductor package substrate with respect to a plane. Here, the first angle and the second angle may be determined such that the size of the package (5100) is smaller than the size of a specific form factor.

[0315] That is, according to the device (10000) according to the second embodiment of the second example of the present disclosure, a 'rotated multi-chip substrate layout' is illustrated for mounting a system-on-chip (SoC) and a plurality of memory (MEM) in a very limited form factor such as M.2 2230. In order to efficiently implement sophisticated and computationally intensive artificial intelligence (AI) operations in such small devices where physical space is extremely constrained, an optimized hardware foundation is essential. Various embodiments of the present disclosure can provide such hardware support, and their main features are as follows.

[0316] First, a Neural Processing Unit (NPU) core with highly optimized Power, Performance, and Area (PPA) can be provided. This NPU core includes an integer operation-oriented processing core (P_CORE) and a Special Function Unit (SFU) that handles efficient floating-point (FP) operations; in particular, the SFU may adopt methods such as Piecewise Affine Function (PAF) to support complex activation function processing.

[0317] Second, innovative packaging technology can be applied to integrate the NPU chip and essential memory (e.g., LPDDR series) at high density to facilitate mounting on standard small form factors such as M.2 cards. As exemplified in FIG. 18, the overall module size can be minimized through space-optimized layouts, such as rotating the system-on-chip (SoC) and memory chips within the package.

[0318] Third, a main memory of sufficient capacity may be provided, positioned either inside the package or very close to the main board, to store weight and computation data of the target AI model and to exchange data with the NPU at high speed. This reduces the need for memory expansion via an external large printed circuit board (PCB), contributing to the miniaturization of the entire system.

[0319] As such, the organic combination of the PPA-optimized NPU core, high-density integrated packaging technology (see Fig. 18), and proximity memory configuration creates significant synergy. This enables the effective implementation of sophisticated on-device AI functions (e.g., advanced computations such as speculative decoding) even in mobile devices, wearable devices, or small embedded systems with extremely limited space and power budgets, where it was previously difficult to install AI accelerators.

[0320] In conclusion, the small form factor AI acceleration device according to the present disclosure may include an NPU die comprising an integer processing core and a floating-point special function unit (SFU) with optimized power, performance, and area, and one or more memory dies electrically connected to the NPU die to store weights and computational data of an AI model. The NPU die and the memory dies are densely integrated within a single semiconductor package mountable on a standard small form factor substrate, such as the M.2 specification, as illustrated in FIG. 18, and the relative positions and orientations of the dies within the package may be configured to minimize the total package area. Additionally, the SFU may include a programmable function approximation circuit to implement various activation functions in an area-efficient manner. Through this, the AI ​​acceleration device of the present disclosure can be successfully integrated into portable or embedded devices with severe constraints on physical size and power consumption, thereby supporting the efficient execution of advanced AI computations (e.g., computations such as speculative decoding) in an on-device environment.

[0321] FIG. 19 illustrates a tree for explaining speculative decoding according to the third example of the present disclosure.

[0322] The speculative decoding method according to the third example of the present disclosure can be performed in the apparatus (10000) described with reference to FIGS. 3 to 18. Accordingly, in describing the third example of the present disclosure, descriptions that overlap with the first and second examples may be omitted for convenience of explanation.

[0323] A transformer-based generative AI model can receive a query as input and generate a response. For example, a language model can receive a query in text form, obtain an input token, and generate an output token from the input token. Subsequently, the query and the generated output token can be input back into the language model to generate a subsequent output token. This token generation operation can be repeated multiple times until a response to the query is completed. A token may correspond to a word or a part of a word. The response may include output tokens. The final response generated by the language model for the input query data may include a series of output tokens generated by inputting the output tokens, which were generated as part of the response generated by the language model, back into the language model.

[0324] The larger the number of parameters in a language model, the higher the cost of generating a response. Therefore, using a language model with a large number of parameters increases the computational cost required to generate a response. Reducing the number of parameters can lower computational costs and improve response speed, but it may compromise response accuracy. In other words, there is a trade-off relationship between the number of parameters and accuracy (or quality) of a language model and the time required for the processor to compute the language model. In particular, to generate a response of a certain quality or higher, the number of parameters in the language model must be significantly large.

[0325] On the other hand, providing usability services by processing language models on the device itself presents difficulties when considering the device's hardware resource limitations (e.g., energy consumption, battery capacity, memory capacity, AI processing capabilities, etc.).

[0326] Accordingly, the third example of the present disclosure describes speculative decoding (or speculation decoding) for running a heterogeneous language model by utilizing the limited hardware resources of the device (10000).

[0327] Speculative decoding is a technique in which multiple language models each perform a prediction and then combine the results to generate the most reliable answer. For example, the first language model selects candidates for the next token and calculates a probability representing the goodness of fit for each candidate token. Subsequently, the second language model can make a final selection of one of the candidate tokens suggested by the first language model by considering the context. Since the second language model must select one of the candidate tokens by reflecting a broader context, it may be a model with more parameters and superior performance than the first language model.

[0328] An apparatus (10000) according to the examples of the present disclosure may operate heterogeneous language models based on at least one system-on-chip (1000). That is, in an on-device manner, speculative decoding may utilize heterogeneous language models to accelerate the token generation speed of the language models. At least one apparatus (10000) may utilize a speculative decoding technique in which heterogeneous language models cooperate with each other to generate a response to an input query.

[0329] A heterogeneous language model according to the third example of the present disclosure may include a first language model and a second language model. Here, the number of parameters of the first language model is less than the number of parameters of the second language model. That is, the number of parameters of the heterogeneous language models is different from each other. A language model with a relatively small number of parameters and a language model with a large number of parameters can be defined as a supermodel and a submodel, respectively. That is, the first language model may be defined as a submodel of the second language model, and the second language model may be defined as a supermodel of the first language model. For example, the number of parameters of the supermodel may be 7 billion, and the number of parameters of the submodel may be 1 billion.

[0330] A sub-model according to the third example of the present disclosure may generate probabilities used for sampling additional tokens based on speculatively generated additional tokens and a currently allowed set of tokens. A super-model according to the third example of the present disclosure generates output tokens using candidate tokens generated by the sub-model. The super-model may accept or reject candidate tokens by sampling the additional tokens speculatively generated by the sub-model. Here, acceptance or rejection by the super-model may be determined by comparing the probability generated by the super-model corresponding to each token with the probability generated by the sub-model corresponding to each token. A candidate token accepted by the super-model may be an output token included in the response.

[0331] According to the speculative decoding according to the third example of the present disclosure, a submodel may generate one or more sets of candidate tokens to generate a response corresponding to a query input to the submodel. A set of candidate tokens is a set containing one or more candidate tokens.

[0332] Candidate Token Candidate Token Set apple, is, red, banana, yellow, car, fast set1: [apple, is, red]set2: [banana, is, yellow]set3: [car, is, fast]

[0333] The upper model may sample for one or more candidate token sets and accept or reject the corresponding candidate token sets. Here, if the device (10000) samples for each candidate token set, the device (10000) can generate more tokens faster than if it samples for each token. Additionally, the probability distributions of the lower model and the upper model can be maintained similarly. Here, the token generation speed may be referred to as TPS (tokens per second). The speculative decoding according to the third example of the present disclosure may operate as an automatic regression token generation method. The automatic regression token generation method generates tokens by inputting a series of tokens generated by the lower model back into the lower model. The lower model generates one token for each automatic regression. Therefore, when the lower model performs N inferences, N series of tokens are generated. The lower model generates a conditional probability distribution related to the N series of tokens. Subsequently, the upper model processes the N series of tokens generated by the lower model and the conditional probability distribution related to the N series of tokens to generate the probability distribution of the upper model. Therefore, the parent model can accept or reject tokens generated by the child model by comparing the probability distribution generated by the parent model with the conditional probability distribution generated by the child model. The parent model may accept tokens based on a threshold. A token is accepted when the similarity between the probability distribution value of the parent model and the conditional probability distribution value of the child model is equal to or greater than the threshold, and the token may be rejected when the similarity is less than the threshold.

[0334] The upper model can accept or reject a series of tokens generated by the lower model based on a threshold. If a specific token among the series is rejected, the tokens preceding the rejected token become the final output tokens. For example, the lower model of the guess decoding can generate tokens in sets. In this case, the upper model can accept or reject multiple sets of tokens generated by the lower model. If the lower model generates candidate tokens in sets, the acceptance or rejection judgment in the upper model can also be performed in sets, which can improve the token generation speed of the guess decoding of the device (10000). Additionally, as the number of token sets generated by the lower model increases, the probability of the token set that the upper model will accept can increase.

[0335] The sub-model generates one or more candidate token sets in response to an input query. Each candidate token set contains a series of tokens, and the sub-model may select a series of tokens with a high probability in a probability distribution as the candidate token set. Each candidate token set may be selected in various ways. For example, each candidate token set may be generated by selecting tokens with the highest probability within a probability distribution, but the present disclosure is not limited thereto. For the super-model to accept a single candidate token set, the probabilities of the tokens included in each candidate token set may be summed. Therefore, the super-model may accept or reject the entire candidate token set based on the summed probability of the candidate token set, rather than the probability of each individual token. To elaborate, if the probability of each individual token is utilized, the probability of rejection by the super-model increases because it may be difficult to match the probabilities of each token generated by the sub-model and the super-model. If the summed probability of the token set is utilized, the probability of acceptance by the super-model increases because the matching of the summed probabilities of the token set generated by the sub-model and the super-model may be more similar.

[0336] The sub-model can generate one or more candidate token sets in response to an input query. The one or more candidate token sets can be generated statically or dynamically. For example, the sequence length of each candidate token set can be pre-set. The sub-model can generate one or more candidate token sets and combine each set to generate tree data. The tree data includes a node at each branch point.

[0337] FIG. 19 illustrates tree data (TD) comprising one or more token sets generated in a sub-model according to one embodiment. Referring to FIG. 19, a first token set (S1), a second token set (S2), a third token set (S3), a fourth token set (S4), and a fifth token set (S5) are shown below the leaf nodes of the tree.

[0338] A first candidate token set (S1) may include four tokens (a, b, c, d). A second candidate token set (S2) may include four tokens (a, b, c, e). A third candidate token set (S3) may include four tokens (a, f, g, h). A fourth candidate token set (S4) may include four tokens (a, f, g, i). A fifth candidate token set (S5) may include four tokens (a, f, j, k). Each of the candidate token sets may be selected or rejected by the upper model.

[0339] The token length of each candidate token set can be set. For example, the sentence length of each token set can be set to 4. However, the present disclosure is not limited thereto, and the length of the token set may be changed dynamically.

[0340] Each token may have an order level. An order level may correspond to a sentence length. Specifically, the first order level (L1) corresponds to token (a). The second order level (L2) corresponds to tokens (b, f). The third order level (L3) corresponds to tokens (c, g, j). The fourth order level (L4) corresponds to tokens (d, e, h, i, k). In an example where the sentence length is set to 4, the maximum order level is set to 4. The order of the tokens in each set of tokens may be arranged according to the corresponding order level. Specifically, among the tokens (a, b, c, d) of the first set of tokens (S1), the first token (a) corresponds to the first order level (L1). Among the tokens (a, b, c, d) of the first set of tokens (S1), the second token (b) corresponds to the second order level (L2). The third token (c) among the tokens (a, b, c, d) of the first set of tokens (S1) corresponds to the third order level (L3). The fourth token (d) among the tokens (a, b, c, d) of the first set of tokens (S1) corresponds to the fourth order level (L4).

[0341] Each token is defined as a node of tree data (TD). For example, a token (g) can be a branch node of the third set of tokens (S3) and the fourth set of tokens (S4) at the third order level (L3). Each node contains branch information.

[0342] That is, the tree data (TD) generated by the sub-model may include one or more sets of tokens (S1 to S5), order levels (L1 to L4), information on the nodes (a, b, c, d, e, f, g, h, i, j, k) of each token, and all or at least part of the selection probability of each candidate set of tokens (e.g., the summed probability of the sets of tokens). The super-model may accept or reject at least one token of the tree data (TD).

[0343] Speculative decoding can also be implemented as hybrid speculative decoding. In hybrid speculative decoding, tokens previously generated speculatively are verified by a higher-level model, and a threshold α can be set to determine whether a token is approved or rejected by comparing the probability distributions of the lower-level model and the higher-level model.

[0344] Speculative decoding can also be implemented as group speculative decoding. Group speculative decoding is configured so that multiple tokens are generated in groups from the lower model and validated by the upper model, and can provide a high level of efficiency.

[0345] FIG. 20 schematically illustrates a speculative decoding operation according to a third example of the present disclosure.

[0346] Referring to FIG. 20, a lower model (LM) and a higher model (UM) are illustrated. An apparatus (10000) according to the third example of the present disclosure may be configured to sequentially process the lower model (LM) and the higher model (UM) for speculative decoding.

[0347] Transformer-based AI models can perform specific computational steps (e.g., prefill and generation steps) to generate a response to an input query. A transformer is an AI model based on an attention mechanism. Transformers utilize numerous matrix multiplication operations. Using input values ​​and parameters such as the query (Q), key (K), and value (V), a transformer can obtain an output value, the attention score (Q, K, V). Based on the output value (i.e., attention (Q, K, V)), the transformer can process various inference operations. Transformers are actively utilized in generative language models.

[0348] A tensor refers to a multidimensional matrix parameter processed in an artificial intelligence model (e.g., an artificial neural network). Tensors can refer to various parameters of a neural network model, such as weights, input parameters, output parameters, and attention parameters.

[0349] The prefill stage is an initial step for processing and interpreting input (e.g., a prompt entered by a user). The prefill stage may include a tokenization stage, a context embedding stage, a processing stage through layers, and a preparation stage for decoding. The prefill stage may be processed by one or more neural processing units. In the tokenization stage, the input text is converted into tokens (sub-words, words, or characters), which are numerical representations used by the AI ​​model. In the context embedding stage, tokens can be mapped to high-density embedding vectors that encode semantic and syntactic information. In the prefill stage, the embeddings are processed through multiple layers of the neural network model (e.g., Transformer blocks) to generate contextual representations of the input tokens. In the preparation stage for decoding, the AI ​​model calculates the probability of the next output token for each token based on the input context, but does not yet finally generate the next token. Instead, it can set the context for the generation stage.

[0350] In the generation phase, one or more neural processing units may sequentially generate one or more output tokens based on the context provided by the AI ​​model in the prefill phase and previously generated output tokens. The generation phase includes an initial prediction phase, an auto-regression phase, and a stopping condition. The generation phase may be processed by one or more neural processing units. In the initial prediction phase, the first token is generated by sampling from the probability distribution calculated in the prefill phase.

[0351] In the autoregressive processing step, each subsequent token is generated by integrating the previous token (including the generated token) into the context of the neural network model. The generation process continues until stopping conditions are met, such as reaching a maximum token limit, the occurrence of a special sequence end token (e.g., end of sequence (EOS)), or the achievement of a specific semantic goal. The series of tokens generated in the generation step may correspond to the tree data (TD) of FIG. 19. The tree data may include at least one set of tokens.

[0352] Referring again to FIG. 20, a third example illustrating the steps of speculative decoding in chronological order is illustrated. Speculative decoding according to the examples of the present disclosure may be processed sequentially or in parallel by one or more neural processing units. In speculative decoding, a higher-level model (UM) and a lower-level model (LM) cooperate with each other to generate a response to an input query. The longer the horizontal length of the block representing each step, the longer the processing time required.

[0353] Specifically, in speculative decoding, the upper model (UM) first performs a prefill operation. Subsequently, the lower model (LM) performs a prefill operation. Then, the lower model (LM) sequentially performs generation steps, and each token is generated in each generation step. Next, the upper model (UM) accepts or rejects the tokens generated by the lower model (LM). The upper model (UM) may accept or reject some or all of the tokens generated by the lower model (LM). The lower model (LM) receives the accepted tokens as input queries, and the lower model (LM) sequentially performs generation steps to generate each token corresponding to each generation step. Next, the upper model (UM) accepts or rejects the tokens generated by the lower model (LM). The above steps are repeated until stopping conditions are satisfied.

[0354] The number of parameters in the upper model (UM) is greater than that of the lower model (LM), resulting in a longer processing time. Consequently, the processing time of the upper model (UM)'s prefill phase is longer than that of the lower model (LM). The processing time of the upper model (UM)'s acceptance or rejection phase is longer than that of the lower model (LM)'s generation phase. Therefore, regarding token generation for creating a response, the system's token generation speed can be improved by configuring the system so that the lower model generates as many tokens as possible and the upper model performs only selective decisions.

[0355] The number of parameters for each of the lower model (LM) and the upper model (UM) can be determined by taking into account the hardware resource limitations of the device (10000) (e.g., the processing performance of the system-on-chip (SoC) and the bandwidth of the memory (MEM). The upper model (UM) can utilize a language model in which the number of parameters is determined by taking into account the hardware resource limitations of the device being processed.

[0356] In some examples, the lower model (LM) may be a model obtained by pruning the parameters of the upper model (UM). In such cases, the parameters of the pruned neural network model can be lightweight, and the required memory bandwidth may be smaller than that of the upper model (UM).

[0357] According to a third example of the present disclosure, the device (10000) may be configured to process a prefill operation using an upper model (UM) to process speculative decoding. Subsequently, the device (10000) may process a prefill operation and a generation operation using a lower model (LM) to process speculative decoding.

[0358] A device (10000) according to the third example of the present disclosure may include at least one interface (not shown). The interface may input and / or output data. The device (10000) may receive an input query through the interface (not shown). The interface may be an electronic circuit capable of receiving an input query such as a prompt, gesture, or voice. The input and / or output interface may include, for example, a mouse, keyboard, touchpad, touchscreen, microphone, wireless communication device, wired communication device, camera, speaker, etc.

[0359] The system-on-chip (SoC) of the device (10000) according to the third example of the present disclosure can perform inference operations on an upper model (UM) and a lower model (LM). The upper model (UM) and the lower model (LM) can cooperate with each other to process speculative decoding. The neural processing unit (NPU) of the system-on-chip (SoC) of the device (10000) according to the third example of the present disclosure can process the upper model (UM) and the lower model (LM) alternately. That is, the neural processing unit (NPU) operates to process the upper model (UM) and the lower model (LM) sequentially. In other words, the neural processing unit (NPU) operates to process the upper model (UM) and the lower model (LM) in a time-sharing manner. Thus, the upper model (UM) and the lower model (LM) can be processed sequentially while increasing the utilization rate of the computational circuits of the neural processing unit (NPU).

[0360] According to the disclosed embodiment, speculative decoding is performed by sequentially processing the upper model (UM) and the lower model (LM) in the neural processing unit (NPU) of the device (10000), and the parameters of the upper model (UM) and the lower model (LM) reside in memory (MEM), thereby reducing the bottleneck caused by the speculative decoding operation.

[0361] The memory (MEM) of the device (10000) according to the third example of the present disclosure is configured to store all weight parameters of the upper model (UM) and the lower model (LM) when the device (10000) is operating. By storing all first weight parameters of the upper model (UM) and second weight parameters of the lower model (LM) in the memory (MEM), the device (10000) can efficiently process speculative decoding. The memory (MEM) of the device (10000) according to the third example of the present disclosure can provide the parameters of the upper model (UM) to the neural processing unit (NPU) when the neural processing unit (NPU) infers the upper model (UM), and provide the parameters of the lower model (LM) to the neural processing unit (NPU) when the neural processing unit (NPU) infers the lower model (LM).

[0362] For example, the device (10000) can set an address map of memory (MEM) during a speculative decoding operation to allocate a memory bank in which weight parameters of the upper model (UM) and the lower model (LM) are stored. Thus, during a speculative decoding operation, weight parameters of both the upper model (UM) and the lower model (LM) can reside in memory (MEM). That is, the capacity of memory (MEM) of the device (10000) according to the third example of the present disclosure may be large, as the total capacity occupied by the first weight parameters of the upper model (UM) and the second weight parameters of the lower model (LM) may be large. If the memory (MEM) of the device (10000) does not have a memory capacity capable of storing the upper model (UM) and the lower model (LM), it may be difficult to process speculative decoding quickly in real time. That is, the capacity of the memory (MEM) can be determined by considering the size of the parameters of the upper model (UM) and the lower model (LM) to be operated in the device (10000). If the parameters of the upper model (UM) and the lower model (LM) are not all stored in the memory (MEM), the conventional device must read the weight parameters of each model from an external storage device for speculative decoding. In this embodiment, the conventional device must command unnecessary memory read and memory write operations for speculative decoding, so the token generation speed may be lower than the token generation speed of the device (10000) according to the third example of the present disclosure.

[0363] A first memory (MEM1) of a device (200) according to a second embodiment of the first example of the present disclosure is configured to store weight parameters of a higher model (UM) and tokens generated from the higher model (UM) when the device (200) is operating.

[0364] The second memory (MEM2) of the device (200) according to the second embodiment of the first example of the present disclosure is configured to store weight parameters of the lower model (LM) and tokens generated from the lower model when the device (200) is operating. When the weight parameters of the upper model (UM) and the lower model (LM) are all stored in the first memory (MEM1) and the second memory (MEM2), the device (200) can efficiently process speculative decoding. To elaborate, the size of the weight parameters of the upper model (UM) and the lower model (LM) may be considerably large and may be difficult to store in a single memory chip. In such cases, the device (200) may configure multiple memories to increase the memory capacity.

[0365] A first memory (MEM1) of a device (200) according to a second embodiment of a first example of the present disclosure is configured to provide parameters of a higher model (UM) to a neural processing unit (NPU) when the neural processing unit (NPU) infers a higher model (UM), and a second memory (MEM2) is configured to provide parameters of a lower model (LM) to a neural processing unit (NPU) when the lower model (LM) is inferred.

[0366] For example, the device (200) can set the address maps of the first memory (MEM1) and the second memory (MEM2), respectively, during a speculative decoding operation to allocate memory banks where weight parameters of the upper model (UM) and the lower model (LM) are stored. Thus, during a speculative decoding operation, weight parameters of the upper model (UM) and the lower model (LM) can reside in both the first memory (MEM1) and the second memory (MEM2).

[0367] Accordingly, the capacity of the plurality of memories (MEM1, MEM2) of the device (200) according to the second embodiment of the first example of the present disclosure is configured to be greater than the total capacity occupied by the weight parameters of the upper model (UM) and the lower model (LM). Thus, the weight parameters of the upper model (UM) and the lower model (LM) can reside in the plurality of memories (MEM1, MEM2). If the capacity of the upper model (UM) and the lower model (LM) in the plurality of memories (MEM1, MEM2) of the device (200) is insufficient, it may be difficult to process speculative decoding efficiently.

[0368] If the parameters of the upper model (UM) and the lower model (LM) are not all stored in multiple memories (MEM1, MEM2), the conventional device must read the weight parameters of each model from an external storage device for speculative decoding. Consequently, the conventional device commands unnecessary memory read and memory write operations for speculative decoding. As a result, the token generation speed of the conventional device is reduced compared to the token generation speed of the device (200) according to the second embodiment of the first example of the present disclosure.

[0369] In summary, FIGS. 19 and 20 illustrate speculative decoding according to a third example of the present disclosure. Furthermore, the speculative decoding method according to a third example of the present disclosure can be performed in the apparatus (10000) described with reference to FIGS. 3 through 18. To support such efficient on-device speculative decoding, the apparatus (10000) according to the present disclosure may be based on the following optimized hardware architecture and system configuration.

[0370] That is, the device (10000) according to the present disclosure includes a low-power NPU (neural processing unit, see 100 in FIG. 3) core architecture, and the low-power NPU core architecture has a separate structure of an integer operation-centered processing core (P_CORE, see 120 in FIG. 3) and a special function unit (SFU, see 150 in FIG. 3) for floating-point operations and complex function processing. The special function unit (SFU, 150) is configured to include dedicated hardware accelerators for functions such as exponential and reciprocal, which are particularly essential for Softmax and attention operations (e.g., NEXP (see FIG. 30), RECIP (see FIG. 28) circuits included in the vector function approximation unit (V_FuncApx, see 1537 in FIG. 5) or scalar function approximation unit (S_FuncApx, see 1545 in FIG. 6) within the SFU (150)), and programmable function approximation (PAF) units that support various activation functions (e.g., function approximation circuits included in the function approximation unit (see FIG. 24, FIG. 25)).

[0371] Accordingly, the device (10000) includes a large-capacity on-device memory system, said memory system includes a main memory (e.g., external DRAM, see 2000 in FIG. 9 or 2000-1 to 2000-N in FIG. 10) with a capacity sufficient to store all weights of the LLM and weights of an auxiliary model that can be used for speculative decoding, and includes an on-chip shared memory (e.g., SRAM, see 400 in FIG. 11 or 400-1 to 400-N in FIG. 12) with a capacity sufficient for high-speed access and reuse of frequently used weights and intermediate activation values. Accordingly, said memory system may have a hierarchical structure.

[0372] Therefore, NPU acceleration-based speculative decoding can be effectively implemented through the optimized hardware configuration of the NPU core (100) and memory system. According to the above configuration, the optimized NPU core (100), rather than being a simple sum of individual technologies, is organically combined with a large-capacity memory system (e.g., 2000, 400) and a speculative decoding algorithm to provide a synergistic effect that enables large-scale language models, which previously required server-level computing power, to be executed in a fully on-device form without response delay while minimizing energy consumption. This enables the implementation of a practical server-independent AI agent.

[0373] In summary, a low-power on-device large-scale language model (LLM) inference system according to the present disclosure can achieve the above effect by including the following components: a processing core (120) that performs integer-based matrix operations on input data; a special function unit (SFU, 150) that performs floating-point-based special operations (including exponentiation, reciprocal, and programmable function approximation) on the result of the integer operations, wherein the special operations are optimized for the attention and activation function processing of the LLM; a main memory (2000) that stores the total weights of the LLM and at least one auxiliary model weight for speculative decoding; and an on-chip shared memory (400) that temporarily stores data frequently accessed by the processing core (120) or the SFU (150) to increase data reusability. A controller (e.g., a controller (110) within an NPU (100) or a system-level control unit including the same) that controls the processing core (120), the SFU (150), the main memory (2000), and the on-chip shared memory (400) to perform speculative decoding operations including candidate token generation using the auxiliary model and candidate token verification using the LLM. Through this system configuration, it is possible to complete the inference of the LLM at low power without the assistance of an external server.

[0374] The fourth example of the present disclosure is described below. The fourth example of the present disclosure describes the circuit configuration and operation method of the vector function approximation unit (V_FuncApx) and the scalar function approximation unit (S_FuncApx) illustrated in FIGS. 5 and 6.

[0375] FIG. 21 is a flowchart illustrating an activation function programming method according to the first embodiment of the fourth example of the present disclosure.

[0376] With reference to FIG. 21, the operation of the vector function approximation unit (1537) and the scalar function approximation unit (1545) is described. The vector function approximation unit (1537) and the scalar function approximation unit (1545) are configured to process a programmed activation function to be described later.

[0377] The activation of a neural network model can be determined based on the transmission strength of signals transmitted between layers by an activation function. The inference accuracy of the neural network model may vary depending on the activation function implementation technology of the neural processing unit (100) according to the fourth example of the present disclosure. That is, the inference accuracy and efficiency of the neural network model may vary depending on the hardware implementation characteristics of the processing circuit of the activation function of the neural processing unit (100).

[0378] Accordingly, the activation function programming method according to the first embodiment of the fourth example of the present disclosure includes the step of generating segment data for segmenting an activation function (S210), the step of segmenting the activation function into a plurality of segments using the generated segment data (S220), and the step of approximating at least one of the plurality of segments as a programmable segment (S230).

[0379] The activation function programming method can be performed by an activation function conversion program (not shown). The activation function conversion program (not shown) may be a computer program, software, firmware, application, or executable code stored on a recording medium. The activation function conversion program (not shown) may be a computer program configured to execute instructions that convert an activation function into a programmed activation function expressed by programmable parameters (i.e., operation coefficients). The activation function conversion program (not shown) may be stored on a computer-readable recording medium. The activation function conversion program (not shown) may be included in a compiler (not shown).

[0380] Segment data is generated in step S210. Segment data is data generated to segment a specific activation function into multiple segments. Segment data will be described later.

[0381] In step S220, the activation function is segmented into multiple segments using the generated segment data. In this disclosure, the term 'segment' refers to a part of an activation function divided into multiple intervals and is distinguished from 'candidate segment' or 'programmable segment,' terms related to the approximation of the activation function.

[0382] Step S220 may include a step of determining the number and width of a plurality of segments based on segment data. In Step S220, the number of segments segmenting the activation function to be transformed and the width of each of the plurality of segments may be determined using the segment data. At least one of the plurality of segments may have a width that is the same as or different from that of another segment.

[0383] The interval of multiple segments can also be represented by the coordinates of the start and end points on the X-axis. On the other hand, it should be understood that once the number of multiple segments and the width of each of the multiple segments are determined, the coordinates of the interval of the multiple segments can be obtained using the number and width of the multiple segments.

[0384] In step S230, at least one of the multiple segments is approximated as a programmable segment. The programmable segment can be programmed according to the circuit design of the vector function approximation unit (1537) and the scalar function approximation unit (1545). That is, the activation function transformation program (not shown) can be configured to program the activation function to be processed by the neural processing unit (100) based on the hardware configuration of the vector function approximation unit (1537) and the scalar function approximation unit (1545). The programmable parameter may be data generated in step S230. The programmable parameter may be configured to have a form compatible with the circuits of the vector function approximation unit (1537) and the scalar function approximation unit (1545) of the neural processing unit (100). For example, each of the vector function approximation unit (1537) and the scalar function approximation unit (1545) may be configured to have hardware configured to perform operations on each programmable segment with a specific gradient and a specific offset. The activation function transformation program (not shown) may be configured to receive configuration information of the vector function approximation unit (1537) and the scalar function approximation unit (1545).

[0385] In this case, an activation function transformation program (not shown) can program a programmable segment in the form of a first-order function or a second-order function or higher having a slope and an offset. For example, the programmable segment may be approximated as a first-order function according to a specific judgment criterion. In this case, the activation function transformation program (not shown) can generate a programmable segment expressed in the form of '(slope a) * (input value x) + (offset b)'. The aforementioned specific slope and specific offset may be programmable parameters. For a programmable segment determined to be approximated as a first-order function, step S230 may include the step of approximating a selected segment with specific slope and specific offset values.

[0386] To elaborate, steps S220 and S230 can effectively be performed as a single step. This is because the step of segmenting the segments and the step of generating programmable parameters for the corresponding programmable segments can be performed simultaneously. To elaborate, steps S220 and S230 can be modified to perform the step of segmenting an activation function into multiple segments using the generated segment data, and approximating at least one of the multiple segments as a programmable segment.

[0387] Meanwhile, the above activation function programming method, in particular the process of generating optimal segment data (e.g., boundary values ​​and widths of segments) and deriving optimal programmable parameters for approximating each segment, can be performed in a systematic and automated manner by substituting it into a learning problem of a deep learning model according to another embodiment of the present disclosure.

[0388] Conventionally, the process of finding optimal approximation parameters relied on empirical or complex numerical analysis, making it difficult to guarantee optimal approximations for all nonlinear functions. However, the present disclosure utilizes a small-scale artificial neural network optimized for representing a piecewise linear function. Through this artificial neural network, training can be performed to most precisely mimic the target activation function to be approximated.

[0389] During the learning process, the artificial neural network optimizes internal weights and biases in a direction that minimizes the loss function, and in this process, the optimal segmentation point (bend point) that best represents the non-linear characteristics of the target function and the linear relationship between each segment are automatically determined. Once learning is complete, a set of programmable parameters can be finally extracted from the optimized artificial neural network parameters, including segment boundary values ​​(SB) that can be directly used by the programmable function approximation circuit (PAF), the gradient (A) and offset (B) of each segment. Through this deep learning-based approach, as shown in FIG. 23, it is possible to automatically derive optimal segments with varying widths and numbers depending on the characteristics of the function and generate the most accurate approximation parameters for them.

[0390] FIG. 22 is a graph showing the process of approximating an activation function by an activation function programming method according to the first embodiment of the fourth example of the present disclosure.

[0391] The activation function illustrated in FIG. 22(a) is segmented into multiple segments (s1, s2, s3, s4) using segment data as illustrated in FIG. 22(b). The multiple segments (s1, s2, s3, s4) are approximated into programmable segments (a1x+b1, a2x+b2, a3x+b3, a4x+b4) as illustrated in FIG. 22(c). Here, an example is described in which an activation function transformation program (not shown) generates programmable parameters such that all programmable segments correspond to a linear function.

[0392] Each programmable segment includes a corresponding programmable parameter. In FIG. 22 (c), all of the multiple segments are approximated as programmable segments in the form of a first-order function. However, in various examples, some of the multiple segments may be approximated as programmable segments of a different form. For example, an activation function transformation program (not shown) can program each programmable segment in the form of a first-order function, a second-order function, a third-order function, a logarithmic function, etc.

[0393] For example, only segments (s1, s3, s4) are approximated as programmable segments, and segment (s2) can be approximated using various methods available on the device where the activation function is processed. Specifically, if a predetermined and stored lookup table, non-linear approximation formula, etc. are available in the hardware for the interval of segment (s2), segment (s2) can be approximated using such a predetermined and stored lookup table, non-linear approximation formula, etc.

[0394] In other words, the activation function transformation program (not shown) can be configured to program each segment (s1, s2, s3, s4) independently. In this case, the activation function transformation program (not shown) receives hardware configuration information of the vector function approximation unit (1537) and the scalar function approximation unit (1545). The activation function transformation program (not shown) can be configured to independently determine the approximation method for each of the segments (s1, s2, s3, s4) based on the hardware configuration information of the vector function approximation unit (1537) and the scalar function approximation unit (1545).

[0395] For example, the vector function approximation unit (1537) and the scalar function approximation unit (1545) may each be designed to include at least one of a linear function approximation circuit, a quadratic function approximation circuit, a reciprocal function approximation circuit, a root mean square function approximation circuit, and a negative exponential function approximation circuit. In this case, the activation function transformation program (not shown) may optionally program at least one of each segment (s1, s2, s3, s4) in the form of a linear function, a quadratic function, a reciprocal function, a root mean square function, and a negative exponential function.

[0396] For example, if the vector function approximation unit (1537) and the scalar function approximation unit (1545) are each configured to include a circuit configured to support at least one specific function operation, the activation function transformation program (not shown) can program each segment (s1, s2, s3, s4) into the form of a corresponding specific function.

[0397] For example, the vector function approximation unit (1537) and the scalar function approximation unit (1545) may be configured to include at least one of a first function operation circuit, a second function operation circuit, an inverse function operation circuit, a root mean square function operation circuit, and a negative exponential function operation circuit or a similar function operation circuit designed in hardware.

[0398] For example, an activation function transformation program (not shown) can program the same activation function in different ways.

[0399] For example, an activation function transformation program (not shown) can program a specific activation function as only a first-order function.

[0400] For example, an activation function transformation program (not shown) can program a specific activation function only as a quadratic function.

[0401] For example, an activation function transformation program (not shown) can program a specific activation function only as an inverse function.

[0402] For example, an activation function transformation program (not shown) can program a specific activation function only as a root mean square function.

[0403] For example, an activation function transformation program (not shown) can program a specific activation function as a negative exponent function only.

[0404] For example, an activation function transformation program (not shown) can program each of multiple segments of a specific activation function with a corresponding approximation function.

[0405] For example, an activation function transformation program (not shown) can program multiple segments of a specific activation function into a set of approximation functions of different formulas.

[0406] For example, the activation function programmed to be processed in the vector function approximation unit (1537) and the scalar function approximation unit (1545) according to the first embodiment of the fourth example of the present disclosure may include at least one of Swish, Mish, Sigmoid, Hyperbolic Tangent (Tanh), Scaled Exponential Linear Unit (SELU), Gaussian Error Linear Unit (GELU), Softplus, Rectified Linear Unit (ReLU), Leaky ReLU, Maxout, Exponential Linear Unit (ELU), Hard Sigmoid, Hard Tanh, Bent Identity, ArcTan, Softsign, Sinusoidal Activation, S-shaped Rectified Linear Unit (SReLU), Parametric Rectified Linear Unit (PReLU), Logistic Activation, Radial Basis Function (RBF), Binary Step, Reciprocal, Root mean square, Reciprocal root mean square, and Negative exponential. Each programmed activation function can be converted into a programmable parameter and provided to a neural processing unit (100).

[0407] FIG. 23 is a graph showing various cases of segmenting an activation function into a plurality of segments by an activation function programming method according to the first embodiment of the fourth example of the present disclosure.

[0408] Referring to Fig. 23 (a), a specific activation function can be segmented with a uniform width and four segments.

[0409] Referring to Fig. 23(b), a specific activation function can be segmented with four segments and different widths.

[0410] Referring to Fig. 23 (c), a specific activation function can be segmented with four segments and different widths.

[0411] Referring to Fig. 23 (d), a specific activation function can be segmented with different widths and six segment numbers.

[0412] The number of multiple segments and the width of each of the multiple segments can be determined using segment data.

[0413] An activation function transformation program (not shown) may be configured to analyze the nonlinearity of an activation function so that multiple segments are segmented with different widths. However, the present disclosure is not limited thereto.

[0414] An activation function transformation program (not shown) may be configured to analyze the nonlinearity of the activation function so that each of the multiple segments is segmented to an optimal width. However, the present disclosure is not limited thereto.

[0415] In the present disclosure, the activation function may be implemented in various forms including characteristic intervals. When segmenting the activation function into multiple segments, the number and width of the multiple segments may be determined in various ways depending on the various forms of the activation function.

[0416] For example, various activation functions such as the swish function, Mish function, sigmoid function, hyperbolic tangent (tanh) function, SELU function, GELU (Gaussian Error Linear Unit) function, SOFTPLUS function, ReLU function, Leaky ReLU function, Maxout function, ELU function, etc., have various forms that are divided into multiple characteristic intervals including (actually) linear intervals and / or non-linear intervals. Accordingly, when approximating a non-linear activation function to be processable in hardware, segmenting by considering these characteristic intervals—that is, determining the number and width of segments by considering (actually) linear intervals, non-linear intervals, etc.—allows for more efficient approximation of the activation function corresponding to the characteristics of each activation function.

[0417] FIG. 24 is a diagram showing a function approximation circuit of a neural processing unit configured to process a programmed activation function according to the second embodiment of the fourth example of the present disclosure.

[0418] The programmable function approximation circuit (PAF) according to the second embodiment of the fourth example of the present disclosure illustrated in FIG. 24 may correspond to the vector function approximation circuit (V_PAF) of the vector function approximation unit (1537) of FIG. 5 and the scalar function approximation circuit (S_PAF) of the scalar function approximation unit (1545) of FIG. 6. For convenience of explanation, the vector function approximation circuit (V_PAF) and the scalar function approximation circuit (S_PAF) will be collectively referred to as the programmable function approximation circuit (PAF) to describe the second embodiment of the fourth example of the present disclosure.

[0419] Specifically, the programmable function approximation circuit (PAF) according to the second embodiment of the fourth example of the present disclosure illustrated in FIG. 24 is an example of a circuit configured to process an activation function programmed as a first-order function.

[0420] A programmable function approximation circuit (PAF) configured to process an activation function programmed as a first-order function may be configured to include a plurality of comparators (COMP0 to COMP(N-2)), a selector (SLT), a multiplier (PAF MUL), and an adder (PAF ADD). However, the present disclosure is not limited thereto, and it is also possible to configure the circuit in various ways to distinguish the regions of each segment. Furthermore, the programmable function approximation circuit (PAF) may be modified to include additional circuit configurations to process the activation function using programming methods other than a first-order function.

[0421] The coordinates of the start and end points of the intervals of multiple segments are defined by segment boundary values ​​(SB). That is, the intervals of multiple segments can each be determined by segment boundary values ​​(SB0 to SB(N-2)). For example, to define the intervals of N segments, N-1 segment boundary values ​​(SB0 to SB(N-2)) may be required.

[0422] Each of the multiple comparators (COMP0 to COMP(N-2)) compares the input value (X) with each of the multiple segment boundary values ​​(SB0 to SB(N-2)). Accordingly, the interval of the segment to which the input value (X) belongs among the intervals of the multiple segments can be determined through the comparison values ​​output from each of the multiple comparators (COMP0 to COMP(N-2)). The output values ​​output from each of the above-described multiple comparators (COMP0 to COMP(N-2)) can be referred to as interval determination data (SDD). The selector (SLT) outputs the slope (A) for the programmable segment corresponding to the interval of the segment to which the input value (X) belongs among the multiple slopes (A0 to A(N-1)) for the multiple programmable segments according to the interval determination data (SDD0 to SDD(N-2)). The selector (SLT) outputs, according to interval determination data (SDD0 to SDD(N-2)), an offset (B) for a programmable segment corresponding to the interval of the segment to which the input value (X) belongs among a plurality of offsets (B0 to B(N-1)) for a plurality of programmable segments.

[0423] When a specific segment is determined based on interval determination data (SDD), the slope (A) and offset (B) of the determined segment can be output from the selector (SLT). That is, the programmable parameter includes the slope (A) and offset (B) for each of the programmable segments. The slope (A) is multiplied with the input value (X) in the multiplier (PAF MUL), and the offset (B) is added to the output value of the multiplier (PAF MUL) in the adder (PAF ADD). That is, the output of the multiplier (PAF MUL) is It can be expressed as, and the output of the adder (PAF ADD) is It can be expressed as.

[0424] Meanwhile, the operation of each of the multiple comparators (COMP0 to COMP(N-2)) can be determined by each of the multiple comparator enable signals (Comp En 1 to Comp En (N-2)). For example, even when an activation function transformation program (not shown) programs the same activation function, the first programmed activation function can be programmed to have 10 segments, and the second programmed activation function can be programmed to have 5 segments. Accordingly, the programmable function approximation circuit (PAF) can control the number of comparators activated in the programmable function approximation circuit (PAF) differently according to each programmed activation function data, even for the same activation function. Accordingly, the accuracy of the artificial neural network operation and the power consumption of the neural processing unit (100) can also vary depending on the programming. That is, depending on the user's requirements, it is possible to provide a high-performance activation function operation function or a low-power activation function operation function, even for the same activation function.

[0425] Meanwhile, the number of multiple comparators that take the segment boundary values ​​(SB) as input must vary depending on the maximum number of segment boundary values ​​(SB). For example, if the maximum number of segment boundary values ​​(SB) is 10, at least 10 comparators may be provided. That is, the minimum number of comparators can be the maximum number of segment boundary values.

[0426] Accordingly, each of the multiple comparators (COMP0 to COMP(N-2)) can determine whether to operate based on each of the multiple comparator enable signals (Comp En 1 to Comp En (N-2)). Thus, by controlling the operation of unnecessary comparators according to the number of multiple segments, the power consumption of the neural processing unit (100) can be reduced.

[0427] However, due to hardware limitations, the number of comparators may be limited. Accordingly, the number of multiple segments segmenting the activation function may be limited depending on the number of comparators of the programmable function approximation circuit (PAF).

[0428] Meanwhile, according to the first embodiment of the fourth example of the present disclosure, it is possible to distinguish between linear and non-linear sections of the activation function, and to minimize the number of segments by providing a variable segment width while minimizing the error value. Therefore, there is an advantage in that the number of gate counts of the hardware of the programmable function approximation circuit (PAF) of the neural processing unit (100) can be minimized by minimizing the number of comparators.

[0429] Additionally, the activation function programming method according to the first embodiment of the fourth example of the present disclosure may be configured to program a specific activation function based on information of a maximum comparator that can be supported.

[0430] Meanwhile, the selector (SLT) may be a multiplexer composed of multiple switching elements, but the configuration of the selector (SLT) can be varied. The programmed activation function operation unit of the programmable function approximation circuit (PAF) may refer to a circuit configured to receive input values ​​(X), slope (A), and offset (B) and calculate an output value (Y). The programmed activation function operation unit of the programmable function approximation circuit (PAF) may be configured to include at least one multiplier (PAF MUL) and one adder (PAF ADD). The programmed activation function operation unit of the programmable function approximation circuit (PAF) may be a hard-wired circuit. In particular, if the programmable function approximation circuit (PAF) is a hard-wired circuit, the programmable function approximation circuit (PAF) can process approximation operations in clock units by inputting only programmable parameters without complex instructions.

[0431] Meanwhile, the parameter storage and utilization method of the present disclosure has the following fundamental differences from the conventional LUT method in its technical concept and operating principle.

[0432] First, the nature of the stored data is different.

[0433] Conventional LUTs store 'final approximate result values' or 'representative values ​​for interpolation' for specific input intervals. This is a passive data table intended to retrieve pre-calculated result values ​​by looking up based on input addresses. In contrast, the programmable parameters stored in this disclosure are not result values, but 'coefficients' (i.e., programmable parameters) for controlling a hard-wired operation unit in real time.

[0434] Second, the role and operation method of the data are different.

[0435] Conventional LUT methods are 'lookup-centric' operations that read values ​​from memory. In contrast, the present disclosure is a 'computation-centric' architecture that directly performs polynomial operations such as Ax+B by supplying operation coefficients selected through a selector (SLT) in real time to a 'Direct Computation Engine' composed of a multiplier (PAF_MUL) and an adder (PAF_ADD). That is, the parameter storage unit of the present disclosure functions as an active 'Coefficient Register File' that dynamically configures the operation circuit, which is one of the core technical features of the present disclosure that is clearly distinguished from conventional LUTs.

[0436] In particular, the above programmable parameters (A0...A(N-1), B0...B(N-1), etc.) have an important technical feature in that they are constant values ​​generated offline through the deep learning-based optimization method described above.

[0437] In other words, these parameters, finally determined through learning, are not changed at runtime when the Neural Processing Unit (NPU) performs the actual inference operation, and are stored in advance in designated memory or coefficient registers within the Programmable Function Approximation (PAF). Therefore, when performing the operation, the NPU immediately loads and processes these pre-optimized constant values ​​without complex calculations, thereby eliminating additional computational overhead and maximizing the speed of function approximation operations.

[0438] In addition, the function approximation unit (FuncApx) of the present disclosure has a structure that allows a single programmable function approximation circuit (PAF) to be commonly reused for approximating various different types of dedicated functions (e.g., reciprocal, square root reciprocal, negative exponent, etc.). The effect of this structure can be maximized when combined with the learning-based parameter generation method. By having only an area-efficient single programmable function approximation circuit (PAF) and selectively loading and using a pre-learned optimal set of constant parameters (i.e., a programmable parameter set) according to the function to be processed, the inefficiency of designing a separate approximation circuit for each function is eliminated, and the utilization of hardware resources is increased while simultaneously ensuring high accuracy and flexibility.

[0439] Meanwhile, the means for storing programmable parameters (slope A, offset B, etc.) used by the PAF circuit of the present disclosure has a fundamental difference from a conventional lookup table (LUT) in its purpose and method of operation.

[0440] Conventional LUTs are passive memory tables that store 'final approximate result values' for specific input intervals and look up the corresponding result values ​​based on input addresses. In contrast, the means for storing parameters in the present invention functions as an active coefficient register file or an equivalent on-chip memory.

[0441] In other words, the values ​​stored here are not the final result, but 'coefficients' used to control hard-wired operations (multipliers, adders) in real time. The Selector (SLT) reads the coefficients corresponding to the determined segment from this register file and supplies them to the 'Direct Computation Engine'. Therefore, this circuit is a 'computation-centric' architecture that calculates results by directly performing arithmetic operations, such as Ax+B, on input values; this is clearly distinct from the conventional 'lookup-centric' LUT method.

[0442] FIG. 25 is a diagram showing a function approximation circuit of a neural processing unit configured to process a programmed activation function according to the third embodiment of the fourth example of the present disclosure.

[0443] The programmable function approximation circuit (PAF) according to the third embodiment of the fourth example of the present disclosure illustrated in FIG. 25 may correspond to the vector function approximation circuit (V_PAF) of the vector function approximation unit (1537) of FIG. 5 and the scalar function approximation circuit (S_PAF) of the scalar function approximation unit (1545) of FIG. 6. For convenience of explanation, the vector function approximation circuit (V_PAF) and the scalar function approximation circuit (S_PAF) are collectively referred to as the programmable function approximation circuit (PAF) to describe the third embodiment of the fourth example of the present disclosure.

[0444] In describing the programmable function approximation circuit (PAF) of FIG. 25, descriptions that overlap with the programmable function approximation circuit (PAF) of FIG. 24 may be omitted for the convenience of explanation.

[0445] Specifically, the programmable function approximation circuit (PAF) according to the third embodiment of the fourth example of the present disclosure illustrated in FIG. 25 is an example of a circuit configured to process an activation function programmed as a quadratic function.

[0446] Accordingly, the coefficients (A, B, C) for the programmable segment described above may include the coefficient of quadratic term (A), the coefficient of linear term (B), and the offset (C).

[0447] A programmable function approximation circuit (PAF) according to the third embodiment of the fourth example of the present disclosure may be configured to include a plurality of comparators (COMP0 to COMP(N-2)), a selector (SLT), a plurality of multipliers (PAF_MUL1, PAF_MUL2, PAF_MUL3), and a plurality of adders (PAF_ADD1, PAF_ADD2).

[0448] Each of the multiple comparators (COMP0 to COMP(N-2)) compares an input value (X) calculated in at least one processing core (P_CORE) with each of the multiple segment boundary values ​​(SB0 to SB(N-2)).

[0449] Accordingly, the interval of the segment to which the input value (X) belongs among the intervals of the plurality of segments can be determined through the output values ​​output from each of the plurality of comparators (COMP0 to COMP(N-2)).

[0450] Meanwhile, the operation of each of the multiple comparators (COMP0 to COMP(N-2)) can be determined by each of the multiple comparator enable signals (Comp En 1 to Comp En (N-2)).

[0451] And, according to the interval determination data (SDD0 to SDD(N-2)), the selector (SLT) outputs the coefficients (A, B, C) for the programmable segment corresponding to the interval of the segment to which the input value (X) belongs among the coefficients (A0 to A(N-1), (B0 to B(N-1)), (C0 to C(N-1)) for a plurality of programmable segments.

[0452] Specifically, a pre-stored set of coefficients or coefficient registers provides the coefficients of the second term (A0 to A(N-1)), the coefficients of the first term (B0 to B(N-1)), and the offset (C0 to C(N-1)) for each of the plurality of programmable segments to the selector (SLT).

[0453] And, the selector (SLT) can determine the segment to which the input value (X) belongs among the segments of the plurality of segments according to the segment determination data (SDD0 to SDD(N-2)) output from each of the plurality of comparators (COMP0 to COMP(N-2)).

[0454] And, the selector (SLT) can output the coefficients of the second term (A0 to A(N-1)), the coefficients of the first term (B0 to B(N-1)), and the offset (C0 to C(N-1)) for a plurality of programmable segments, the coefficients of the second term (A), the coefficients of the first term (B), and the offset (C) for a programmable segment corresponding to the interval of the determined segment.

[0455] Accordingly, the selector (SLT) can output the coefficient of the second term (A), the coefficient of the first term (B), and the offset (C) for the programmable segment corresponding to the interval of the segment to which the input value (X) belongs.

[0456] The programmed activation function operation part of the programmable function approximation circuit (PAF) may refer to a circuit part configured to receive input values ​​(X), coefficients of the second term (A), coefficients of the first term (B), and an offset (C) to calculate an output value (Y).

[0457] The programmed activation function operation unit of the programmable function approximation circuit (PAF) may be configured to include a plurality of multipliers (PAF_MUL1, PAF_MUL2, PAF_MUL3) and a plurality of adders (PAF_ADD1, PAF_ADD2) to process a second-order function or a first-order function.

[0458] The programmed activation function operation part of a programmable function approximation circuit (PAF) can be a hard-wired circuit.

[0459] A plurality of multipliers in the programmed activation function operation unit may include a first multiplier (PAF_MUL1), a second multiplier (PAF_MUL2), and a third multiplier (PAF_MUL3).

[0460] The first multiplier (PAF_MUL1) multiplies the input value (X) with the coefficient (A) of the quadratic term for the programmable segment corresponding to the interval of the segment to which the input value (X) belongs.

[0461] Specifically, the first multiplier (PAF_MUL1) multiplies the input value (X) calculated in at least one processing core (P_CORE) by the coefficient (A) of the second term for the programmable segment output from the selector (SLT).

[0462] Accordingly, the first multiplier (PAF_MUL1) can output the coefficient (A) of the quadratic term for the programmable segment by multiplying the input value (X). That is, the output of the first multiplier (PAF_MUL1) is It can be expressed as.

[0463] Then, the second multiplier (PAF_MUL2) multiplies the output value from the first multiplier (PAF_MUL1) with the input value (X).

[0464] Specifically, the second multiplier (PAF_MUL2) multiplies the input value (X) calculated by at least one processing core (P_CORE) by the output value output from the second multiplier (PAF_MUL2).

[0465] Accordingly, the output of the second multiplier (PAF_MUL2) is It can be expressed as. However, the above-described configuration is This is merely an example for implementing it, and variations can be implemented through various circuit combinations.

[0466] The third multiplier (PAF_MUL3) multiplies the input value (X) with the coefficient (B) of the first term for the programmable segment corresponding to the interval of the segment to which the input value (X) belongs.

[0467] Specifically, the third multiplier (PAF_MUL3) multiplies the input value (X) calculated in at least one processing core (P_CORE) by the coefficient (B) of the first term for the programmable segment output from the selector (SLT).

[0468] Accordingly, the third multiplier (PAF_MUL3) can output the coefficient (B) of the first-order term for the programmable segment by multiplying the input value (X). That is, the output of the third multiplier (PAF_MUL3) is It can be expressed as.

[0469] Multiple adders may include a first adder (PAF_ADD1) and a second adder (PAF_ADD2).

[0470] The first adder (PAF_ADD1) adds the output value of the third multiplier (PAF_MUL3) to the output value of the second multiplier (PAF_MUL2).

[0471] Specifically, the first adder (PAF_ADD1) can output the sum of the second term and the first term of each of a plurality of programmable segments composed of second terms. That is, the output of the first adder (PAF_ADD1) is It can be expressed as.

[0472] Then, the second adder (PAF_ADD2) adds the offset (C) for the programmable segment corresponding to the interval of the segment to which the input value (X) belongs to the output value of the first adder (PAF_ADD1).

[0473] Specifically, the adder (PAF_ADD) adds the offset (C) for the programmable segment to the sum of the quadratic and linear terms of the programmable segment composed of quadratic terms. That is, the output of the second adder (PAF_ADD2) is It can be expressed as.

[0474] Accordingly, the adder (PAF_ADD) can output an activation value to which an activation function programmed as a quadratic function is applied to the operation value of the input value (X).

[0475] According to the configuration described above, the programmable function approximation circuit (PAF) operates to process second-order polynomials.

[0476] Meanwhile, the operation of the second multiplier (PAF_MUL2), the third multiplier (PAF_MUL3), and the second adder (PAF_ADD2) can be controlled by the first enable signal (EN1).

[0477] Specifically, when the second multiplier (PAF_MUL2), the third multiplier (PAF_MUL3), and the second adder (PAF_ADD2) are not operated by the first enable signal (EN1), the operation is as described below.

[0478] The first multiplier (PAF_MUL1) multiplies the input value (X) with the coefficient (A) of the quadratic term for the programmable segment corresponding to the interval of the segment to which the input value (X) belongs.

[0479] Specifically, the first multiplier (PAF_MUL1) multiplies the input value (X) calculated in at least one processing core (120) by the coefficient (A) of the second term for the programmable segment output from the selector (SLT).

[0480] Accordingly, the first multiplier (PAF_MUL1) can output the coefficient (A) of the quadratic term for the programmable segment by multiplying the input value (X). That is, the output of the first multiplier (PAF_MUL1) is It can be expressed as.

[0481] And, since the second multiplier (PAF_MUL2) and the third multiplier (PAF_MUL3) do not operate, the output of the first multiplier (PAF_MUL1) is input directly to the first adder (PAF_ADD1). That is, the arithmetic unit disabled by the first enable signal (EN1) can be bypassed.

[0482] Then, the first adder (PAF_ADD1) adds the coefficient (B) of the first term for the programmable segment corresponding to the interval of the segment to which the input value (X) belongs to the output value of the first multiplier (PAF_MUL1).

[0483] Specifically, the first adder (PAF_ADD1) adds the coefficient of the first term (B) for the programmable segment to the value obtained by multiplying the input value (X) by the coefficient (A) of the second term for the programmable segment. That is, the output of the first adder (PAF_ADD1) is It can be expressed as.

[0484] And, since the second adder (PAF_ADD2) does not operate, the output of the first adder (PAF_ADD1) is output as is. That is, the arithmetic unit disabled by the first enable signal (EN1) can be bypassed.

[0485] That is, the first adder (PAF_ADD1) can output an activation value to which an activation function programmed as a first-order function is applied to the operation value, which is the input value (X).

[0486] According to the configuration described above, the programmable function approximation circuit (PAF) operates to process first-order polynomials.

[0487] As described above, some configurations of a plurality of multipliers and a plurality of adders can be controlled by a first enable signal (EN1). Accordingly, depending on the first enable signal (EN1), the programmable function approximation circuit (PAF) can be driven not only when each programmable segment is a second-order polynomial, but also when each programmable segment is a first-order polynomial.

[0488] To elaborate, according to one example of the present disclosure, at least one processing core (120) and a pipelined programmable function approximation circuit (PAF) may be composed of a hard-wired circuit configured to implement both a second-order function and a first-order function activation function.

[0489] Therefore, there is an advantage in being able to handle various cases of a Programmable Function Approximation (PAF) with a single PAF.

[0490] Meanwhile, the programmable function approximation circuit (PAF) includes a plurality of comparators (COMP0 to COMP(N-2)) (COMP0 to COMP(N-2)), a selector (SLT), a plurality of multipliers (PAF_MUL1, PAF_MUL2, PAF_MUL3) and a plurality of adders (PAF_ADD1, PAF_ADD2), and may further include a logarithm operator.

[0491] FIG. 26 is a drawing showing an example of a number system processed in a special function unit of a neural processing unit that can be applied to the examples of the present disclosure.

[0492] The special function unit (150) of the neural processing unit (100) according to the examples of the present disclosure is designed to process at least one floating-point number system.

[0493] For example, the special function unit (150) may be designed to handle floating-point numbers. Referring to FIG. 26, the floating-point number is represented in the brain floating-point 16 format. Referring to FIG. 26, the floating-point number system of the BF16 format includes a sign bit (S), an exponent bit (E), and a mantissa bit (M).

[0494] However, the number system of the special function unit (150) according to the examples of the present disclosure may use IEEE standard 754, and other number systems may be applied to improve the computational efficiency, reduce memory usage, and reduce power consumption of the special function unit (150). However, the examples of the present disclosure are not limited thereto, and Dynamic Floating-Point, Variable Precision Floating-Point (VPFP), Custom Floating-Point (CFP), Flexible Floating-Point (FFP), etc. may be applied. Accordingly, the bit width allocation of the exponent part and the bit width allocation of the mantissa part may vary depending on the number system supported by the special function unit (150) of the neural processing unit (100).

[0495] For example, when a special function unit (150) according to the examples of the present disclosure is configured to use the FP32 number system, the special function unit (150) has 31 of weight parameters, input parameters, and output parameters st Recognize the bit as the sign bit (S), and 23 rd bit to 30 th bit is recognized as an exponent bit (E), and 0 th bit to 22 nd It is configured to recognize bit as a mantissa bit (M).

[0496] For example, when a special function unit (150) according to the examples of the present disclosure is configured to use the FP16 number system, the special function unit (150) has 15 of a weight parameter, an input parameter, and an output parameter.th Recognize the bit as the sign bit (S), and 10 th bit to 14 th bit is recognized as an exponent bit (E), and 0 th bit to 9 th It is configured to recognize bit as a mantissa bit (M).

[0497] For example, when a special function unit (150) according to the examples of the present disclosure is configured to use the Brain Floating 16 (BF16) number system, the special function unit (150) has 15 of a weight parameter, an input parameter, and an output parameter. th Recognize the bit as the sign bit (S), and 7 th bit to 14 th bit is recognized as an exponent bit (E), and 0 th bit to 6 th It is configured to recognize bit as a mantissa bit (M).

[0498] This can be expressed as a mathematical formula, as shown in <Mathematical Formula 6>.

[0499]

[0500] Here, x is the input in floating-point form, s represents the sign of the input (x), e represents the exponent of the input (x), and m represents the mantissa of the input (x). The bias of the BF16 is 127, and the actual value range of the exponent is -126 to 127. The actual exponent value can be obtained by subtracting the bias from the stored exponent.

[0501] FIG. 27 is a block diagram illustrating a function approximation part according to the fourth embodiment of the fourth example of the present disclosure.

[0502] With reference to FIGS. 5, 6, 24, 25, and 27, a function approximation unit (FuncApx) according to the fourth embodiment of the fourth example of the present disclosure will be described.

[0503] Referring to FIG. 27, a function approximation unit (FuncApx) according to the fourth embodiment of the fourth example of the present disclosure includes a programmable function approximation circuit (PAF), an inverse function circuit (RECIP), an inverse of the square root function circuit (RSQRT), and a negative exponential function circuit (NEXP). These circuits are individual circuits configured to independently process the operations of functions required for the attention score calculation of an artificial intelligence model.

[0504] According to the fourth embodiment of the fourth example of the present disclosure, the function approximation unit (FuncApx) of FIG. 27 may correspond to the vector function approximation unit (V_FuncApx) of FIG. 5 or the scalar function approximation unit (S_FuncApx) of FIG. 6. According to the fourth embodiment of the fourth example of the present disclosure, the programmable function approximation circuit (PAF) of FIG. 24 may correspond to the programmable function approximation circuit (PAF) of FIG. 25 or the programmable function approximation circuit (PAF) of FIG. 27. According to the fourth embodiment of the fourth example of the present disclosure, the reciprocal function circuit (RECIP) of FIG. 27 may correspond to the vector reciprocal function circuit (V_RECIP) of FIG. 5 or the scalar reciprocal function circuit (S_RECIP) of FIG. 6. According to the fourth embodiment of the fourth example of the present disclosure, the inverse square root function circuit (RSQRT) of FIG. 27 may correspond to the vector inverse square root function circuit (V_RSQRT) of FIG. 5 or the scalar inverse square root function circuit (S_RSQRT) of FIG. 6. According to the fourth embodiment of the fourth example of the present disclosure, the negative exponential function circuit (NEXP) of FIG. 27 may correspond to the vector negative exponential function circuit (V_NEXP) of FIG. 5 or the scalar negative exponential function circuit (S_NEXP) of FIG. 6.

[0505] Each of the programmable function approximation (PAF), reciprocal function (RECIP), reciprocal of the square root (RSQRT), and negative exponential function (NEXP) circuits is a hard-wired operation circuit. That is, when each of the above circuits receives input parameters, they can perform operations very quickly without complex instructions through the hard-wired operation circuits. That is, the function approximation unit (FuncApx) activates at least one circuit corresponding to the function to be processed. Subsequently, when the function approximation unit (FuncApx) provides input parameters to the activated circuit, the function is calculated along the pipeline of the calculation circuit without complex instruction control. That is, each of the programmable function approximation (PAF), reciprocal function (RECIP), reciprocal of the square root (RSQRT), and negative exponential function (NEXP) circuits is a dedicated operation circuit for the function corresponding to each circuit. To elaborate, hard-wired computational circuits can only process predefined operations and, due to their lack of versatility, cannot process other operations. However, each of the Programmable Function Approximation (PAF), Reciprocal Function (RECIP), Reciprocal of the Square Root (RSQRT), and Negative Exponential Function (NEXP) circuits can process predefined operations quickly and at low power. In particular, since all of the above computational circuits can be used for attention score calculation, transformer-based artificial intelligence models can be computed quickly.

[0506] For example, the function approximation unit (FuncApx) may be designed to process function approximation operations by activating only the programmable function approximation circuit (PAF). For example, the programmable function approximation circuit (PAF) may process at least one of Swish, Mish, Sigmoid, Hyperbolic Tangent (Tanh), Scaled Exponential Linear Unit (SELU), Gaussian Error Linear Unit (GELU), Softplus, Rectified Linear Unit (ReLU), Leaky ReLU, Maxout, Exponential Linear Unit (ELU), Hard Sigmoid, Hard Tanh, Bent Identity, ArcTan, Softsign, Sinusoidal Activation, S-shaped Rectified Linear Unit (SReLU), Parametric Rectified Linear Unit (PReLU), Logistic Activation, Radial Basis Function (RBF), Binary Step, Reciprocal, root mean square, and negative exponential as a programmable parameter input.

[0507] Furthermore, the function approximation unit (FuncApx) can be designed to process the corresponding reciprocal function, square root function, and negative exponential function more precisely by activating one of the reciprocal function circuit (RECIP), the reciprocal of the square root function circuit (RSQRT), and the negative exponential function (NEXP) circuit while activating the programmable function approximation circuit (PAF).

[0508] The feature of the aforementioned function approximation unit (FuncApx) lies in its hybrid architecture designed to achieve both generality and efficiency simultaneously.

[0509] That is, the function approximation unit (FuncApx) is equipped with a programmable function approximation circuit (PAF) to flexibly respond to new non-linear functions of various forms or that may appear in the future, such as GELU and Swish, and a dedicated function circuit to process functions that are frequently used in neural network operations and have a fixed form, such as RECIP, RSQRT, and NEXP, at the highest speed and efficiency.

[0510] When the control unit (110) receives a command indicating the type of function to be processed, it selectively activates either a PAF circuit or a specific dedicated function circuit accordingly. At this time, power supply to other circuit blocks that are not activated is cut off through power gating or clock gating, thereby minimizing unnecessary power consumption.

[0511] In addition, instead of each dedicated function circuit (RECIP, RSQRT, NEXP) having its own approximation circuit, a single PAF circuit can be reused as a shared operation unit. Each dedicated function circuit preprocesses the input value according to its function characteristics (e.g., mantissa normalization) and then passes the result to the shared PAF circuit to delegate the approximation operation. In this case, the control unit can be configured to provide the PAF circuit with an optimal set of pre-learned programmable parameters (P_PARA-1, P_PARA-2, etc.) tailored to the corresponding function.

[0512] This structure can provide the key effect of the present disclosure, which drastically reduces the hardware area compared to a method of implementing a separate complex approximation circuit for each function, while maintaining high accuracy and performance through preprocessing circuits and parameters optimized for each function.

[0513] FIG. 28 is a block diagram illustrating an inverse function processing circuit according to the fifth embodiment of the fourth example of the present disclosure.

[0514] The inverse function processing circuit according to the fifth embodiment of the fourth example of the present disclosure is a hardware circuit configuration for inverse operations implemented within a function approximation unit (FuncApx) of a neural processing unit (NPU). The inverse function processing circuit is configured to utilize an inverse function circuit (RECIP) and a programmable function approximation circuit (PAF) within the function approximation unit (FuncApx) to perform inverse operations. FIG. 28 illustrates the overall configuration of a function approximation unit (FuncApx) including the inverse function processing circuit (RECIP), and FIG. 29 illustrates the detailed configuration of the inverse function processing circuit (RECIP).

[0515] Referring to FIG. 28, the neural processing unit (NPU) is configured to activate the programmable function approximation circuit (PAF) and the reciprocal function circuit (RECIP) within the function approximation unit (FuncApx) to perform reciprocal function processing. The function approximation unit (FuncApx) is configured such that the input and output terminals of each circuit are electrically connected so that the programmable function approximation circuit (PAF) and the reciprocal function circuit (RECIP) communicate with each other.

[0516] As illustrated in FIG. 28, a data path for processing the inverse function is connected from the input value (IN) to the inverse function circuit (RECIP). The output of the inverse function circuit (RECIP) is configured to be connected to the programmable function approximation circuit (PAF). The output of the programmable function approximation circuit (PAF) is configured to be connected back to one of the input terminals of the inverse function circuit (RECIP). Through this, the inverse function processing circuit is configured to process the inverse function calculation by performing iterative processing between the inverse function circuit (RECIP) and the programmable function approximation circuit (PAF). The final output of the inverse function circuit (RECIP) is configured to be output as the output value (OUT).

[0517] In one embodiment, when the reciprocal function circuit (RECIP) and the programmable function approximation circuit (PAF) are enabled, other dedicated operation circuits within the function approximation unit (FuncApx), such as the reciprocal of the square root function circuit (RSQRT) and the negative exponential function circuit (NEXP), may be configured to be disabled. The function approximation unit (FuncApx) is configured to reduce power consumption of the disabled circuits by utilizing at least one of power gating, clock gating, and register retention functions.

[0518] Another key technical concept of the present disclosure is to overcome the inefficiency of approximating a function by executing multiple instructions sequentially. Unlike conventional methods that require separate instructions for each stage, such as preprocessing, approximation, and postprocessing, thereby causing computational latency, the dedicated function circuits of the present disclosure (see FIG. 29, 31, and 33) are implemented as a hard-wired pipeline that is triggered by a single instruction and operates autonomously.

[0519] Specifically, the control unit (110) transmits a single command, for example, "perform an inverse operation on input value A" to the function approximation unit (FuncApx). The control logic of the function approximation unit interprets the command and activates the inverse function circuit (RECIP). Once the input value A enters the pipeline of the inverse function circuit, all subsequent multi-stage operations proceed autonomously sequentially by the hard-wired connections of the internal circuit and the pipeline registers (REG1, REG2, REG3) without additional command intervention from the control unit. At this time, the multi-stage operations may include (1) a pre-processing step of decomposing the input value into an exponent and a mantissa, (2) an approximation step using a common PAF circuit, and (3) a post-processing step of recombining the results.

[0520] When all pipeline stages are completed, the final result is output, and this entire process is completed in a single instruction. This minimizes the overhead associated with instruction fetching and decoding and enables complex function operations to be completed within a few clock cycles, thereby dramatically improving the throughput and performance of the neural processing unit.

[0521] FIG. 29 is a drawing illustrating the detailed configuration of an inverse function processing circuit according to the fifth embodiment of the fourth example of the present disclosure.

[0522] FIG. 29 is a block diagram showing the detailed configuration of an inverse function processing circuit. The inverse function processing circuit is a combination of dedicated arithmetic circuits designed to generate an output value (OUT) representing an approximation of the inverse value of an input value (IN) in a floating-point format. The input value (IN) has a floating-point format consisting of a sign bit (S), an exponent bit (E), and a mantissa bit (M), and may have a BF16 format as an example, but the present disclosure is not limited thereto. The inverse function processing circuit is configured to separate and process the exponent bit (E) and the mantissa bit (M) constituting the input value (IN) individually, and then combine the processed results to obtain the inverse value.

[0523] The inverse function processing circuit is designed to have a pipeline structure for hardware acceleration, and for this purpose, it is configured to include registers (input register (REG1), intermediate register (REG2), output register (REG3)) that store intermediate results of each processing step. The configuration of each register (REG1, REG2, REG3) can correspond to the bit width of the numeric format of the input data. For example, if the input data is BF16, the corresponding register can be configured as a 16-bit buffer.

[0524] This is differentiated from conventional approximation methods that require a large number of instructions and operation cycles by relying on the sequential execution of computer program instructions for implementation.

[0525] The reciprocal function processing circuit is configured to include at least the following components.

[0526] The input register (REG1) is configured to function as an input buffer that temporarily stores an input value (IN) input from the outside. The input register (REG1) is positioned to correspond to the first stage input of the pipeline.

[0527] The preprocessing circuit (PRE1) is configured to convert the input value (IN) stored in the input register (REG1) into a form suitable for reciprocal function calculation. The preprocessing circuit (PRE1) is configured to include a first decomposition circuit (DIV1), a first decoder (DEC1-1), and an input value generation circuit (INPUT_GEN-1). The first decomposition circuit (DIV1) is configured to separate the sign bit (S), the exponent bit (E), and the mantissa bit (M) from the input value (IN). The preprocessing circuit (PRE1) is configured to reduce the approximation error by converting the input value (IN) of the input register (REG1) into a form suitable for reciprocal function approximation.

[0528] The first decoder (DEC1-1) is configured to receive an exponent bit (E) separated from the first decomposition circuit (DIV1), decode it into an integer data form, and subtract a bias value stored in the decoded exponent bit (E) from the exponent bit (E) using a built-in binary calculation circuit (BCC). The binary calculation circuit (BCC) is configured to process at least binary subtraction. Through this process, the first decoder (DEC1-1) is configured to represent the actual value of the input exponent bit (E) (i.e., an exponent value in integer form with bias removed) and prepare it for use in subsequent exponent conversion and scaling operations.

[0529] The input value generation circuit (INPUT_GEN-1) is configured to receive a mantissa bit (M) separated from the first decomposition circuit (DIV1) and to generate a floating-point value that can be processed by the programmable function approximation circuit (PAF).

[0530] Specifically, the input value generation circuit (INPUT_GEN-1) generates an input value in the form of a floating-point number by combining the exponent bit and the sign bit (0) that are pre-set in the mantissa bit (M), and this mantissa (M) is in the form of a floating-point number (e.g., x = (-1) S × M × 2 (E-bias) It is configured to be normalized into the 1.M form (e.g., BF16 standard) to represent significant figures.

[0531] Through this process, the input value generation circuit (INPUT_GEN-1) uses the mantissa bit (M) extracted from the first decomposition circuit (DIV1) to convert it into a floating-point value mapped to a range of 1 or greater and less than 2 [1, 2] that the programmable function approximation circuit (PAF) can use for computation. The converted value is connected as an input to the programmable function approximation circuit (PAF), and consequently, the mantissa bit (M) of the generated floating-point number always falls within the range of 1 ≤ M < 2. This is intended to reduce the approximation error in the subsequent programmable function approximation circuit (PAF) by shifting the input range subject to function approximation to a domain where the change in function value is small.

[0532] The exponent conversion circuit (CONV_E-1) is configured to receive the exponent bit (E) decoded from the first decoder (DEC1-1) and output a value with its sign converted to a negative number. The output of the exponent conversion circuit (CONV_E-1) is configured to be connected to the first input of the adder (ADD). Therefore, the exponent conversion circuit (CONV_E-1) [is based on] the mathematical characteristics of the reciprocal operation (1 / 2 E =2 -E ) can be efficiently implemented in hardware.

[0533] The programmable function approximation circuit (PAF) is configured to receive a floating-point value converted from the input value generation circuit (INPUT_GEN-1) and to generate an approximation result of the reciprocal function by executing a programmed activation function. The programmable function approximation circuit (PAF) is configured to utilize a programmable parameter (P_PARA-1) configured to approximate the reciprocal function corresponding to a specific input interval of 1 or greater and less than 2.

[0534] Meanwhile, the above-mentioned programmable parameter (P_PARA-1) may be a value derived through deep learning-based learning according to one embodiment of the present disclosure.

[0535] Specifically, the programmable parameter (P_PARA-1) may be a coefficient derived by training a small artificial neural network optimized for expressing a piecewise linear function to approximate the target function. In the case of the reciprocal function circuit (RECIP), since the mantissa of the input is always normalized to a narrow range of [1, 2) during the preprocessing stage, the target function for training can be set to f(x) = 1 / x, and its definition range can be x ∈ [1, 2).

[0536] Artificial neural networks are trained to minimize the error with respect to the target function within predictable intervals with small variations. Once training is complete, the learned weights and biases of the neural network can generate an optimal set of programmable parameters from which a programmable function approximation circuit (PAF) can directly use hardware. This set includes segment boundary values, the width of each segment, and the gradient (A) and offset (B) of the corresponding interval. This approach enables the derivation of the most effective parameters that minimize the approximation error of the inverse function within limited hardware resources.

[0537] In particular, this process has the advantage of enabling hardware-software co-optimization that incorporates hardware constraints of the programmable function approximation circuit (PAF) (e.g., maximum number of comparators) into the learning process, thereby deriving programmable parameters (P_PARA-3) that perform the most precise approximation within given hardware resources.

[0538] That is, the programmable function approximation circuit (PAF) is configured to receive a programmable parameter (P_PARA-1) programmed with an inverse function. The programmable function approximation circuit (PAF) is configured to effectively reduce the approximation error compared to approximating over a wide input range by performing function approximation in a range of 1 to less than 2, where the change in function value is small. The programmable function approximation circuit (PAF) may internally utilize fixed-point number operations to perform function approximation, and in this case, it may be configured to include a process of converting a floating-point input to a fixed-point number and converting it back to a floating-point form after operation. The output of the programmable function approximation circuit (PAF) is connected to be transmitted to an intermediate register (REG2). For a detailed description of processing the programmed activation function using the programmable function approximation circuit (PAF), refer to FIGS. 21 through 25.

[0539] The second decomposition circuit (DIV2) is configured to receive the output value (approximation result) of the programmable function approximation circuit (PAF) through the intermediate register (REG2) and to separate the sign bit (Sp), exponent bit (Ep), and mantissa bit (Mp) of the operation result. The bits (Sp, Mp) separated from the second decomposition circuit (DIV2) are configured to be transmitted to the combining circuit (CONCAT) via the intermediate register (REG2), and the exponent bit (Ep) is connected to be transmitted to the second decoder (DEC2).

[0540] The second decoder (DEC2) is configured to receive the exponent bit (Ep) separated by the second decomposition circuit (DIV2), decode it into an integer data form, and then subtract the bias. Here, the integer value of the bias is 127. Through decoding, the actual value of the exponent bit (Ep) (i.e., a fixed-point exponent value in integer form with bias removed) can be expressed. That is, the second decoder (DEC2) is configured to decode the actual value by subtracting the bias from the exponent bit (Ep) of the reciprocal of the square root function approximated by the programmable function approximation circuit (PAF). The output of the second decoder (DEC2) can be configured to be connected to the second input of the adder (ADD).

[0541] The adder (ADD) is configured to obtain the exponent bit of an integer value with bias removed by adding the values ​​received from the first input section and the second input section. The adder (ADD) is a circuit configured to receive two binary input values ​​and process binary addition operations. That is, the decoded exponent bits input to the first and second input sections of the adder (ADD), respectively, are added so that the output of the adder (ADD) is restored to the initial range of the input value (IN). In other words, the exponent part of the final result can be determined by subtracting the offset (the difference between the input exponent and the bias) obtained from the initial input exponent from the exponent obtained through the calculation of the reciprocal of the mantissa part. The output of the adder (ADD) is connected to an encoder (ENC) and is configured to be transmitted to the encoder (ENC).

[0542] The encoder (ENC) is configured to generate the encoded exponent bit (En) by adding a predefined bias value (e.g., 127) to the unbiased exponent value, which is the output of the adder (ADD). In other words, the encoder (ENC) is configured to restore the exponent bit to a format suitable for the floating-point system by adding back the bias that was removed during the decoding process. The output of the encoder (ENC) is configured to be connected to a coupling circuit (CONCAT).

[0543] The concatenation circuit (CONCAT) is configured to combine the encoded exponent bit (En) input from the encoder (ENC), the sign bit (Sp) and mantissa bit (Mp) of the second decomposition circuit (DIV2) input through the intermediate register (REG2), in a preset order. Through this process, a floating-point bit sequence corresponding to the final approximation of the function is generated. The output of the concatenation circuit (CONCAT) is connected to be transmitted to the output register (REG3).

[0544] The output register (REG3) is configured to function as an output buffer that stores the output of the coupling circuit (CONCAT). The output register (REG3) is positioned as the final stage of the pipeline.

[0545] The process of calculating the reciprocal function can be explained through three calculation paths.

[0546] The first calculation path is a path that decodes an exponent bit (E) extracted from an input value (IN) and converts the decoded value into an inverse, and is configured to include an exponent processing part of a preprocessing circuit (PRE1) (e.g., an exponent extraction part of the first decomposition circuit (DIV1), a first decoder (DEC1-1)) and an exponent conversion circuit (CONV_E-1). The output of this path is provided as the first input of an adder (ADD).

[0547] The second calculation path is a path that converts the mantissa bit (M) of the input value (IN) into the input of a programmable function approximation circuit (PAF), calculates an approximation of the reciprocal function in the programmable function approximation circuit (PAF), separates the exponent bit (Ep) and mantissa bit (Mp) of the approximation value, and decodes the exponent bit (Ep). It is configured to include a mantissa processing part of a preprocessing circuit (PRE1) (e.g., the mantissa extraction part of the first decomposition circuit (DIV1), an input value generation circuit (INPUT_GEN-1)), a programmable function approximation circuit (PAF), a second decomposition circuit (DIV2), and a second decoder (DEC2). The output of the second decoder (DEC2) is provided as the second input of an adder (ADD), and the output (Sp, Mp) of the second decomposition circuit (DIV2) is provided to a combination circuit (CONCAT) via an intermediate register (REG2).

[0548] The third calculation path is configured to include circuits (e.g., an adder (ADD), an encoder (ENC), and a combination circuit (CONCAT)) arranged to calculate the output value (OUT) of an input value (IN) (i.e., the reciprocal of the input value) by adding the output value of the first calculation path (i.e., the output value of the exponent conversion circuit (CONV_E-1)) and the output value of the second calculation path (i.e., the output value of the second decoder (DEC2)) and combining the exponent bit (En) that encoded the added value, the mantissa bit (Mp) of the first function approximation value, and the sign bit (Sp) in a preset order.

[0549] The reciprocal function processing circuit is designed to have a pipeline structure by placing registers (input register (REG1), intermediate register (REG2), and output register (REG3)) between each operation stage. The pipeline structure is configured to provide a hardware acceleration effect that significantly improves the total number of operations (throughput) that can be processed per unit time while maintaining the completion time (latency) of individual operations, by allowing multiple reciprocal calculation tasks to be processed simultaneously at different stages of the circuit. This differentiates it from conventional approximation methods, which require a large number of instructions and operation cycles because they rely on the sequential execution of computer program instructions for implementation. The reciprocal function processing circuit of the present disclosure is designed such that when input parameters required for a function operation (e.g., input value (IN)) are provided to an input register (REG1), the necessary calculations are automatically performed in dedicated hardware blocks (PRE1, DIV1, DEC1-1, INPUT_GEN-1, CONV_E-1, PAF, DIV2, DEC2, ADD, ENC, CONCAT, etc.) corresponding to each operation step, and no separate instructions are required to control intermediate operations. That is, once input data enters the pipeline, each operation step proceeds sequentially according to the data flow through the hardware's own hard-wired connections and registers, thereby enabling high-speed pipeline operations.

[0550] According to the configuration, the reciprocal function can be calculated at high speed by activating only hard-wired dedicated computational circuits (Reciprocal Function Circuit (RECIP) and Programmable Function Approximation Circuit (PAF)). In particular, when the Reciprocal Function Circuit (RECIP) and the Programmable Function Approximation Circuit (PAF) are used together, the configuration is designed to reduce approximation errors over a wider range of inputs than when only the Programmable Function Approximation Circuit (PAF) is used. According to the present disclosure, other dedicated circuits, such as the Reciprocal of Square Roots Circuit (RSQRT) or the Negative Exponential Function Circuit (NEXP), which are not used during the calculation of the reciprocal function, are configured to be deactivated, thereby efficiently reducing the power consumption of the neural processing unit (NPU).

[0551] The reciprocal function processing circuit according to the fifth embodiment of the fourth example of the present disclosure proposes a dedicated hardware accelerator to overcome the limitations of the operation cycles of the existing software instruction-based approximation method. In particular, to efficiently process complex floating-point operations, a partitioned processing and pipeline structure based on the characteristics of the input data (exponent and mantissa) may be adopted.

[0552] In the pipeline structure of the circuit, each stage of operation is implemented in dedicated hardware, so data that has entered the pipeline is processed following a continuous hardware flow. This eliminates the overhead of fetching, decoding, and executing separate instructions for each arithmetic or logical operation in conventional processors, thereby dramatically improving the processing speed of inverse function operations.

[0553] Furthermore, the reciprocal function processing circuit effectively reduces approximation errors arising from interval-based approximation methods by pre-mapping the input mantissa to a range favorable for function approximation. This enables the provision of high-accuracy reciprocal function approximations even with limited hardware resources.

[0554] The reciprocal function processing circuit includes a power management function that selectively activates only the necessary dedicated computation circuits and disables the remaining circuits. This allows for high-performance computation while minimizing unnecessary power consumption, thereby contributing to increased energy efficiency of the entire system.

[0555] FIG. 30 is a block diagram illustrating a circuit for processing the inverse of a square root function according to the sixth embodiment of the fourth example of the present disclosure.

[0556] The inverse square root function processing circuit according to the sixth embodiment of the fourth example of the present disclosure is also a hardware circuit configuration for the inverse square root operation implemented within the function approximation unit (FuncApx) of the neural processing unit (NPU). The inverse square root function processing circuit is configured to utilize the programmable function approximation circuit (PAF) and the inverse square root function circuit (RSQRT) within the function approximation unit (FuncApx) to perform the inverse square root operation.

[0557] FIG. 30 illustrates the overall configuration of a function approximation unit (FuncApx) that includes a circuit for processing the inverse of a square root function, and FIG. 31 illustrates the detailed configuration of the circuit for processing the inverse of a square root function.

[0558] Referring to FIG. 30, the neural processing unit (NPU) is configured to activate the programmable function approximation circuit (PAF) and the inverse function of the square root circuit (RSQRT) within the function approximation unit (FuncApx) to perform processing of the inverse function of the square root. The function approximation unit (FuncApx) is configured such that the input and output terminals of each circuit are electrically connected so that the programmable function approximation circuit (PAF) and the inverse function of the square root circuit (RSQRT) communicate with each other.

[0559] As illustrated in FIG. 30, a data path for processing the inverse of a square root function is connected from an input value (IN) to an inverse of a square root function circuit (RSQRT). The output of the inverse of a square root function circuit (RSQRT) is configured to be connected to a programmable function approximation circuit (PAF). The output of the programmable function approximation circuit (PAF) is configured to be connected again to one of the input terminals of the inverse of a square root function circuit (RSQRT). Through this, the inverse of a square root function processing circuit is configured to process the calculation of the inverse of a square root function by performing iterative processing between the inverse of a square root function circuit (RSQRT) and the programmable function approximation circuit (PAF). The final output of the inverse of a square root function circuit (RSQRT) is configured to be output as an output value (OUT).

[0560] In one embodiment, when the inverse square root function circuit (RSQRT) and the programmable function approximation circuit (PAF) are enabled, other dedicated operation circuits within the function approximation unit (FuncApx), such as the inverse function circuit (RECIP) and the negative exponential function circuit (NEXP), may be configured to be disabled. The function approximation unit (FuncApx) is configured to reduce power consumption of the disabled circuits by utilizing at least one of power gating, clock gating, and register retention functions.

[0561] FIG. 31 is a diagram illustrating the detailed configuration of a circuit for processing the inverse of a square root function according to the sixth embodiment of the fourth example of the present disclosure.

[0562] FIG. 31 is a block diagram showing the detailed configuration of a circuit for processing the inverse of a square root function. The circuit for processing the inverse of a square root function is a combination of dedicated arithmetic circuits designed to generate an output value (OUT) representing an approximation of the inverse of a square root value for an input value (IN) in a floating-point format. The input value (IN) has a floating-point format consisting of a sign bit (S), an exponent bit (E), and a mantissa bit (M), and may have a BF16 format as an example, but the present disclosure is not limited thereto. The circuit for processing the inverse of a square root function is configured to separate and process the exponent bit (E) and the mantissa bit (M) constituting the input value (IN) individually, and then combine the processed results to obtain the inverse of a square root value.

[0563] The circuit processing the inverse of a square root function is designed to have a pipeline structure for hardware acceleration, and for this purpose, it is configured to include registers (input register (REG1), intermediate register (REG2), output register (REG3)) that store intermediate results of each processing step. The configuration of each register (REG1, REG2, REG3) can correspond to the bit width of the numeric format of the input data. For example, if the input data is BF16, the corresponding register can be configured as a 16-bit buffer.

[0564] This is differentiated from conventional approximation methods that require a large number of instructions and operation cycles by relying on the sequential execution of computer program instructions for implementation.

[0565] The circuit for processing the reciprocal of a square root function is configured to include at least the following components.

[0566] The input register (REG1) is configured to function as an input buffer that temporarily stores an input value (IN) input from the outside. The input register (REG1) is positioned to correspond to the first stage input of the pipeline.

[0567] The preprocessing circuit (PRE2) is configured to convert the input value (IN) stored in the input register (REG1) into a form suitable for calculating the inverse of a square root function. The preprocessing circuit (PRE2) is configured to include a first decomposition circuit (DIV1), a first decoder (DEC1-2), and an input value generation circuit (INPUT_GEN-2). The first decomposition circuit (DIV1) is configured to separate the sign bit (S), the exponent bit (E), and the mantissa bit (M) from the input value (IN). The preprocessing circuit (PRE2) is configured to reduce approximation error by converting the input value (IN) stored in the input register (REG1) into a form suitable for calculating the inverse of a square root function.

[0568] The first decoder (DEC1-2) is configured to receive an exponent bit (E) separated from the first decomposition circuit (DIV1), decode it into an integer data form, and subtract a bias from the decoded exponent bit (E) using a built-in binary calculation circuit (BCC). The binary calculation circuit (BCC) is configured to process at least binary subtraction.

[0569] At this time, the first decoder (DEC1-2) can be configured to select a bias value (e.g., 127 when even, 126 when odd in the case of BF16) depending on whether the decoded exponent bit (E) is even or odd. This conditional bias selection is intended to reduce rounding errors that may occur when the square root operation (division) is processed as a bit shift operation in the subsequent exponent conversion circuit (CONV_E-2).

[0570] Consequently, the first decoder (DEC1-2) is configured to represent the actual value of the input exponent bit (E) (i.e., the exponent value with bias removed) and to pass it to the subsequent exponent conversion circuit (CONV_E-2) and input value generation circuit (INPUT_GEN-2) so that it can be utilized in calculations. The function approximation unit (FuncApx) may include a selector or multiplexer for this conditional bias selection, and the bias value may be determined according to the numeric format of the input value (IN) and stored in memory or a register.

[0571] The input value generation circuit (INPUT_GEN-2) is configured to receive a mantissa bit (M) separated from the first decomposition circuit (DIV1) and to generate a floating-point value that can be processed by the programmable function approximation circuit (PAF).

[0572] Specifically, the input value generation circuit (INPUT_GEN-2) generates an input value in the form of a floating-point number by combining a mantissa bit (M), an exponent bit corresponding to information on whether the input exponent (E) provided by the first decoder (DEC1-2) is even or odd, and a sign bit set to 0. At this time, the value applied to the exponent bit is 127 if the input exponent is even, and 128 if it is odd. This exponent bit setting is intended to effectively map the mantissa part to the range 1≤M<4 depending on whether the original input exponent is odd or even. In particular, if the original exponent is odd, the mantissa is doubled (e.g., 2M) to make the valid range [2, 4], and thereby converts the exponent to an even number to reduce rounding errors that may occur when processing division via bit shift operations during square root operations.

[0573] Through this process, the input value generation circuit (INPUT_GEN-2) converts the floating-point mantissa bits (M) into a floating-point value mapped to a range of 1 to 4 that the programmable function approximation circuit (PAF) can use for computation. The converted value is connected to the input of the programmable function approximation circuit (PAF), and consequently, the mantissa bits (M) of the generated floating-point value always remain within the range of 1 to 4. This is intended to reduce the approximation error in the subsequent programmable function approximation circuit (PAF) by shifting the input range subject to function approximation to a domain where the change in function value is small.

[0574] The exponential conversion circuit (CONV_E-2) is configured to receive the exponent bit (E) decoded from the first decoder (DEC1-2), convert the sign of the value to a negative number, and then output a value obtained by dividing by 2. This is based on the mathematical characteristics of the reciprocal of the square root function (rsqrt(x)) (rsqrt(2 E )=2 -E / 2 or 2 -(E-1) / 2 The purpose is to calculate the inverse scaling value of the square root corresponding to the exponent part according to ). Specifically, the exponent conversion circuit (CONV_E-2) is configured to generate an output value by dividing the input exponent value by 2 and applying a minus sign. At this time, CONV_E-2 can be configured to perform this division without rounding errors through an efficient bit-shift operation by selecting an optimized bias based on whether the exponent bit (E) in the first decoder (DEC1-2) is even or odd. The output of the exponent conversion circuit (CONV_E-2) is configured to be connected to the first input of the adder (ADD).

[0575] The programmable function approximation circuit (PAF) is configured to receive a value in a converted floating-point format from the input value generation circuit (INPUT_GEN-2), execute a programmed activation function, and generate an approximation result of the reciprocal of the square root function. The programmable function approximation circuit (PAF) is configured to use a programmable parameter (P_PARA-2) configured to approximate the reciprocal of the square root function corresponding to specific input intervals of 1 to 4.

[0576] The above programmable parameter (P_PARA-2) may be a value derived through deep learning-based learning according to one embodiment of the present disclosure.

[0577] Specifically, the programmable parameter (P_PARA-2) may be a coefficient derived by training a small artificial neural network optimized for expressing a piecewise linear function to approximate the target function. In the case of the reciprocal square root function circuit (RSQRT), since the mantissa is normalized to the range [1, 4) depending on whether the input exponent is odd or even during the preprocessing stage, the target function for training can be set to f(x) = 1 / √x, and its definition range can be set to x ∈ [1, 4).

[0578] Artificial neural networks are trained to minimize the error with respect to the target function within predictable intervals with small variations. Once training is complete, the learned weights and biases of the neural network can generate an optimal set of programmable parameters from which a programmable function approximation circuit (PAF) can directly use hardware. This set includes segment boundary values, the width of each segment, and the gradient (A) and offset (B) of the corresponding interval. This approach enables the derivation of the most effective parameters that minimize the approximation error of the inverse function within limited hardware resources.

[0579] In particular, this process has the advantage of enabling hardware-software co-optimization that incorporates hardware constraints of the programmable function approximation circuit (PAF) (e.g., maximum number of comparators) into the learning process, thereby deriving programmable parameters (P_PARA-3) that perform the most precise approximation within given hardware resources.

[0580] That is, the programmable function approximation circuit (PAF) is configured to receive a programmable parameter (P_PARA-2) programmed with the reciprocal of the square root function. The programmable function approximation circuit (PAF) is configured to effectively reduce the approximation error compared to approximating over a wide input range by performing function approximation within a range of 1 to 4, where the change in function value is small. The programmable function approximation circuit (PAF) may internally utilize fixed-point operations to perform function approximation, and in this case, it may be configured to include a process of converting the floating-point input to a fixed-point value and converting it back to a floating-point form after operation. The output of the programmable function approximation circuit (PAF) is connected to be transmitted to an ...

Claims

Claim 1 A function approximation unit configured to approximate a non-linear function within a neural processing unit, comprising: a programmable function approximation (PAF) circuit configured to directly compute a piecewise function according to pre-stored computation coefficients; one or more dedicated function circuits configured to process at least one specific function among reciprocal, reciprocal square root, and negative exponential functions through a fixed hard-wired computation pipeline; and control logic configured to selectively activate one of the PAF circuit or the dedicated function circuit according to the type of non-linear function to be processed. Claim 2 In claim 1, the function approximation unit is configured such that the PAF circuit performs an operation of a segment-specific linear function or a segment-specific polynomial function having a degree of 1 or higher as a segment-specific function using the operation coefficients. Claim 3 A function approximation unit according to claim 1, wherein the dedicated function circuit processes the floating-point representation of an input value by decomposing it into an exponent part and a mantissa part, and the PAF circuit is configured to be used as a common operation unit that performs approximation operations on a specific normalized range of the decomposed mantissa part. Claim 4 In paragraph 3, the operation coefficient used by the PAF circuit is a constant value that is pre-optimized through artificial neural network learning to approximate the specific function within the specific normalized range, in a function approximation unit. Claim 5 In claim 1, the function approximation unit is configured such that the control logic performs power gating or clock gating to block power consumption of an inactive circuit. Claim 6 In claim 1, the function approximation unit is configured to load the operation coefficient from the on-chip shared memory of the neural processing unit in which the operation coefficient is stored, or to load the operation coefficient previously stored in a register within the function approximation unit. Claim 7 In claim 1, the neural processing unit includes a special function unit, and the function approximation unit is configured to be included in the special function unit. Claim 8 In claim 1, the programmable function approximation (PAF) circuit comprises: a plurality of comparators configured to process a floating-point input value (X) including a sign bit, an exponent, and a mantissa, and to determine the segment to which the input value belongs by comparing a plurality of segment boundary values ​​(SB) with the input value (X); a selector configured to select a set of programmable operation coefficients corresponding to the determined segment from a plurality of stored coefficient sets; and a hard-wired operation unit configured to directly perform a predefined arithmetic operation using the selected set of operation coefficients and the input value (X). Claim 9 delete Claim 10 In claim 8, the arithmetic operation performed by the hard-wired operation unit is a linear function per interval or a polynomial function per interval having a degree of 1 or higher, and the function approximation unit is configured to dynamically change the degree of the polynomial function per interval by deactivating the operation of some circuits of the hard-wired operation unit through an enable signal. Claim 11 In claim 8, the floating-point format is any one of Brain Floating-Point 16 (BF16), TensorFloat-32 (TF32), 8-bit Floating-Point (FP8), or Block Floating-Point (BFP), a function approximation part. Claim 12 delete Claim 13 In claim 11, the function approximation unit, wherein the set of programmable operation coefficients is pre-optimized to approximate a specific interval of the target function through artificial neural network learning, and the artificial neural network learning is configured to perform hardware-software co-optimization by considering the number of comparators included in the PAF circuit. Claim 14 delete Claim 15 delete Claim 16 delete Claim 17 delete Claim 18 delete Claim 19 delete Claim 20 delete

Images