Intelligent control server system including one or more ai modules

Abstract

The present disclosure provides a scalable intelligent control server system. The system may include a plurality of slots on a main board, at least one AI module mounted in the slots to perform AI computations, and a main processor that controls the entire system. Users can flexibly expand the AI ​​computation performance of the entire system by adjusting the number of AI modules as needed. Each AI module is equipped with a low-power NPU specialized for AI computation and adopts a standard form factor (M.2, E1.S, etc.) to improve cost-effectiveness and scalability. This resolves the issues of high power consumption and cost inherent in GPU-based systems and enables efficient support for various AI applications. The intelligent control server system can provide various services based on AI video analysis, such as smart factories, control, and retail analysis.

Description

Technology Field

[0001] The invention relates to an intelligent control server system comprising one or more AI modules having scalable AI computational performance. Background Technology

[0002] Artificial intelligence (AI) is a technology that mimics human intelligent activities such as perception, generation, classification, and reasoning. Just as the human brain processes information through the connections of numerous neurons and synapses, AI operates based on Artificial Neural Networks (ANNs) that model this process.

[0003] With the recent advancement of neural network technology, AI inference services utilizing big data are expanding into various fields. In particular, generative AI, such as Large Language Models (LLMs) based on the Transformer architecture, is demonstrating innovative performance in areas like text generation, translation, and summarization, and is being rapidly applied in real life. Transformers identify contextual relationships in input data through an attention mechanism, which consists of complex matrix multiplication and non-linear function operations.

[0004] As the performance and accuracy of these AI models improve, the parameter size, computational complexity, and memory bandwidth requirements are increasing exponentially. Currently, these large-scale computations are primarily processed by Graphics Processing Units (GPUs). However, GPUs face several fundamental limitations in the mass commercialization of AI technology.

[0005] First is the massive power consumption. According to the International Energy Agency (IEA), power consumption by data centers worldwide is expected to more than double by 2026, a level equivalent to the electricity consumption of a nation. The high power consumption of GPUs, reaching 700W each, further exacerbates this problem, making it difficult to build AI data centers in regions lacking power infrastructure.

[0006] Second, there are high costs and rigid scalability. High-performance GPUs are very expensive and occupy a significant amount of physical space, increasing the initial setup costs of AI systems. This issue poses a significant burden, particularly for providers of AI services. In a situation where it is difficult to accurately predict service demand, adopting GPU-based systems that require massive initial investment entails substantial business risk. If demand falls short of expectations, losses may occur due to overinvestment; conversely, if demand surges, it becomes difficult to rapidly expand the system, which can lead to a decline in service quality. In other words, existing systems have an inherent limitation in that they force large-scale investments, making flexible expansion—which involves gradually adjusting the investment scale in line with service growth—difficult.

[0007] Third is architectural inefficiency. GPUs are designed as general-purpose parallel processors (SIMT: Single Instruction, Multiple Threads) for graphics rendering, featuring a structure where numerous computational cores exchange data with external high-bandwidth memory (HBM, GDDR, etc.). However, this general-purpose structure is not optimized for the structured dataflow of AI inference, where matrix operations and non-linear functions are sequentially repeated. Consequently, bottlenecks occur in the data transfer path, leading to inefficiencies where computational cores become idle while waiting for data. This makes it difficult to fully utilize theoretical performance (TOPS) and results in reduced power efficiency.

[0008] Therefore, to overcome the limitations of GPUs and to commercialize AI technology more widely, there is an urgent need for a new low-power, low-cost, and miniaturized AI system architecture specialized for AI computation.

[0009] More specifically, the limitations of GPU-based systems are not merely limited to issues of power, cost, and scalability, but may stem from a fundamental mismatch between the inherent characteristics of AI inference workloads and GPU architecture. At the heart of this inefficiency lies the phenomenon of a "memory bottleneck." AI workloads can be divided into "compute-bound" tasks, where performance is limited by computational power itself, and "memory-bound" tasks, where performance is limited by data transfer speeds. Recent research indicates that the inference process of Large Language Models (LLMs) remains memory-intensive even when batch sizes are increased, and that external DRAM bandwidth saturation in GPUs is a primary cause of performance degradation.

[0010] GPUs are general-purpose parallel processors (SIMT: Single Instruction, Multiple Threads) originally designed for graphics rendering, optimized for thousands of computational cores to process independent operations in parallel. However, the inference process of neural network models is not merely a collection of random parallel operations, but is characterized by a structured dataflow in which formalized operations—such as matrix multiplication, activation function application, and normalization—are processed sequentially according to the layer order. The general-purpose architecture of GPUs is not optimized for this dataflow unique to AI inference, which can lead to structural inefficiencies.

[0011] This may be because GPU architectures rely on continuous data exchange between numerous compute cores and high-bandwidth memory (HBM, GDDR) located outside the chip. During the AI ​​inference process, bottlenecks occur in the data transfer path when bringing massive amounts of data, such as weights and activation values, from external memory to the compute cores; this can result in the compute cores entering an idle state while waiting for data. Analysis of actual LLM inference workloads revealed that GPU core occupancy often struggles to exceed 50%, and that more than 50% of attention kernel execution cycles stall due to data access delays.

[0012] Consequently, a significant discrepancy arises between the theoretical peak computational performance (TOPS, Tera Operations Per Second) presented by GPUs and the effective performance achieved in actual AI inference environments. In other words, the high TOPS figures of GPUs exhibit an "illusion of TOPS," where actual performance is not fully realized due to the structural limitation of memory bottlenecks.

[0013] The inefficiency of GPU architecture can directly lead to massive power consumption and cost issues. The power consumption of the latest AI accelerator GPUs is increasing exponentially with each generation. For example, the Thermal Design Power (TDP) of NVIDIA's Ampere A100 GPU was 400W, but its successor, the Hopper H100, increased to 700W; the latest Blackwell B200 has a TDP of 1000W, with some high-end versions reaching up to 1200W. The maximum power consumption of a single DGX B200 system equipped with eight B200 GPUs reaches approximately 14.3kW.

[0014] Such high power consumption is a major cause of skyrocketing operating costs for data centers. The International Energy Agency (IEA) projected that global power consumption by data centers, currently at approximately 415 TWh in 2024, will more than double to exceed 945 TWh by 2030 due to the proliferation of AI technology; this figure surpasses Japan's current total annual electricity consumption. AI is identified as the primary driver of this increase in power demand.

[0015] In terms of cost, GPUs are also becoming a major barrier to the popularization of AI systems. The market price of a single high-performance NVIDIA H100 GPU reaches approximately $31,000, placing a massive initial investment burden on companies looking to launch AI services. Introducing such expensive equipment on a large scale under uncertain demand forecasts entails significant business risk, and flexible expansion—such as gradually adjusting the scale of investment in response to changing demand—is difficult.

[0016] In conclusion, the evolutionary path of existing GPU-based systems is moving toward integrating more transistors and consuming higher power, suggesting that they are facing limits to sustainability in terms of energy and cost, which exacerbates the global energy crisis and the economic viability issues of data centers. Therefore, there is a need to shift the existing paradigm through a low-power, low-cost, and high-efficiency architecture specialized for AI inference computations, and this disclosure aims to propose a new architecture to address these issues. The problem to be solved

[0017] The present disclosure aims to present an alternative to mitigate power and cost issues associated with GPU-based AI systems and to promote the commercialization of AI technology.

[0018] The first task is to fundamentally resolve the high power consumption and cost issues of GPU systems based on low-power, low-cost neural network processing units (NPUs) specialized for AI computation.

[0019] The second challenge is to address the difficulty for AI service providers in predicting initial investment costs, timing, and business scale. Existing GPU-based systems require massive initial investment due to the difficulty of gradual expansion, resulting in significant business risks amidst the uncertainty of service demand. Accordingly, this disclosure aims to resolve such investment uncertainty by providing a system that allows for the flexible expansion of AI computing performance as desired by the user and at the necessary time by combining standardized AI modules.

[0020] The third task is to reduce manufacturing costs to accelerate the commercialization of AI technology. By applying industry-standard hardware specifications (form factors) such as M.2 and E1.S to AI modules to enable mass production, we aim to achieve 'economies of scale' and thereby lower the barriers to adoption of AI systems.

[0021] Ultimately, the problem that the present disclosure aims to solve is to implement a new AI system that flexibly expands low-power AI modules implemented in a standardized form factor to provide the computational performance required by various AI applications economically and efficiently.

[0022] The problems that this disclosure aims to solve are not limited to those mentioned above, and other unmentioned problems will be clearly understood by a person skilled in the art from the description below. means of solving the problem

[0023] An AI system according to one aspect of the present disclosure for solving the above problem comprises: a plurality of slots provided on a main board; at least one AI module mounted on at least one of the plurality of slots and performing AI (artificial intelligence) operations; and at least one main processor that controls the overall operation of a system including the AI ​​module, wherein the overall AI operation performance of the AI ​​system may be configured to be expandable based on the number of AI modules mounted in the plurality of slots.

[0024] Additionally, an AI module according to another aspect of the present disclosure may include: a standard printed circuit board; a connector formed on one side of the printed circuit board and electrically connected to a slot provided on the main board of the AI ​​system; and at least one NPU mounted on the printed circuit board and processing AI operations.

[0025] Furthermore, according to another aspect of the present disclosure, a neural network processing unit (NPU) may include: a processing core that parallelizes matrix operations based on integer data; a special function unit (SFU) that processes special operations including non-linear functions based on floating-point data; and a controller that controls the operation of the processing core and the special function unit based on compiled executable code.

[0026] Specific details of other examples are included in the detailed descriptions and drawings. Effects of the invention

[0027] According to the present disclosure, the following effects can be achieved.

[0028] First, it can provide flexible performance scalability. Users can start the system with the necessary number of AI modules without the burden of initial costs, and easily expand the overall AI computing performance by simply adding modules as demand increases in the future. This enables the construction of an economical and rational system.

[0029] Second, it is possible to build low-power, high-efficiency AI systems. By adopting NPUs specialized for AI computation and low-power standard form factors such as M.2 and E1.S, the overall power consumption of the system can be significantly reduced compared to GPU-based systems, even when multiple AI modules are installed. This can contribute to reducing operating costs and realizing eco-friendly data centers.

[0030] Third, it can support high compatibility and rapid execution. The system of the present disclosure can support a compiler that converts neural network models developed with various AI frameworks, such as TensorFlow and PyTorch, into machine code optimized for the NPU. Since code compiled once is stored and reused, various AI applications can be executed quickly and reliably in each AI module.

[0031] The effects according to the present disclosure are not limited to those exemplified above, and a wider variety of effects may be included within the present disclosure. Brief explanation of the drawing

[0032] FIG. 1 is a diagram illustrating a computing system for processing a transformer-based generative artificial intelligence language model that can be applied to the examples of the present disclosure. FIG. 2 is a flowchart illustrating a method in which a neural processing unit applicable to the examples of the present disclosure generates output data using an artificial intelligence model from input data. FIG. 3 is a block diagram showing a neural processing unit that can be applied to the examples of the present disclosure. FIG. 4 is a block diagram illustrating a processing element that can be applied to examples of the present disclosure. FIG. 5 is a block diagram showing a vector operation unit of a special function unit that can be applied to examples of the present disclosure. FIG. 6 is a block diagram showing a scalar operation unit of a special function unit that can be applied to the examples of the present disclosure. FIG. 7 is a diagram schematically showing the energy consumption per unit operation of a neural processing unit that can be applied to the examples of the present disclosure. FIG. 8 is a flowchart illustrating a method for controlling a neural processing unit that can be applied to examples of the present disclosure. FIG. 9 is a block diagram showing a system-on-chip according to a first embodiment of the first example of the present disclosure. FIG. 10 is a block diagram showing a system-on-chip according to a second embodiment of the first example of the present disclosure. FIG. 11 is a block diagram showing a system-on-chip according to a third embodiment of the first example of the present disclosure. FIG. 12 is a block diagram showing a system-on-chip according to a fourth embodiment of the first example of the present disclosure. FIG. 13 is a block diagram showing a system-on-chip according to the fifth embodiment of the first example of the present disclosure. FIG. 14 is a block diagram showing a system-on-chip according to the sixth embodiment of the first example of the present disclosure. FIG. 15 is a block diagram showing a system-on-chip according to the seventh embodiment of the first example of the present disclosure. FIG. 16 is a block diagram showing a system-on-chip according to the eighth embodiment of the first example of the present disclosure. FIG. 17 is a drawing showing an apparatus according to the first embodiment of the second example of the present disclosure. FIG. 18 is a drawing showing an apparatus according to a second embodiment of a second example of the present disclosure. FIG. 19 is a diagram showing a tree for explaining speculative decoding according to the third example of the present disclosure. FIG. 20 is a diagram schematically illustrating a speculative decoding operation according to a third example of the present disclosure. FIG. 21 is a flowchart illustrating an activation function programming method according to the first embodiment of the fourth example of the present disclosure. FIG. 22 is a graph showing the process of approximating an activation function by an activation function programming method according to the first embodiment of the fourth example of the present disclosure. FIG. 23 is a graph showing various cases of segmenting an activation function into a plurality of segments by an activation function programming method according to the first embodiment of the fourth example of the present disclosure. FIG. 24 is a diagram showing a function approximation circuit of a neural processing unit configured to process a programmed activation function according to the second embodiment of the fourth example of the present disclosure. FIG. 25 is a diagram showing a function approximation circuit of a neural processing unit configured to process a programmed activation function according to the third embodiment of the fourth example of the present disclosure. FIG. 26 is a drawing showing an example of a number system processed in a special function unit of a neural processing unit that can be applied to the examples of the present disclosure. FIG. 27 is a block diagram illustrating a function approximation part according to the fourth embodiment of the fourth example of the present disclosure. FIG. 28 is a diagram schematically illustrating an AI system according to the first embodiment of the fifth example of the present disclosure. FIG. 29 is a drawing illustrating an M.2 AI module according to the first embodiment of the sixth example of the present disclosure. FIG. 30 is a drawing illustrating an M.2 AI module including an NPU and dedicated memory according to the second embodiment of the sixth example of the present disclosure. FIG. 31 is a drawing illustrating an M.2 AI module including an NPU and a plurality of NPU-dedicated memories according to the third embodiment of the sixth example of the present disclosure. FIG. 32 is a drawing illustrating an E1.S AI module including a plurality of NPUs according to the fourth embodiment of the sixth example of the present disclosure. FIG. 33 is a drawing illustrating an E1.S AI module including a plurality of NPUs and NPU-dedicated memory according to the fifth embodiment of the sixth example of the present disclosure. FIG. 34 is a drawing illustrating an E1.S AI module including a plurality of NPUs and a plurality of NPU-dedicated memories according to the sixth embodiment of the sixth example of the present disclosure. FIG. 35 is a diagram schematically illustrating an AI system including a storage unit according to the first embodiment of the seventh example of the present disclosure. FIG. 36 is a diagram schematically illustrating an AI system including a main processor including an encoder and a decoder unit according to the second embodiment of the seventh example of the present disclosure. FIG. 37 is a diagram schematically illustrating an AI system including a decoder unit according to the third embodiment of the seventh example of the present disclosure. FIG. 38 is a diagram schematically illustrating an AI system including an encoder and a decoder unit according to the fourth embodiment of the seventh example of the present disclosure. FIG. 39 is a diagram schematically illustrating an AI system including a VPU according to the fifth embodiment of the seventh example of the present disclosure. FIG. 40 is a diagram schematically illustrating an AI system including a GPU according to the sixth embodiment of the seventh example of the present disclosure. Specific details for implementing the invention

[0033] Specific structural or step descriptions regarding the examples according to the concept of the present disclosure disclosed in this disclosure are provided merely for the purpose of explaining the examples according to the concept of the present disclosure.

[0034] Embodiments according to the concept of the present disclosure may be implemented in various forms. The present disclosure shall not be interpreted as being limited to the embodiments described herein.

[0035] Embodiments according to the concept of the present disclosure may be subject to various modifications. The present disclosure may take various forms. Accordingly, specific examples are illustrated in the drawings and described in detail in the present disclosure. However, this is not intended to limit the examples according to the concept of the present disclosure to specific forms of disclosure. Accordingly, all modifications, equivalents, and substitutions that fall within the spirit and scope of the present disclosure should be understood to be included in the present disclosure.

[0036] Terms such as first and / or second may be used to describe various components. However, the present disclosure should not be limited by such terms.

[0037] The above terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of rights according to the concept of the present disclosure, the first component may be named the second component, and similarly, the second component may be named the first component.

[0038] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. On the other hand, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

[0039] Other expressions describing the relationship between components, such as "between" and "exactly between," or "adjacent to" and "directly adjacent to," should be interpreted in the same way.

[0040] In this document, expressions such as “A or B,” “at least one of A or / and B,” or “one or more of A or / and B” may include all possible combinations of items listed together. For example, “A or B,” “at least one of A and B,” or “at least one of A or B” may refer to cases including (1) at least one A, (2) at least one B, or (3) both at least one A and at least one B.

[0041] Expressions such as "first," "second," "first," or "second" used in this disclosure may modify various components regardless of order and / or importance. Such expressions are used only to distinguish one component from another and do not limit said components. For example, a first user device and a second user device may represent different user devices regardless of order or importance. For example, without departing from the scope of rights set forth in this document, a first component may be named a second component, and similarly, a second component may be renamed a first component.

[0042] The terms used in this disclosure are used merely to describe specific embodiments and are not intended to limit the scope of other examples.

[0043] Singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as generally understood by a person of ordinary knowledge in the technical field described in this document.

[0044] Terms used in this disclosure that are defined in general dictionaries may be interpreted as having the same or similar meaning as they have in the context of the relevant technology. Terms used in this disclosure are not to be interpreted in an ideal or overly formal sense unless explicitly defined in this document. In some cases, even terms defined in this document are not to be interpreted to exclude the embodiments of this document.

[0045] The terms used in this disclosure are used merely to describe specific embodiments and are not intended to limit this disclosure.

[0046] The singular expression includes the plural expression unless the context clearly indicates otherwise. In this disclosure, terms such as “comprising” or “having” are intended to specify the existence of the described features, numbers, steps, actions, components, parts, or combinations thereof. Accordingly, it should be understood that the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof is not excluded in advance.

[0047] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which this disclosure pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology. Unless explicitly defined in this disclosure, they are not to be interpreted in an ideal or overly formal sense.

[0048] The features of each of the various embodiments of the present disclosure may be combined or combined with one another, either partially or wholly. As will be fully understood by those skilled in the art, the various embodiments of the present disclosure are technically capable of various interlocking and operation. Each of the embodiments of the present disclosure may be implemented independently of one another or may be implemented together in an interlocking relationship.

[0049] In describing each example, technical details that are well known in the technical field to which this disclosure belongs and are not directly related to this disclosure are omitted. This is intended to convey the essence of this disclosure more clearly without obscuring it by omitting unnecessary explanations.

[0050] In the present disclosure, 'unit' may refer to a hardware module or a hardware circuit. For example, 'unit' may be a hardware circuit for performing a specific function. However, 'unit' does not necessarily mean a dedicated circuit physically separated from other circuits to perform only that function. 'Unit' may be configured based on multiple circuits, and these circuits may exist at different locations and perform a specific function by being interconnected under a control signal. For example, 'Unit A' and 'Unit B' may include at least some common electronic circuits.

[0051] In this disclosure, the term "artificial intelligence model" may include not only generative AI models but also traditional AI models that perform judgment, classification, prediction, etc. Artificial intelligence models may include language models, models that interpret images or videos, and any model that generates output data from input data using an artificial neural network may qualify as an artificial intelligence model. For example, an artificial intelligence model may generate output text from input text. An artificial intelligence model may classify input data or generate prediction data from input data. An artificial intelligence model may search for or track objects of interest from images or videos. An artificial intelligence model may perform situational judgments by analyzing images or videos. An artificial intelligence model may generate an image / video corresponding to the input text based on the input text. An artificial intelligence model may generate output text describing the input image / video from the input image / video. Input data and output data may each be text, voice signals, images, videos, etc., and are not limited to any specific type.

[0052] An artificial intelligence model can be implemented as an artificial neural network. Therefore, as a representative embodiment, processing or computation regarding an 'artificial intelligence model' refers to processing or computation of an artificial neural network, and the parameters of the artificial intelligence model may include, but are not limited to, the weights of the artificial neural network. For example, it should be understood that processing or computation of an artificial intelligence model may include mathematical operations that are not included in the neural network structure, such as softmax.

[0053] 'Transformer' is an artificial intelligence model implemented based on an attention mechanism and can be used in language models (Large Language Model, LLM and Small LLM). Language models include BERT (Bidirectional Encoder Representations from Transformers), GPT (Generative Pre-trained Transformer), RoBERTa (Robustly Optimized BERT Pretraining Approach), ALBERT (A Lite BERT), ELECTRA (Efficiently Learning an Encoder that Classifies Token Replacements Accurately), Transformer-XL (Transformer with Extra Long Context), and XLNet (which combines the advantages of GPT and BERT). Combined model), BART (Bidirectional and Auto-Regressive Transformers), CTRL (Conditional Transformer Language), T5 (Text-to-Text Transfer Transformer), LaMDA (Language Model for Dialogue Applications), Gopher (DeepMind's LLM), InstructGPT (Fine-tuned model based on GPT-3), PanGu (Huawei's Chinese model), PaLM (Pathways Language Model), OPT 175B (Open Pretrained Transformer 175B), BLOOM (BigScience Large Examples include the Open-science Open-access Multilingual Model and HyperCLOVA (Naver's massive Korean model), and new language models that may emerge in the future may also be included.

[0054] Transformers can be utilized not only in natural language processing (NLP) but also in computer vision, and representative transformer-based image models include ViT (Vision Transformer) and Swin Transformer (Sliding Window-based Transformer for Vision).

[0055] Therefore, the Transformer is one of the core techniques of various artificial intelligence models utilizing attention mechanisms. For example, a Transformer-based language model can output 'yes you are an expert' when 'npu is an ai accelerator' is input.

[0056] According to examples of the present disclosure, devices, methods, systems, and computer-readable media that enable generative artificial intelligence services to operate on the device itself (also referred to as on-device) may be provided.

[0057] Generally, a transformer-based generative AI model receives a query as input and generates a response. For example, a language model receives a query in text form and generates tokens. Subsequently, the query and the generated tokens are input back into the language model to generate the next token. This token generation operation can be repeated multiple times until a response to the query is completed. Tokens may correspond to words or parts of words. The response generated by the language model may include a series of additional tokens generated by passing the tokens generated as part of the response back into the language model.

[0058] Below, some embodiments will be described clearly and in detail with reference to the accompanying drawings so that those skilled in the art to which this disclosure pertains (hereinafter, people skilled in the art) can easily practice this disclosure.

[0059] First, FIGS. 1 and 2 illustrate the process of inferring a Transformer-based generative artificial intelligence language model on an NPU. The computing system according to the present disclosure operates on an edge device and can infer input text data through embedding, attention, feedforward, softmax, etc., to output output tokens. This process consists of numerous matrix multiplications and non-linear operations, and the disclosed NPU can enable high-speed inference through the optimization of these operations.

[0060] FIG. 1 is a block diagram illustrating a computing system for processing a language model that can be applied to the examples of the present disclosure.

[0061] Referring to FIG. 1, a computing system (1) can receive input data and generate output data. The computing system (1) may include a Neural Processing Unit (NPU) (100). Additionally, the computing system (1) may include additional computational units such as a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and memory in addition to the Neural Processing Unit (100), and embedding transformation for the input text may be performed by the Central Processing Unit or the Graphics Processing Unit, etc. The transformed embedding vector is transmitted to the Neural Processing Unit (100) and may be utilized by the Neural Processing Unit (100) for subsequent inference operations. The Neural Processing Unit (100) may perform inference operations for a Transformer-based generative artificial intelligence model. The Neural Processing Unit (100) may be referred to as an artificial intelligence accelerator, an artificial intelligence hardware accelerator, etc., but is not limited thereto. Here, the computing system (1) may be an example of a device (10000) and may include a smartphone, a robot, an ADAS system, etc.

[0062] The neural processing unit (100) can generate output data from input data. According to one embodiment, the input data may be text data. The output data may be text data or image / video data. Text data consists of at least one token. For example, when text containing five tokens 'npu', 'is', 'an', 'AI', and 'accelerator' is input to the NPU, the neural processing unit (100) can perform a transformer-based artificial neural network operation to output text containing five tokens 'yes', 'you', 'are', 'an', and 'expert'.

[0063] Transformer-based artificial neural network computation can be performed on-device. That is, the computing system (1) can be included and operated on an edge device such as a smartphone, camera, CCTV, robot, or drone, and does not require wired or wireless communication with the outside of the computing system (1) for artificial neural network computation.

[0064] Transformer-based artificial neural network operations can be performed based on an attention mechanism. Under the attention mechanism, input data in text form is converted into embedding vectors through an embedding process, and these embedding vectors can be transformed into Query, Key, and Value vectors using the vector-matrix multiplication (MatMul) with the Query weight matrix, Key weight matrix, and Value weight matrix. The Query weight matrix, Key weight matrix, and Value weight matrix are each pre-trained values. Subsequently, the similarity between the Query (Q) and the Key (K) is calculated, and an attention score can be obtained by assigning a value (V) to the calculated similarity.

[0065] FIG. 2 is a flowchart illustrating a method in which a neural processing unit applicable to the examples of the present disclosure generates output data from input data using an artificial intelligence model.

[0066] Referring to FIG. 2, the neural processing unit (100) can obtain an attention score for input data (S11).

[0067] The input data may include multiple tokens, and each token has a predetermined dimension (d model It can be expressed as an embedding vector of ). The number of dimensions of the embedding vector (d model ) can be, for example, 128, 256, 1024, 2048, or 4096, etc. For convenience of explanation below, d model The explanation assumes 4096. In this embodiment, one token in the computing system (1) can be represented as an embedding vector of 1×4096 dimensions.

[0068] For example, if a user inputs “npu is an AI accelerator” consisting of 5 tokens into the computing system (1), the computing system (1) can convert the input text into an embedding vector of size 5×4096 (i.e., 5 1×4096), and the neural processing unit (100) can receive the embedding vector. The neural processing unit (100) can obtain a query vector Q by performing a vector-matrix product of the query weight matrix on the embedding vector. The neural processing unit (100) can obtain a key vector K by performing a vector-matrix product of the key weight matrix on the embedding vector. Then, the neural processing unit (100) can obtain similarity through the inner product between the query vector Q and the key vector K. When the query vector Q and the key vector K are multiplied by an inner product, a scalar value is obtained, and consequently, a vector QK of the same size (1×5) as the number of tokens in the input data is obtained. T is obtained. QK T represents the similarity between the current token and key k (each of the 5 tokens). However, the vector QK of the obtained size T Since the elements of can be numerically very large, QK for numerical stability TThe square root of the magnitude of the key vector K ( A normalized final similarity can be obtained by dividing by ) and then applying a softmax operation. <Mathematical Formula 1> below is a mathematical representation of the softmax-based similarity calculation. k can be one of 1, 2, …, N (N is the number of tokens included in the input data, in this embodiment N = 5).

[0069]

[0070] The softmax operation is an operation that converts multiple numbers into probabilities between 0 and 1. <Mathematical Formula 2> below represents the vector [z1, z2, … , z n Represents a formula that performs a softmax operation value for ].

[0071]

[0072] However, z i If there is a value that is too large, a numerical instability problem may occur. To resolve this, the maximum value z max By using the negative exponential trick, numerically more stable calculations can be made possible. <Mathematical Equation 3> below substitutes z for <Mathematical Equation 2> max This is a softmax calculation formula modified using .

[0073]

[0074] Through <Equation 1>, similarity can be obtained as a 1×5 vector of size [0.007, 0.993, 0, 0, 0], for example. Similarity is a value in which the similarity between the current query (e.g., 'npu') and other tokens is expressed as a probability.

[0075] According to one embodiment, the neural processing unit (100) can assign a value to the similarity. The value vector is similar to the method used to calculate the key vector. The embedding vector can be converted into a value vector through a vector-matrix multiplication with a pre-learned value weight matrix. An attention score can be obtained by performing a vector-matrix multiplication of the value vector V with the similarity obtained through <Equation 1>. Consequently, an attention score of size 5×4096, which is the same dimension as the input data, can be obtained by performing an operation such as <Equation 4> below.

[0076]

[0077] That is, the neural processing unit (100) repeatedly performs the attention operation of <Equation 4> to generate output data. In this process, it can be seen that the neural processing unit (100) repeatedly performs high-dimensional calculations including operations such as vector-matrix multiplication, division, and multiplication. According to one embodiment, the neural processing unit (100) may additionally perform residual linking and layer normalization. Residual linking and layer normalization are intended to compensate for the possibility of gradient vanishing that may occur in the differentiation-based learning method of deep learning, and can minimize information loss by adding input values ​​to the output values ​​that have passed through the layers of the artificial neural network. After the attention score is calculated for residual linking, the embedding vector of the input data can be added to the attention score. Layer normalization can mitigate covariant shift by adjusting values ​​based on the mean and variance of the output value (i.e., the result of residual linking) in natural language processing.

[0078] Next, the neural processing unit (100) can process the attention score through a feed-forward neural network (S12).

[0079] A feed-forward neural network is designed to predict non-linear phenomena and enables the inference of contextual meanings, such as phrases or words, by recombining relational information of tokens for which attention scores have been calculated. As an example, a feed-forward neural network may use a fully connected hidden layer with two layers having 2,048 nodes, and the output may be a 4,096-dimensional vector identical to the input. The feed-forward neural network is pre-trained, and vector-matrix multiplication operations may be repeatedly performed for inference. Here, the input is represented as a 4,096-dimensional vector as the result of residual connection and normalization in step S11, and the residual connection and normalization of step S11 may also be additionally performed on the output of the hidden layer composed of two layers.

[0080] Next, the neural processing unit (100) can generate a final output token (S13).

[0081] To this end, the output vector generated in step S12 must be expanded to the total number of tokens in the set that the language model can process (e.g., 30,000). For example, the output vector from step S12 can be expanded into a logit vector of size 1 × 30,000 by performing a matrix multiplication operation with the final output weight matrix (e.g., 4,096 × 30,000). The size of the final output weight matrix for the above final output is d. modelAs ×SIZE_DIC, SIZE_DIC represents the total number of tokens that can be output from a language model undergoing inference operations in the neural processing unit (100) (30,000 in this embodiment). A softmax operation is applied to the calculated logit vector of size 1×30,000 so that all 30,000 tokens are normalized to probability values ​​between 0 and 1, and the token having the highest probability value among the 30,000 probability values ​​can be selected as the next output token to be generated. For example, if the input to the neural processing unit (100) is 'npu is an AI accelerator', the output token is determined to be 'yes', and if it is 'npu is an AI accelerator yes you are an', the output token can be determined to be 'expert'.

[0082] As described above with reference to FIGS. 1 and 2, the neural processing unit repeatedly performs numerous matrix multiplication operations (MatMul), vector operations, scalar operations, etc., for the inference operations of a transformer-based artificial neural network model, and the neural processing unit needs to be optimized for these operations. Below, the structure and operation of the neural processing unit (100) will be described.

[0083] FIG. 3 is a block diagram showing a neural processing unit that can be applied to the examples of the present disclosure.

[0084] Hereinafter, the neural processing unit (100) will be described in detail with reference to FIG. 3. The neural processing unit (100) is a dedicated processor specialized for Deep Neural Network (DNN) computations, separate from processors such as a central processing unit or a graphics processing unit. The neural processing unit (100) may be referred to as a Tensor Processing Unit (TPU), an AI accelerator, a Neural Signal Processor (NSP), a Neural Network Processor (NNP), etc.

[0085] The neural processing unit (100) may be a semiconductor implemented as an electrical / electronic circuit. That is, each component of the neural processing unit (100) may be a semiconductor circuit composed of numerous electronic elements (e.g., transistors, capacitors, electrical wiring, etc.). Therefore, some of these may be difficult to identify and distinguish with the naked eye and may be identified only by operation.

[0086] This neural processing unit (100) is intended to accelerate artificial neural network computations and is also called an artificial intelligence hardware accelerator. The artificial neural network can output computations of Transformer-based language models. However, it is not limited to this and can also process vision-based models such as YOLO or ResNet. It is mounted on a device (10000) and can output various information based on artificial intelligence. In addition to the neural processing unit (100), the device (10000) may include a central processing unit, a graphics processing unit, an application processor (AP), a microcontroller unit (MCU), etc. The device (10000) may include a microphone, a camera, a touchpad, a keyboard, etc. to receive input data, and may include a monitor, a speaker, etc. to output output data. The device (10000) may include, but is not limited to, a smartphone, a virtual assistant terminal located in a home or office, home appliances, a VCU or ADAS system of a car, a camera such as a CCTV, a robot, smart mobility, etc.

[0087] The neural processing unit (100) according to the examples of the present disclosure is specialized in accelerating the processing of an attention algorithm of a transformer-based neural network model at low power, and this will be explained in detail later. However, the neural processing unit (100) of the present disclosure is not limited to a transformer neural network and can process various neural network models such as a Convolutional Neural Network (CNN).

[0088] A neural processing unit (100) applicable to the examples of the present disclosure may include a controller (CTL) (110), a processing core (P_CORE) (120), a direct memory access unit (DMA) (130), an internal memory (I_MEM) (140), and a special function unit (SFU) (150).

[0089] The processing core (120), vector core (V_CORE) (153), and scalar core (S_CORE) (154) of the neural processing unit (100) may be computational circuits configured to efficiently process the attention layer of the neural network model. The attention layer can obtain an attention score (Q, K, V), which is a computational value, using parameters such as input values ​​and a query (Q), a key (K), and a value (V).

[0090] The processing core (120), vector core (153), and scalar core (154) are designed to process operations such as matrix multiplication, root mean square normalization, and softmax for attention operations. The tensor data form required for the above operations may be one of a multidimensional matrix form, a two-dimensional matrix form, a one-dimensional array form, or a scalar value, and one of the processing core (120), vector core (153), and scalar core (154) may be selected for the operation depending on the data form. Accordingly, the neural processing unit (100) can rapidly accelerate the operation of a generative artificial intelligence model with low power consumption by appropriately adopting a hardware module appropriate for the operation.

[0091] That is, the processing core (120) is capable of processing mathematical operations required for artificial neural network inference operations and may include an Arithmetic Logic Unit (ALU), a Multiply-Accumulate Unit (MAC), an Adder Tree, etc.

[0092] The controller (110) may be electrically connected to the processing core (120), the direct memory access unit (130), the internal memory (140), and the special function unit (150). The controller (110) may be configured to control operations related to neural network operations of the neural processing unit (100). The controller (110) may control each circuit included in the neural processing unit (100) to process inference operations of the neural network model based on executable code generated by a compiler (not shown). Here, the compiler (not shown) may be a semiconductor circuit or a software module operating in a separate device from the neural processing unit (100).

[0093] A compiler (not shown) can obtain executable code that can be executed on a neural processing unit (100) by compiling a specific neural network model. That is, the compiler (not shown) can generate executable code that can be executed exclusively on a specific neural processing unit having specific performance. The executable code may also be referred to as machine code or binary code. The compiler receives a neural network model written in a high-level AI framework such as TensorFlow or PyTorch and converts it into low-level machine code optimized for the NPU architecture of the present disclosure (e.g., integer operation core, special function unit).

[0094] The controller (110) can control the processing core (120), direct memory access unit (130), internal memory (140), and special function unit (150), etc., based on various instructions and operation schedules included in the execution code. The compiler (not shown) can receive information regarding the hardware characteristics of the neural processing unit (100) (e.g., operation performance of the processing core (120), operation performance of the vector core (153), operation performance of the scalar core (154), processing performance of the number system conversion unit (quantization unit (QUANT) (151), dequantization unit (DEQUANT) (152)), capacity of the internal memory (140), and the main memory (not shown) bandwidth of the corresponding bus (not shown), etc.). In addition, the compiler (not shown) can receive information regarding the structure of the neural network model to be processed by the neural processing unit (100), information on the algorithms included in the neural network model, and information on the size of the parameters. Then, the compiler (not shown) can generate execution code to control the read / write order of data required for neural network operations, the processing order of neural network operations, and whether each component of the neural processing unit (100) operates. The controller (110) receives the execution code from the compiler and can control the neural processing unit (100) based on the execution code. Each execution code can be generated corresponding to each neural network model, and the neural processing unit (100) can be configured to process at least one execution code.

[0095] The processing core (120) may include electronic circuits specialized for matrix multiplication, which accounts for the largest amount of computation among neural network computations. For example, the processing core (120) may be composed of multiple processing elements. For example, the processing core (120) may be designed as an array structure of M×N processing elements (where M and N are integers) or an adder tree structure, but is not limited thereto. The processing core (120) processes data in parallel based on multiple processing elements (PE) (not shown), and parameters (corresponding to input data) e.g. It can be configured to process matrix multiplication operations of weights, kernels, queries (Q), keys (K), values ​​(V), etc. To elaborate, the PE is composed of a MAC unit at its core that processes multiplication and accumulation operations in a single cycle, and functions as the basic unit of matrix operations.

[0096] To this end, processing elements (not shown) may each include a MAC (Multiply and Accumulate) operator. Thus, the processing core (120) can efficiently process matrix multiplication or convolution operations. To elaborate, the processing core (120) may be a circuit designed to simultaneously process M×N tensor data. Here, the number of processing cores (120) may be at least one. The processing core (120) may exchange data with the vector core (153), scalar core (154), internal memory (140), and main memory (not shown) of the special function unit (150). The processing core (120) may be configured to receive integer parameters as input and output integer parameters. The input integer parameters may include input parameters and weights. The input parameters may be configured to have a first bit width, and the weight parameters may be configured to have a second bit width. The output integer parameter may be configured to have a third bit width, and the third bit width may be configured to be larger than the first bit width or the second bit width.

[0097] As the number of processing elements (not shown) included in the processing core (120) increases, the number of transistors in the ASIC (Application Specific Integrated Circuit) increases, and the manufacturing cost may increase. If the circuit of the processing core (120) is designed to process integer parameters rather than floating-point numbers, the number of transistors in the ASIC can be significantly reduced, and the power consumption of the neural processing unit (100) can be significantly reduced.

[0098] The bit width of integer parameters that can be processed by the processing core (120) can be determined based on the hardware design of the processing element (not shown). This is described later in FIG. 4. The number of processing elements (not shown) may range from hundreds to tens of thousands. To elaborate, the number of transistors included in a processing element designed to process floating-point parameters is about four times greater than the number of transistors included in a processing element designed to process integer parameters. Therefore, if the circuit of the processing core (120) of the neural processing unit (100) according to the examples of the present disclosure is designed to exclude floating-point parameter operations and only process integer parameters, the number of transistors can be reduced by about 75% compared to the floating-point operation circuit. In this embodiment, the number of processing elements (not shown) included in the processing core (120) can be easily increased in the same chip area compared to floating-point, and there is an effect of improving computation processing performance with low power while reducing the size of the semiconductor chip.

[0099] The special function unit (150) may refer to a collection of various circuit sections configured to process various neural network operations that are not processed by the processing core (120). That is, it is a circuit capable of accelerating special operations that are difficult to perform in the ALU. The special function unit (150) can perform operations such as exponentiation, logarithmic functions (log, ln), square roots (sqrt), reciprocals and inverse square roots (reciprocal, 1 / sqrt(x)), trigonometric functions (Sin, Cos, Tan, ArcSin, ArcCos), and quantization (INT8 / FP16 conversion). For example, the special function unit (150) may include a quantization section (151), an inverse quantization section (152), a vector core (153), and a scalar core (154). The quantization section (151) and the inverse quantization section (152) may be collectively referred to as a number system conversion section, but are not limited thereto.

[0100] The quantization unit (151) of the special function unit (150) can convert floating point (FP) data into integer (INT) data, and the inverse quantization unit (152) can inversely convert integer data into floating point data.

[0101] The vector core (153) of the special function unit (150) may refer to a set of circuits that process at least some of the operations of the artificial intelligence model, excluding matrix multiplication processed by the processing core (120). The vector core (153) may be designed to include a plurality of specialized vector operation circuits. For example, the vector core (153) may include a vector maximum value operation unit (V_MAX) (1531), a vector multiplication unit (V_MUL) (1532), a vector addition unit (V_ADD) (1533), a MAC operation unit (V_MAC) (1534), a mask unit (V_MASK) (1535), a rounding clipping unit (RND_CLIP) (1536), and a vector function approximation unit (V_FunApx) (1537). The operation of each module of the vector core (153) will be described later with reference to FIG. 5.

[0102] The scalar core (154) of the special function unit (150) may refer to a set of circuits that process at least some of the remaining operations among the operations of the artificial intelligence model, excluding those processed by the processing core (120) and the vector core (153). The scalar core (154) may be designed to include a plurality of specialized scalar operation circuits. For example, the scalar core (154) may include a scalar maximum value operation unit (S_MAX) (1541), a scalar multiplication unit (S_MUL) (1542), a scalar addition unit (S_ADD) (1543), a scalar summation unit (S_SUM) (1544), and a scalar function approximation unit (S_FunApx) (1545). The operation of each of the modules of the scalar core (154) will be described later with reference to FIG. 6.

[0103] The direct memory access unit (130) can access at least one memory electrically connected to the neural processing unit (100) to perform data read / write operations. The at least one memory may include at least one of a main memory (not shown), a shared memory (not shown), a storage device (not shown), etc. The neural processing unit (100) can transmit various data related to artificial intelligence operations to at least one memory (not shown) or read data from at least one memory (not shown) through the direct memory access unit (130). The direct memory access unit (130) may be configured to perform operations such as setting the address of the internal memory (140), generating read / write commands, and controlling. The direct memory access unit (130) may be configured to control at least one memory by being connected to at least one communication bus.

[0104] Internal memory (140) is memory placed in the on-chip area of ​​the neural processing unit (100). Internal memory (140) may include memory for caching or storing data processed in the on-chip area and / or a register file for storing instructions for running a compiled neural network model. The register file may also be configured to be included in the controller (110). For example, when a memory (not shown) connected to a bus is provided, internal memory (140) may read and store data required for computation from said memory. Internal memory (140) may include SRAM, a register file, a DRAM, a resistive RAM, a magneto-resistive RAM, a phase-change RAM, a ferroelectric RAM, or a flash memory. Internal memory (140) may be composed of at least one memory unit. Internal memory (140) may be composed of a homogeneous memory unit or a heterogeneous memory unit. The capacity of the internal memory (140) may be several megabytes to tens of megabytes, but is not limited thereto.

[0105] The quantization unit (151) of the special function unit (150) may be configured to include a quantization circuit. The inverse quantization unit (152) may be configured to include an inverse quantization circuit. The quantization unit (151) and the inverse quantization unit (152) may be referred to as a number system conversion unit.

[0106] According to one embodiment, the input and output of the processing core (120) may be in the form of integers. The input and output of the vector core (153) and the scalar core (154) may be in the form of floating-point numbers. The quantization unit (151) and the dequantization unit (152) can convert data into the data form (number system) required by each hardware during data transmission between the processing core (120), the vector core (153), and the scalar core (154). That is, the quantization circuit of the quantization unit (151) can convert floating-point parameters into integer parameters. The dequantization circuit of the dequantization unit (152) can convert integer parameters into floating-point parameters.

[0107] According to the examples of the present disclosure, floating-point numbers are not limited to IEEE standard 754, and Brain Floating Point, Dynamic Floating-Point, Variable Precision Floating-Point (VPFP), Custom Floating-Point (CFP), or Flexible Floating-Point (FFP) may be applied to improve computational efficiency, reduce memory usage, reduce power consumption, etc. of the neural processing unit (100). VPFP is a floating-point format that allows the bit width of the exponent and mantissa to be dynamically set or adjusted. Using VPFP allows for the optimization of power and performance by allowing flexible precision levels based on specific calculations. Using a dynamic floating-point format, which is used interchangeably with variable precision, allows for the immediate adjustment of mantissa and exponent sizes during runtime, enabling real-time response to data characteristics. CFP represents an application-specific format where the bit allocation for the mantissa and exponent is adjusted to meet the requirements of a specific use case. This may be useful for neural processing unit (100) implementations that select an accurate bit width to optimize hardware resources and data accuracy. FFP refers to a floating-point format that allows for adjustable bit allocation between the exponent and mantissa. FFP may be implemented in a neural processing unit (100) to provide an optimal balance between range and precision. The bit width of the floating-point number according to the examples of the present disclosure may be between 4 bits and 32 bits. According to the examples of the present disclosure, the bit width of the integer may be between 4 bits and 32 bits.

[0108] The neural processing unit (100) disclosed in the present disclosure may provide at least one of power gating, clock gating, and register retention functions that enable the neural processing unit (100) to operate at low power.

[0109] Power gating is a technology that eliminates leakage power by cutting off power to unused circuit blocks. The circuit blocks to which power gating is applied are provided with power switches. Accordingly, specific circuit blocks of the neural processing unit (100) can be disconnected from the power supply using the power switches. Power gating can eliminate both dynamic power and leakage power, thereby enabling the device to operate in a low-power mode.

[0110] Clock gating is a technology that reduces dynamic power consumption by blocking clock signals from unused circuit blocks. Circuit blocks to which clock gating is applied can selectively block clock signals input to specific blocks using clock control logic. When clock gating is applied, power is continuously supplied to the Neural Processing Unit (NPU), allowing the circuit's state to be maintained. Since clock gating does not turn off the power, operation can resume immediately upon reactivating the clock, thereby reducing dynamic power consumption by preventing unnecessary switching activities.

[0111] Register retention is a technology designed to allow a specific register of the neural processing unit (100) to maintain its state even when the power is turned off. Registers to which register retention is applied can retain data stored in the register using only low power in sleep mode. Therefore, even if the register is switched to power-saving mode, important register values ​​are retained, so initialization is not required upon return. Register retention can provide low-power operation and can retain data using extremely little power even when the power of the neural processing unit (100) is turned off.

[0112] The hardware operation between each component of the neural processing unit (100) of FIG. 3 will be explained in detail below based on FIG. 8.

[0113] FIG. 4 is a block diagram illustrating a processing element that can be applied to examples of the present disclosure.

[0114] Referring to FIG. 4, the processing element (121) may be configured to include a multiplier (1211), an adder (1212), and an accumulator (1213). The processing element (121) of FIG. 4 may be one of a plurality of processing elements included in the processing core (120) of FIG. 3, and the processing element (121) may refer to each processing element in the following description.

[0115] The multiplier (1211) multiplies the input (N)-bit data and (M)-bit data. The operation value of the multiplier (1211) is output as (N+M)-bit data. Here, N and M are integers greater than 0. The first input unit may be configured to receive (N)-bit data. The second input unit may be configured to receive (M)-bit data. For example, the first input unit may be configured to receive input parameters and the second input unit may be configured to receive weight parameters.

[0116] The bit width of the parameters input to the first input and second input sections of the processing element (121) can be determined when the neural processing unit (100) compiles the current neural network model being processed to obtain executable code. That is, the bit width of the input parameters and the bit width of the weight parameters of the neural network model can be determined at the compilation stage by the compiler. For example, the bit widths of the input parameters and the weight parameters can be quantized equally, and the bit widths can be 32 bits, 16 bits, 8 bits, 4 bits, etc., but the present disclosure is not limited thereto. For example, the bit width of the input parameters and the bit width of the weight parameters can be determined differently from each other, and the parameters can be quantized based on their respective bit widths. For example, the bit width of the input parameters and the bit width of the weight parameters can be quantized to 16 bits and 8 bits, respectively. For example, the bit width of the input parameters and the bit width of the weight parameters can be quantized to 8 bits and 4 bits, respectively. That is, the bit widths of the parameters input to each input of the processing element (121) may differ from one another. The processing element (121) may include quantization information of the data input to each input in the execution code.

[0117] The accumulator (1213) accumulates the operation value of the multiplier (1211) and the operation value of the accumulator (1213) using the adder (1212) for (L) loops. Accordingly, the bit width of the data in the output and input sections of the accumulator (1213) can be output as (N+M+log2(L)) bits. Here, L is an integer greater than 0. When accumulation is finished, an initialization signal (INTIALIZATION RESET) is input so that the data stored inside the accumulator (1213) can be initialized to 0. The accumulator (1213) of the processing element (121) is configured to maintain the accumulated value when zero skipping is enabled. Based on the maximum value that can be accumulated in the accumulator (1213), the output data (X) bit can be set to a bit width such that no overflow of the output data (X) bit occurs. For example, (X) bit can be 16 bits to 64 bits.

[0118] To elaborate, the quantization unit (151) of the special function unit (150) can convert an integer parameter output from the processing core (120) into a floating-point number and transmit it to the vector core (153), scalar core (154), and / or internal memory (140). The quantization unit (151) can be controlled under a control signal from the controller (110).

[0119] The controller (110) can restrict the operation of the multiplier (1211) so that it does not perform operations (e.g., zero-skipping operation) based on the fact that when 0 is input to one of the first input and second input sections of the multiplier (1211), the result of the operation becomes 0 even without performing the operation. For example, when 0 is input to one of the first input and second input sections of the multiplier (1211) of the processing element (121), the multiplier (1211) can operate in a zero-skipping manner. For zero-skipping, each processing element (121) included in the processing core (120) can be enabled or disabled. The controller (110) can provide an enable or disable signal (CLOCK-GATING CONTROL) to each processing element (121) on a clock-by-clock basis. When the processing element (121) is disabled, the multiplier (1211) is disabled. Accordingly, the power consumed in the operation of the multiplier (1211) can be reduced. An example of the power consumption of the multiplier (1211) can be shown in FIG. 7.

[0120] A processing element (121) may be designed to receive a control signal (CLOCK-GATING CONTROL) from a controller (110) for zero-skipping operation control (i.e., activation or deactivation). Specifically, a multiplier (1211) of the processing element (121) may be designed to receive a respective control signal (CLOCK-GATING CONTROL) for zero-skipping operation control from the controller (110). Alternatively, an adder (1212) of the processing element (121) may be modified to receive a control signal (CLOCK-GATING CONTROL) for zero-skipping operation control from the controller (110). Alternatively, each multiplier (1211) and adder (1212) of the processing element (121) may be modified to receive a respective control signal (CLOCK-GATING CONTROL) for zero-skipping operation control simultaneously from the controller (110).

[0121] The controller (110) can optimize the operation path by pre-reflecting and integrating a multiplication operation (FpMul) based on the normalization coefficients included in the weights of the neural network model at compile time before the softmax operation is performed, and omitting the multiplication operation at runtime. This corresponds to a static optimization method for reducing computational efficiency and power consumption by eliminating unnecessary runtime multiplication operations.

[0122] Additionally, the controller (110) may selectively control whether to perform a multiplication operation between a query vector and a key vector depending on the type of operation request (e.g., attention operation, feedforward operation, etc.) or the timing of the operation (e.g., decoding phase, learning phase, etc.). This configuration may be implemented by controlling the clock supply of a processing element (121) including a multiplier (1211) or by bypassing the operation path itself to skip the operation.

[0123] Such control may be processed within the neural processing unit (100) or may be performed through an integrated controller in a system-on-chip (SoC) architecture that includes multiple computation units. In this case, the controller (110) may be configured to dynamically control whether computation is enabled for each computation unit by including clock gating or power control signals for each unit.

[0124] FIG. 5 is a block diagram showing a vector core of a special function unit that can be applied to the examples of the present disclosure.

[0125] The vector core (153) may refer to a set of circuits that process some of the operations of the artificial intelligence model, excluding matrix multiplication, which is processed by the processing core (120). The vector core (153) differs from the processing core (120) in that it is composed of floating-point arithmetic circuits. That is, the vector core (153) may be configured to receive floating-point parameters as input and output floating-point parameters. The vector core (153) may be designed to be pipelined with the processing core (120) and the scalar core (154) for computational efficiency with the processing core (120) and the scalar core (154). Therefore, by allowing the processing core, vector core, and scalar core to process different data simultaneously like a conveyor belt, the total computational throughput can be maximized and idle time minimized. To elaborate, the vector core (153) is configured to exchange data with the processing core (120), scalar core (154), internal memory (140), and main memory (not shown). To elaborate, the vector core (153) can be specialized to efficiently process a portion of the attention operations of a transformer-based neural network model. That is, the vector core (153) may include a plurality of specialized vector operation circuits.

[0126] For example, the vector core (153) may be designed to include a vector maximum value operation unit (V_MAX) (1531), a vector multiplication unit (V_MUL) (1532), a vector addition unit (V_ADD) (1533), a MAC operation unit (V_MAC) (1534), a mask unit (V_MASK) (1535), a rounding clipping unit (RND_CLIP) (1536), and a vector function approximation unit (V_FuncApx) (1537). The vector maximum value operation unit (1531) is configured to include a circuit that processes a maximum value operation of the input tensor data. The vector multiplication unit (1532) is configured to include a circuit that processes a multiplication operation of the input tensor data. The vector addition unit (1533) is configured to include a circuit that processes an addition operation of the input tensor data. The MAC operation unit (1534) is configured to include a circuit that processes Multiply And Accumulate (MAC) operations on the input tensor data. The mask unit (1535) is configured to include a circuit that processes masking operations on the input tensor data. The rounding clipping unit (1536) is configured to include a circuit that processes rounding and clipping operations on the input tensor data. The vector function approximation unit (1537) is configured to include a circuit that processes approximation operations of various functions and can be applied to exponential function operations or vector-unit weighted sum operations performed during softmax operations. In particular, the vector core (153) can be configured to generate an input vector for softmax operations by multiplying the inner product result between a query vector and a key vector by a normalization factor, create a stabilized softmax distribution by subtracting the maximum value from each element of the generated input vector, and output a final attention result by multiplying the softmax operation result by a value vector. Here, the normalization factor may be a constant value calculated at compile time and stored in memory.This operation flow can be implemented around circuits such as a vector maximum value operation unit (1531), a vector multiplication unit (1532), a vector addition unit (1533), and a vector function approximation unit (1537), and can be configured in a pipelined and parallel processing manner to consider operation efficiency. If necessary, it can be further configured to include an auxiliary operation flow before and after the softmax operation by including a MAC operation unit (1534) for Multiply-And-Accumulate operation, a mask unit (1535) for masking processing, or a rounding clipping unit (1536) for output precision correction.

[0127] Below, the vector function approximation part (1537) will be described in more detail.

[0128] For example, the vector core (153) may be a circuit designed to simultaneously process M×1 tensor data. The vector core (153) may be designed to receive M×1 tensor data as input and output M×1 tensor data. Here, M of the M×1 tensor data received by the vector core (153) may be the same as M, which is the number of rows of M×N tensor data received by the processing core (120). That is, a pipeline circuit design can be implemented by matching the size of a specific dimension of the tensor data of the processing core (120) and the vector core (153) respectively. To elaborate, the vector core (153) may include a register file for storing one-dimensional array data. Therefore, the vector core (153) can improve computational processing efficiency by processing the tensor data output from the processing core (120) in units of a specific dimension. As described above, the vector core (153) is configured to process various function operations using a floating-point arithmetic unit. Therefore, if the vector core (153) is designed to process the same tensor size as the processing core (120), the semiconductor chip size and power consumption of the neural processing unit (100) may increase. However, since the vector core (153) is designed to process data in the form of a one-dimensional array, the increase in the semiconductor chip size of the neural processing unit (100) can be reduced, power consumption can be reduced, and the bottleneck of neural network computation can be reduced by being pipelined with the processing core (120). In some examples, the vector core (153) may be designed to receive L×1 size tensor data as input. Here, L may be a power of 2 value within a specific range based on M. For example, when M is 64, L may be 8, 16, 32, 64, 128, or 256.

[0129] FIG. 6 is a block diagram showing a scalar core of a special function unit that can be applied to the examples of the present disclosure.

[0130] The scalar core (154) may refer to a set of circuits that process some of the operations of the neural network model, excluding the operations processed by the processing core (120) and the vector core (153). The scalar core (154) differs from the processing core (120) in that it is composed of floating-point arithmetic circuits. That is, the scalar core (154) may be configured to receive floating-point parameters as input and output floating-point parameters. The scalar core (154) may be designed to be pipelined with the processing core (120) and the vector core (154) for computational efficiency with the processing core (120) and the vector core (153). To elaborate, the scalar core (154) is configured to exchange data with the processing core (120), the vector core (153), the internal memory (140), and the main memory (not shown). To elaborate, the scalar core (154) can be specialized to efficiently process part of the attention operation of a transformer-based neural network model. That is, the scalar core (154) may include a plurality of specialized scalar operation circuits.

[0131] For example, the scalar core (154) may be designed to include a scalar maximum value operation unit (1541), a scalar multiplication unit (1542), a scalar addition unit (S_ADD) (1543), a scalar summation unit (S_SUM) (1544), and a scalar function approximation unit (S_FuncApx) (1545). The scalar maximum value operation unit (1541) is configured to include a circuit that processes a maximum value operation of input tensor data. The scalar multiplication unit (1542) is configured to include a circuit that processes a multiplication operation of input tensor data. The scalar addition unit (1543) is configured to include a circuit that processes an addition operation of input tensor data. The scalar summation unit (1544) is configured to include a circuit that accumulates a plurality of input scalar values ​​or performs an aggregation operation according to a certain range or condition. The scalar function approximation unit (1545) is configured to include circuits that process approximation operations of various functions. These function approximation operations can be applied to the calculation of correction coefficients, such as the approximation of exponential functions used in softmax operations and the reciprocal operation of the total sum. In other words, the scalar core (154) can be configured to calculate the maximum value among the results of the product operation between the normalized query vector and the key vector, calculate the total sum of the exponential function, and then calculate the reciprocal to calculate the correction coefficient for the softmax operation.

[0132] Below, the scalar function approximation part (1545) will be described in more detail.

[0133] For example, the scalar core (154) may be a circuit designed to process M×1 size tensor data simultaneously. The scalar core (154) may be designed to receive M×1 size tensor data as input and output 1×1 size tensor data. The scalar core (154) may be designed to receive M×1 size tensor data as input and output 1×1 size scalar data. Here, the M×1 size of the tensor data received by the scalar core (154) may be the same as the M×1 size received by the vector core (153). That is, a pipeline circuit design can be implemented by matching the size of a specific dimension of the tensor data of the vector core (153) and the scalar core (154) respectively. To elaborate, the scalar core (154) may include a register file for storing one-dimensional array data. Accordingly, the scalar core (154) can improve computational processing efficiency by processing tensor data output from the vector core (153) in units of specific dimensions. As described above, the scalar core (154) is configured to process various function operations as a floating-point arithmetic unit. Therefore, if the scalar core (154) is designed to process the same tensor size as the processing core (120), the semiconductor chip size and power consumption of the neural processing unit (100) may increase. However, since the scalar core (154) is designed to process data in a scalar form, the increase in the semiconductor chip size of the neural processing unit (100) can be reduced, power consumption can be reduced, and bottlenecks in neural network operations can be reduced by being pipelined with the vector core (153). In some examples, the scalar core (154) may be designed to receive L×1 size tensor data as input. Here, L can be a power of 2 value within a specific range based on the above M. For example, when M is 64, L can be 8, 16, 32, 64, 128, 256.

[0134] In some embodiments, the vector core (153) shown in FIG. 5 and the scalar core (154) shown in FIG. 6 may be transformed into a single integrated 'vector-scalar operation unit' with increased integration density by sharing functionally overlapping circuits. This integrated architecture can improve the efficiency of hardware resources by sharing operation circuits of similar functions, going beyond simply physically placing the two units adjacent to each other. Referring to FIG. 5, the vector core (153) is designed to perform parallel operations on each element of a vector composed of multiple data elements (e.g., M). This can be conceptually understood as a structure in which M independent scalar operation lanes are arranged in parallel. On the other hand, the scalar core (154) of FIG. 6 processes a single data (1×1 scalar). Therefore, in this embodiment, based on the structural similarity, it may be configured to perform scalar operations by selectively activating only some lanes of the vector operation circuit according to a control signal of the control unit (110). According to the configuration described above, the integrated vector-scalar operation unit can be implemented as an optional resource sharing architecture in which circuits are shared so that existing vector operators replace scalar operators with redundant functions, and only scalar operations with specialized functions, such as reduction, are maintained separately. This allows for a significant improvement in power-to-performance ratio (PPA) by providing all functions identically while reducing the total number of gates and chip area of ​​the special function unit (SFU, 150).

[0135] FIG. 7 is a diagram schematically showing the energy consumption per unit operation of a neural processing unit that can be applied to the examples of the present disclosure.

[0136] The power consumption of the processing core (120), vector core (153), and scalar core (154) of the neural processing unit (100) will be explained below with reference to FIG. 7.

[0137] FIG. 7 schematically illustrates the energy consumed per unit operation of various arithmetic units of a neural processing unit (100). For example, energy consumption can be described by dividing it into addition operations and multiplication operations. However, the energy consumed per unit operation may vary depending on the semiconductor chip's foundry process technology (e.g., 2nm technology, 5nm technology, 7nm technology, 14nm technology, 28nm technology, etc.) or the technological capability of a specific foundry company.

[0138] The processing core (120) may be designed to perform operations on integer parameters of a specific bit width. "8b INT Add" in FIG. 7 refers to an 8-bit integer addition operation of the adder (1212) of the processing element (121). An 8-bit integer addition operation may consume 0.03 pj of energy. "16b INT Add" refers to a 16-bit integer addition operation of the adder (1212) of the processing element (121). A 16-bit integer addition operation may consume 0.05 pj of energy. "32b INT Add" refers to a 32-bit integer addition operation of the adder (1212) of the processing element (121). A 32-bit integer addition operation may consume 0.1 pj of energy. "8b INT Mult" refers to an 8-bit integer multiplication operation of the multiplier (1211) of the processing element (121). An 8-bit integer multiplication operation can consume 0.2 pj of energy. "32b INT Mult" refers to a 32-bit integer multiplication operation of the multiplier (1211) of the processing element (121). A 32-bit integer multiplication operation can consume 3.1 pj of energy.

[0139] To elaborate, when the processing core (120) is composed of thousands to tens of thousands of processing elements designed as integer arithmetic units, the power consumption of the neural processing unit (100) can be significantly reduced compared to when the processing core is composed of the same number of processing elements designed as floating-point arithmetic units. Generally, since most of the computation in a generative artificial intelligence model is processed in the processing core (120), the design of an arithmetic unit with a specific integer bit width is important for the low-power design of the neural processing unit (100).

[0140] The vector core (153) and scalar core (154) of the special function unit (150) may be designed to perform operations on floating-point parameters. "16b FP Add" refers to a 16-bit floating-point addition operation of the adder of the special function unit (150). A 16-bit floating-point addition operation may consume 0.4 pj of energy. "32b FP Add" refers to a 32-bit floating-point addition operation of the adder of the special function unit (150). A 32-bit floating-point addition operation may consume 0.9 pj of energy. "16b FP Mult" refers to a 16-bit floating-point multiplication operation of the multiplier of the special function unit (150). A 16-bit floating-point multiplication operation may consume 1.1 pj of energy. "32b FP Mult" refers to a 32-bit floating-point multiplication operation of the multiplier of the special function unit (150). A 32-bit floating-point multiplication operation can consume 3.7 pj of energy. To elaborate, if the special function unit (150) is designed with floating-point arithmetic units, designing the number of arithmetic units of the special function unit (150) to be relatively smaller than the number of processing elements (121) of the processing core (120) is advantageous in terms of power consumption of the neural processing unit (100) and miniaturization of the semiconductor chip size. Therefore, the number of floating-point arithmetic units of the special function unit (150) can be designed to be smaller than the number of integer arithmetic units of the processing core (120).

[0141] According to one embodiment, the computational efficiency of the neural processing unit (100) can be improved by implementing a data pipeline circuit design of a processing core (120), a vector core (153), and a scalar core (154) so ​​that the array size of a specific dimension of tensor data processed by the neural processing unit (100) is compatible with each other among hardware components.

[0142] FIG. 8 is a flowchart illustrating a method for controlling a neural processing unit that can be applied to examples of the present disclosure.

[0143] Referring to FIG. 8, the neural processing unit (100) performs integer operations based on input data through the processing core (120) in response to an input neural network operation request (S110). Here, the integer operations may include, for example, operations such as vector-matrix multiplication (MatMul), vector accumulation, or integer-based activation functions on quantized tensor data, and these operations may be performed in parallel in a plurality of processing elements arranged inside the processing core (120) of FIG. 3. These operations are mainly performed in parallel in units of a plurality of processing elements inside the processing core (120), and by processing the input data and neural network weights in an integer format (INT8, INT16, etc.), high speed and low power characteristics can be satisfied simultaneously.

[0144] Next, the neural processing unit (100) performs a floating-point-based special operation on the integer operation result performed by the processing core (120) in step S110 through a special function unit (150) (S120). At this time, the special function unit (150) may be configured to perform, for example, activation function operations (ReLU, GELU, tanh, etc.), normalization, softmax, or other non-linear function approximation operations on the integer operation result converted to a floating-point number.

[0145] At this time, the special function unit (150) is configured to include a quantization unit (151), an inverse quantization unit (152), a vector core (153), and a scalar core (154), as shown in FIG. 3. The inverse quantization unit (152) inversely quantizes integer data into floating-point formats such as FP16 and BFLOAT, and the converted data is input into non-linear function operations. Additionally, the vector core (153) performs operations such as softmax, normalization, ReLU, and GELU in a high-speed parallel manner and is suitable for multi-dimensional tensor data. On the other hand, the scalar core (154) is used for conditional branching, precise function approximation, and single-value-based calculations, and may include an FSM-based control circuit for complex logic processing. These operations are suitable for high-precision calculations or non-linear activation processing that are difficult to implement with integer-based operations, and contribute to ensuring the accuracy of the entire operation flow.

[0146] Next, the neural processing unit (100) checks the operation status of each integer operation by step S110 and special operation by step S120 through the controller (110), and controls the operation of the processing core (120) or special function unit (150) according to the operation status (S130). At this time, as shown in FIG. 3, the controller (110) communicates with each module and controls the operation clock of each unit by considering the operation load, operation priority, operation condition flag, etc., or can readjust the order of operations if necessary. For example, if the input data consists entirely of zeros, power consumption can be minimized by performing clock gating or power gating by omitting the corresponding integer operation or floating-point operation. In addition, operation scheduling can be optimized based on the decoding information of the execution instruction to prevent resource conflicts between operation modules or pipeline bottlenecks.

[0147] In this way, the control flow of FIG. 8 operates in close coordination with each hardware component of FIG. 3 and is designed with a structure that can improve power efficiency along with improved computational performance.

[0148] FIG. 9 is a block diagram illustrating a system-on-chip according to a first embodiment of the first example of the present disclosure.

[0149] Referring to FIG. 9, an apparatus (10000) according to a first embodiment of a first example of the present disclosure is described. The apparatus (10000) includes a circuit board (BD), a memory (2000), and a system-on-chip (1000). The memory (2000) and the system-on-chip (1000) may be placed on a circuit board (not shown). The system-on-chip (1000) is configured to include a neural processing unit (100), a central processing unit (200), a first bus (BUS_1) (310), and a second bus (BUS_2) (320). The memory (2000) is configured to be electrically connected to the first bus (310). Each example and each embodiment of the present disclosure may be modified and combined with other examples and other embodiments of the present disclosure.

[0150] The system-on-chip (1000) may refer to a system manufactured from a single semiconductor substrate. The system-on-chip (1000) may be configured to further include a package (not shown) that protects the semiconductor substrate.

[0151] The neural processing unit (100) may be configured to communicate with the memory (2000). A first bus (310) may be provided between the neural processing unit (100) and the memory (2000). For example, the first bus (310) may be an AXI (Advanced Extensible Interface) bus. However, the examples of the present disclosure are not limited thereto. The first bus (310) may be configured to support a read and write address / control interface between the memory (2000) and the neural processing unit (100). The first bus (310) may be configured as an independent data communication channel to support high-speed data transmission for transaction processing in addition to memory control commands. The first bus (310) may be configured as a dedicated bus for transmitting input parameters, output parameters, and weight parameters of a neural network model, and it is desirable to appropriately design the bandwidth of the first bus (310) considering the parameter sizes of the neural network models to be processed.

[0152] The neural processing unit (100) may be configured to communicate with the central processing unit (200). A second bus (320) may be provided between the neural processing unit (100) and the central processing unit (200). For example, the second bus (320) may be an Advanced High-Performance Bus (AHB). However, the examples of the present disclosure are not limited thereto. The second bus (320) may be provided for efficient communication between the neural processing unit (100), the central processing unit (200), and peripheral devices of the system-on-chip (1000). The second bus (320) may provide a master-slave architecture to improve data throughput. The second bus (320) may be configured to support burst transmission and pipeline computation operations of the system-on-chip (1000). The second bus (320) may be configured to provide an on-the-fly function. The second bus (320) can be configured so that the master of the second bus (320) can execute new commands on the fly without waiting while the central processing unit (200) or the neural processing unit (100) reads or writes data in real time. Thus, real-time data processing of the neural processing unit (100) can be optimized, and at least one core of the neural processing unit (100) can be configured to use the bus independently. Thus, the second bus (320) can support the system-on-chip (1000) in performing various commands in real time.

[0153] To elaborate, the first bus (310) provides a higher memory bandwidth than the second bus (320), and the second bus (320) provides an on-the-fly function, thereby allowing the central processing unit (200) to dynamically control the neural processing unit (100). Accordingly, the system-on-chip (1000) can dynamically process various input queries to the generative neural network model.

[0154] Refer to FIGS. 3 to 7 for a description of the neural processing unit (100). Accordingly, a redundant description of the neural processing unit (100) is omitted.

[0155] The central processing unit (200) may be configured to control the neural processing unit (100) based on the execution code of the artificial intelligence model. The execution code may be generated by a compiler (not shown). The central processing unit (200) may control the neural processing unit (100) by directly setting the register values ​​of each circuit of the neural processing unit (100) based on the execution code. Here, the execution code may include each register value. Additionally, the central processing unit (200) may offload and process operations that are not processed by the neural processing unit (100). In this case, the neural network model may be compiled to offload specific operations of the neural network model to the central processing unit (200), and an execution code may be obtained.

[0156] The memory (2000) may include, but is not limited to, a main memory located outside the system-on-chip (1000). The memory (2000) may be electrically connected to the system-on-chip (1000) via a first bus (310). The memory (2000) of the device (10000) according to the first embodiment of the first example of the present disclosure may include at least one memory such as SRAM, LPDDR RAM, Resistive RAM, Magneto-resistive RAM, Phase-change RAM, Ferroelectric RAM, Flash Memory, HBM, etc. The memory (2000) may be composed of at least one memory unit (e.g., a bank, etc.). The memory (2000) may be composed of homogeneous memory or heterogeneous memory. It is preferable that the capacity of the memory (2000) be greater than the total size of the weight parameters of the neural network model. In this case, all of the weight parameters of the neural network model can be loaded into the memory (2000) at once and reside there. If the storage capacity of the memory (2000) is insufficient, only some of the weight parameters are loaded, so it may be difficult to process the inference operations of the neural network model quickly in real time. For example, the capacity of the memory (2000) may be one of 4 GByte or 8 GByte. That is, the capacity of the memory (2000) may be 4 GByte to 8 GByte, and may be determined by considering the size of the parameters of the neural network model to be run on the system-on-chip (1000).

[0157] That is, the main memory (2000) is provided as dedicated memory for the neural processing unit (100) and can be configured to sufficiently store and access at high speed all essential data, including all learned weights of the neural network model as well as activation values ​​generated during the inference operation and used in the next operation. In the past, for large-scale models, some of the weights or activation values ​​had to rely on external slow memory or servers, whereas the present disclosure keeps all computation data resident within the on-device main memory, thereby minimizing data movement bottlenecks and eliminating external dependencies. This establishes a foundation for completing and executing the entire inference process of the neural network model on the device itself (server-independent operation) without exchanging data with an external server, which provides significant advantages in terms of real-time responsiveness and data privacy. This memory structure supports the neural processing unit (100) to process even complex models stably without delay. As a result, a true on-device AI environment can be realized.

[0158] The bandwidth of the first bus (310) can be determined according to the processing performance of the neural processing unit (100) (e.g., Tera Operations Per Second (TOPS) of the neural processing unit (100)). For example, a bandwidth of 2 GB / s to 4 GB / s of memory (2000) per 1 TOPS may be required. For example, the memory (2000) may be configured to provide a bus bandwidth of 20 GB / s to 40 GB / s for 10 TOPS of neural processing units (100). The bandwidth of the first bus (310) can be determined according to the operating frequency of the memory (2000) and the number of communication channels. For example, the following formula may be used to configure a memory bandwidth of 40 GB / s using LPDDR5.

[0159] Memory Bandwidth (GB / s) = (Data Rate (Gbps per pin) × Bus Width (bits) × Number of Channels × 8).

[0160] To elaborate, LPDDR5 generally supports a maximum data rate of 6400 Mbps per pin. The bus width of an LPDDR channel is generally 16 bits (2 bytes). However, the number of channels cannot be a decimal point and is an integer. However, the examples of the present disclosure are not limited to the bandwidth of the first bus (310).

[0161] The device (10000) according to the first embodiment of the first example of the present disclosure may be configured to provide a low-power function for use on a device.

[0162] The low-power mode of the device (10000) according to the first embodiment of the first example of the present disclosure may be provided, for example, a sleep mode, a deep sleep mode, a retention mode, etc. In some examples, the low-power mode may be referred to as a standby mode, a hibernation mode, an idle mode, etc.

[0163] A sleep mode can be implemented, for example, by applying clock gating to at least one of the processing core (120), vector core (153), and scalar core (154) of the neural processing unit (100).

[0164] Deep sleep mode can be implemented, for example, by applying power gating to at least one of the processing core (120), vector core (153), scalar core (154), or memory (2000) of the neural processing unit (100).

[0165] The retention mode can be implemented by supplying a minimum retention voltage to the controller (110) or memory (2000) of the neural processing unit (100).

[0166] For example, the neural processing unit (100) of the device (10000) according to the first embodiment of the first example of the present disclosure may be configured to provide at least one of power gating, clock gating, and register retention functions. Further details, when the device (10000) enters sleep mode, the neural processing unit (100) may activate the clock gating function. Further details, when the device (10000) enters deep sleep mode, the neural processing unit (100) may activate the power gating function. Further details, when the device (10000) enters retention mode, the supply voltage input to the neural processing unit (100) may be reduced or the operating frequency may be reduced.

[0167] For example, the memory (MEM) of the device (10000) according to the first embodiment of the first example of the present disclosure may provide at least one of deep sleep, retention, and maintain parameters functions. To elaborate, when the device (10000) enters deep sleep mode, the memory (2000) may be power-gated, and in the case of a volatile memory device, data may be lost. To elaborate, when the device (10000) enters retention mode, the supply voltage input to the memory (2000) may be reduced or the operating frequency may be reduced, and in the case of a volatile memory device, data may be maintained. To elaborate, when the device (10000) enters parameter maintenance mode, the supply voltage input to the memory (2000) may be reduced or the operating frequency may be reduced, and in the case of a DRAM, the refresh period of the memory cell may be increased, and in the case of a volatile memory device, data may be maintained.

[0168] A low-power mode may be activated when no input query for speculative decoding is entered into the device (10000) for a certain period of time or longer. The certain period may be provided, for example, based on a timer, counter value, etc. The low-power mode may be activated based on a specific threshold time and may be deactivated when an input query processed by speculative decoding is entered into the device (10000). The state in which the low-power mode is deactivated may be referred to as a wake-up state.

[0169] The device (10000) according to the first embodiment of the first example of the present disclosure may be a device capable of operating in a battery environment. Accordingly, the neural processing unit (100) may be designed to operate at battery voltage.

[0170] A device (10000) according to the first embodiment of the first example of the present disclosure may be configured to provide at least one low-power mode while simultaneously providing speculative decoding in an on-device environment. The device (10000) may enter a low-power mode based on preset conditions, thereby reducing the power consumption of the device (10000).

[0171] The system-on-chip (1000) according to the first embodiment of the first example of the present disclosure has the effect of rapidly accelerating inference operations of a generative neural network model at low power through a pipeline circuit structure of a processing core (120), a vector core (153), and a scalar core (154).

[0172] FIG. 10 is a block diagram illustrating a system-on-chip according to a second embodiment of the first example of the present disclosure.

[0173] Referring to FIG. 10, an apparatus (10000) according to a second embodiment of the first example of the present disclosure is described. The apparatus (10000) includes a circuit board (BD), a plurality of main memories (M_MEM_1, …, M_MEM_N) (2000-1, …, 2000-N), and a system-on-chip (1000). In describing the apparatus (10000) according to the second embodiment of the first example of the present disclosure, any content that overlaps with the description with reference to FIG. 3 through FIG. 9 may be omitted for convenience of explanation. Each example and each embodiment of the present disclosure may be modified and implemented by combining with other examples and other embodiments of the present disclosure.

[0174] The system-on-chip (1000) may refer to a system manufactured from a single semiconductor substrate. The system-on-chip (1000) may be configured to further include a package (not shown) that protects the semiconductor substrate.

[0175] A system-on-chip (1000) according to a second embodiment of the first example of the present disclosure is configured to include a neural processing unit (100), a central processing unit (200), a first bus (310), and a second bus (320). A plurality of main memories (2000-1, …, 2000-N) are configured to be electrically connected to the first bus (310). A plurality of main memories (2000-1, …, 2000-N) are configured to include a plurality of semiconductor chips.

[0176] A first bus (310) may be provided between the neural processing unit (100) and a plurality of main memories (2000-1, …, 2000-N). A second bus (320) may be provided between the neural processing unit (100) and the central processing unit (200).

[0177] The operation of the neural processing unit (100) has been described in detail with reference to FIGS. 3 through 9, so a detailed description of the operation of the neural processing unit (100) is omitted. The operation of the central processing unit (200) has been described in detail with reference to FIG. 9, so a detailed description of the operation of the central processing unit (200) is omitted.

[0178] A plurality of main memories (2000-1, …, 2000-N) may be composed of a plurality of memories disposed outside the system-on-chip (1000). However, the present disclosure is not limited thereto. A plurality of main memories (2000-1, …, 2000-N) may each be electrically connected to the system-on-chip (1000) through a first bus (310). Each of the plurality of main memories (2000-1, …, 2000-N) of the device (10000) according to the second embodiment of the first example of the present disclosure may include SRAM, LPDDR RAM, Resistive RAM, Magneto-resistive RAM, Phase-change RAM, Ferroelectric RAM, Flash Memory, or HBM, etc. A plurality of main memories (2000-1, …, 2000-N) may be composed of homogeneous memory or heterogeneous memory.

[0179] It is desirable that the total capacity of the multiple main memories (2000-1, …, 2000-N) be greater than the total capacity occupied by the weight parameters of the artificial intelligence model. In this case, the weight parameters of the neural network model can be loaded and reside in the multiple main memories (2000-1, …, 2000-N) at once. If the storage capacity of the multiple main memories (2000-1, …, 2000-N) is insufficient, it may be difficult to process the inference operations of the neural network model quickly in real time. For example, the total capacity of the multiple main memories (2000-1, …, 2000-N) may be one of 8 GByte, 16 GByte, 32 GByte, 64 GByte, or 128 GByte. That is, the total capacity of the multiple main memories (2000-1, …, 2000-N) may be 8 GByte to 128 GByte, and may be determined by considering the parameter size of at least one artificial intelligence model to be run on the system-on-chip (1000). Accordingly, the device (10000) can reduce the manufacturing cost of the system-on-chip (1000) while easily increasing the capacity of the main memory by placing the multiple main memories for loading and storing the parameters of the artificial intelligence model outside the system-on-chip (1000). If the multiple main memories are placed inside a package protecting the system-on-chip (1000), the size of the package may increase and the manufacturing cost may increase, but the criteria of a specific form factor (e.g., M.2 2230, M.2 2242) may be satisfied.

[0180] The bandwidth of the first bus (310) may be determined according to the processing performance of the neural processing unit (100) (e.g., Tera Operations Per Second (TOPS) of the neural processing unit (100)). For example, a plurality of main memories (2000-1, …, 2000-N) may be configured to provide a bus bandwidth of 60 GB / s to 120 GB / s for a neural processing unit (N100) of 30 TOPS. The bandwidth of the first bus (310) may be determined according to the operating frequency of the plurality of main memories (2000-1, …, 2000-N) and the number of communication channels. However, the examples of the present disclosure are not limited to the bandwidth of the first bus (310).

[0181] The device (10000) according to the second embodiment of the first example of the present disclosure provides a plurality of main memories (2000-1, …, 2000-N), and the total capacity of the main memories may be a capacity capable of storing all parameters of at least one neural network model at once. In addition, according to the configuration of the second embodiment of the first example, the memory capacity can be expanded to correspond to the size of a generative neural network model in which the size of the weight parameters is substantial. Thus, the system-on-chip (1000) can rapidly process the neural network model in real time. To elaborate, the plurality of main memories (2000-1, …, 2000-N) of the device (10000) according to the embodiments of the present disclosure are configured as dedicated memories of the system-on-chip (1000). If the parameters of the artificial intelligence model processed by the neural processing unit (100) are not all stored in multiple main memories (2000-1, …, 2000-N), in particular, if all weight parameters and attention scores cannot be stored in multiple main memories (2000-1, …, 2000-N), the parameters must be stored in a separate storage device, and a decrease in computational speed due to large-capacity memory operations is caused.

[0182] FIG. 11 is a block diagram illustrating a system-on-chip according to a third embodiment of the first example of the present disclosure.

[0183] With reference to FIG. 11, an apparatus (10000) according to a third embodiment of the first example of the present disclosure is described. The apparatus (10000) includes a circuit board (BD), a main memory (2000), and a system-on-chip (1000). In describing the system-on-chip (1000) according to the third embodiment of the first example of the present disclosure, any redundancy in the descriptions above with reference to FIG. 3 through FIG. 10 may be omitted for convenience of explanation. Each example and each embodiment of the present disclosure may be modified and implemented by combining with other examples and other embodiments of the present disclosure.

[0184] The system-on-chip (1000) may refer to a system manufactured from a single semiconductor substrate. The system-on-chip (1000) may be configured to further include a package (not shown) that protects the semiconductor substrate.

[0185] A system-on-chip (1000) according to the third embodiment of the first example of the present disclosure is configured to include a neural processing unit (100), a central processing unit (200), a first bus (310), a second bus (320), a third bus (330), and a shared memory (S_MEM) (400). The shared memory (400) is configured as an on-chip memory of the system-on-chip (1000). The main memory (2000) is configured to be electrically connected to the first bus (310). The main memory (2000) is configured to include at least one semiconductor chip. The shared memory (400) is configured to be electrically connected to the third bus (330).

[0186] A first bus (310) may be located between the neural processing unit (100) and the main memory (2000). A second bus (320) may be located between the neural processing unit (100) and the central processing unit (200). A third bus (330) may be located between the neural processing unit (100) and the shared memory (400). For example, the third bus (330) may be an AXI (Advanced Extensible Interface) bus. However, the examples of the present disclosure are not limited thereto. The third bus (330) may be configured to support a read and write address / control interface between the shared memory (400) and the neural processing unit (100). The third bus (330) may be configured as an independent data communication channel to support high-speed data transmission for transaction processing in addition to memory control commands. The third bus (330) may be configured as a dedicated bus for transmitting input parameters, output parameters, and weight parameters of the neural network model. Here, the first bus (310) and the third bus (330) are configured to be electrically connected. Accordingly, the shared memory (400) and the main memory (2000) can transmit tensor data through the first bus (310) and the third bus (330).

[0187] The operation of the neural processing unit (100) has been described in detail with reference to FIGS. 3 to 10, so a detailed description is omitted. The operation of the central processing unit (200) has been described in detail with reference to FIGS. 9 to 10, so a detailed description is omitted.

[0188] The main memory (2000) may consist of at least one memory placed outside the system-on-chip (1000). The main memory (2000) may consist of homogeneous memory or heterogeneous memory. The capacity of the main memory (2000) may be determined by considering the parameter size of at least one neural network model to be run on the system-on-chip (1000). The bandwidth of the first bus (310) may be determined based on the processing performance of the neural processing unit (100).

[0189] The shared memory (400) may be composed of at least one memory placed inside the system-on-chip (1000). The shared memory (400) of the system-on-chip (1000) according to the third embodiment of the first example of the present disclosure may include at least one memory such as SRAM, LPDDR RAM, Resistive RAM, Magneto-resistive RAM, Phase-change RAM, Ferroelectric RAM, Flash Memory, HBM, etc. The shared memory (400) may be composed of homogeneous memory or heterogeneous memory. As a representative embodiment, the shared memory (400) may be implemented as SRAM. The capacity of the shared memory (400) is designed to be smaller than the capacity of the main memory (2000). If the capacity of the shared memory (400) increases, the semiconductor manufacturing cost may increase rapidly. Therefore, the capacity of the shared memory (400) may be configured to be 4 MByte to 128 MByte. Preferably, the capacity of the shared memory (400) may be configured to be 16 MByte to 64 MByte. The capacity of the shared memory (400) may be larger than the capacity of the internal memory (140) of the neural processing unit (100).

[0190] The operation of the shared memory (400) and main memory (2000) of FIG. 9 is explained with reference to FIG. 7. “32b S_MEM Read” in FIG. 7 refers to a 32-bit data read operation of the main memory (2000). Here, the main memory (2000) may be an LPDDR RAM. In this case, the 32-bit data read operation of the main memory (2000) may consume 640 pj of energy. “32b S_MEM Read” in FIG. 7 refers to a 32-bit data read operation of the shared memory (400). Here, the shared memory (400) may be an SRAM. In this case, the 32-bit data read operation of the shared memory (400) may consume 5 pj of energy. That is, when the neural processing unit (100) uses a shared memory (400) implemented in SRAM, the energy consumption of the device (10000) can be significantly reduced compared to when only the main memory (2000) is used. However, since the manufacturing cost of the shared memory (400) is high, it is difficult to implement it in a large capacity. Therefore, when generating executable code, the compiler (not shown) can set a data placement strategy so that parameters with a high reuse frequency are preferentially allocated to the high-speed shared memory (400). That is, the compiler (not shown) can determine reusable parameters during a series of neural network model operations and provide a function to store them in the shared memory (400) for reuse.

[0191] A system-on-chip (1000) according to the third embodiment of the first example of the present disclosure can reduce power consumption of the system-on-chip (1000) by providing a shared memory (400) and executing an execution code configured to store reusable parameters in the shared memory (400).

[0192] FIG. 12 is a block diagram illustrating a system-on-chip according to a fourth embodiment of the first example of the present disclosure.

[0193] Referring to FIG. 12, an apparatus (10000) according to a fourth embodiment of the first example of the present disclosure is described. The apparatus (10000) includes a circuit board (BD), a main memory (2000), and a system-on-chip (1000). In describing the system-on-chip (1000) according to the fourth embodiment of the first example of the present disclosure, descriptions that are redundant in FIG. 3 through FIG. 11 may be omitted for convenience of explanation. Each example and each embodiment of the present disclosure may be modified and implemented by combining with other examples and other embodiments of the present disclosure.

[0194] The system-on-chip (1000) may refer to a system manufactured from a single semiconductor substrate. The system-on-chip (1000) may be configured to further include a package (not shown) that protects the semiconductor substrate.

[0195] A system-on-chip (1000) according to the fourth embodiment of the first example of the present disclosure is configured to include a neural processing unit (100), a central processing unit (200), a first bus (310), a second bus (320), a third bus (330), and a plurality of shared memories (400-1, …, 400-N). The plurality of shared memories (400-1, …, 400-N) are composed of on-chip memories of the system-on-chip (1000). A main memory (2000) is configured to be electrically connected to the first bus (310). The main memory (2000) is configured to include at least one semiconductor chip. The plurality of shared memories (400-1, …, 400-N) are configured to be electrically connected to the third bus (330).

[0196] A first bus (310) may be located between the neural processing unit (100) and the main memory (2000). A second bus (320) may be located between the neural processing unit (100) and the central processing unit (200). A third bus (330) may be located between the neural processing unit (100) and a plurality of shared memories (400-1, …, 400-N). Here, the first bus (310) and the third bus (330) are configured to be electrically connected.

[0197] Since the operation of the neural processing unit (100) has been described with reference to FIGS. 3 to 11, a redundant description of the operation of the neural processing unit (100) is omitted. Since the operation of the central processing unit (200) has been described with reference to FIGS. 9 to 11, a redundant description of the central processing unit (200) is omitted.

[0198] The main memory (2000) may consist of at least one memory placed outside the system-on-chip (1000). The main memory (2000) may consist of homogeneous memory or heterogeneous memory. The capacity of the main memory (2000) may be determined by considering the parameter size of at least one neural network model to be run on the system-on-chip (1000). The bandwidth of the first bus (310) may be determined according to the processing performance of the neural processing unit (100).

[0199] A plurality of shared memories (400-1, …, 400-N) may be composed of a plurality of memories disposed within the system-on-chip (1000). Each shared memory may be configured to operate independently of one another. A plurality of shared memories (400-1, …, 400-N) of the system-on-chip (1000) according to the fourth embodiment of the first example of the present disclosure may include at least one memory among SRAM, LPDDR RAM, Resistive RAM, Magneto-resistive RAM, Phase-change RAM, Ferroelectric RAM, Flash Memory, HBM, etc. A plurality of shared memories (400-1, …, 400-N) may be composed of homogeneous memory or heterogeneous memory. Here, a plurality of shared memories (400-1, …, 400-N) is described as an example implemented as SRAM. The capacity of the multiple shared memories (400-1, …, 400-N) is designed to be smaller than the capacity of the main memory (2000). If the capacity of the multiple shared memories (400-1, …, 400-N) increases, the semiconductor manufacturing cost may increase rapidly. Therefore, the total capacity of the multiple shared memories (400-1, …, 400-N) can be configured to be 4 MByte to 128 MByte. Preferably, the total capacity of the multiple shared memories (400-1, …, 400-N) can be configured to be 16 MByte to 64 MByte. The capacity of the multiple shared memories (400-1, …, 400-N) can be configured to be larger than the capacity of the internal memory (140) of the neural processing unit (100).

[0200] When the neural processing unit (100) uses multiple shared memories (400-1, …, 400-N) implemented as SRAM, the energy consumption of the device (10000) can be significantly reduced compared to when only the main memory (2000) is used. Accordingly, when the compiler (not shown) generates executable code for a neural network model to be processed by the system-on-chip (1000), it can set a data reuse command to preferentially store reusable input parameters, output parameters, and weight parameters in multiple shared memories (400-1, …, 400-N) located inside the system-on-chip (1000) rather than in the main memory (2000).

[0201] A system-on-chip (1000) according to the fourth embodiment of the first example of the present disclosure provides a plurality of shared memories (400-1, …, 400-N) and can reduce the power consumption of the system-on-chip (1000) by executing an execution code configured to preferentially store reusable parameters in the plurality of shared memories (400-1, …, 400-N). In addition, by providing a plurality of independent shared memories for each domain (e.g., weight domain, attention score domain, etc.), memory operations of each domain (e.g., weight, input parameter, output parameter) of the neural network model parameters can be easily processed.

[0202] A system-on-chip (1000) according to the fourth embodiment of the first example of the present disclosure provides a plurality of shared memories (400-1, …, 400-N) and can reduce the power consumption of the system-on-chip (1000) by executing an execution code configured to preferentially store reusable parameters in the plurality of shared memories (400-1, …, 400-N). In addition, by providing a plurality of independent shared memories, memory operations of each domain of the parameters of the neural network model (e.g., weights, input parameters, output parameters) can be easily processed.

[0203] FIG. 13 is a block diagram illustrating a system-on-chip according to the fifth embodiment of the first example of the present disclosure.

[0204] With reference to FIG. 13, an apparatus (10000) according to the fifth embodiment of the first example of the present disclosure is described. The apparatus (10000) may include a circuit board (BD), a plurality of main memories (2000-1, …, 2000-N), and a system-on-chip (1000). In describing the system-on-chip (1000) according to the fifth embodiment of the first example of the present disclosure, descriptions that are redundant in FIG. 3 through FIG. 12 may be omitted for convenience of explanation. Each example and each embodiment of the present disclosure may be modified and implemented by combining with other examples and other embodiments of the present disclosure.

[0205] The system-on-chip (1000) may refer to a system manufactured from a single semiconductor substrate. The system-on-chip (1000) may be configured to further include a package (not shown) that protects the semiconductor substrate.

[0206] A system-on-chip (1000) according to the fifth embodiment of the first example of the present disclosure is configured to include a neural processing unit (100), a central processing unit (200), a first bus (310), a second bus (320), a third bus (330), and a plurality of shared memories (400-1, …, 400-N). The plurality of shared memories (400-1, …, 400-N) are composed of on-chip memories of the system-on-chip (1000). A plurality of main memories (2000-1, …, 2000-N) are configured to be electrically connected to the first bus (310). A plurality of main memories (2000-1, …, 2000-N) are configured to include at least one semiconductor chip. A plurality of shared memories (400-1, …, 400-N) are configured to be electrically connected to the third bus (330).

[0207] A first bus (310) may be provided between the neural processing unit (100) and a plurality of main memories (2000-1, …, 2000-N). A second bus (320) may be provided between the neural processing unit (100) and the central processing unit (200). A third bus (330) may be provided between the neural processing unit (100) and a plurality of shared memories (400-1, …, 400-N). Here, the first bus (310) and the third bus (330) are configured to be electrically connected.

[0208] Refer to FIGS. 3 to FIGS. 12 for a description of the neural processing unit (100). Accordingly, a redundant description of the neural processing unit (100) is omitted. Refer to FIGS. 9 to FIGS. 12 for a description of the central processing unit (200). Accordingly, a redundant description of the central processing unit (200) is omitted. Refer to FIG. 10 for a description of the plurality of main memories (2000-1, …, 2000-N). Accordingly, a redundant description of the plurality of main memories (2000-1, …, 2000-N) is omitted. Refer to FIG. 12 for a description of the plurality of shared memories (400-1, …, 400-N). Accordingly, a redundant description of the plurality of shared memories (400-1, …, 400-N) is omitted.

[0209] Multiple main memories (2000-1, …, 2000-N) may be composed of multiple memories placed outside the system-on-chip (1000). The total capacity of the multiple main memories (2000-1, …, 2000-N) may be determined by considering the parameter size of at least one neural network model to be run on the system-on-chip (1000). The total capacity of the multiple main memories (2000-1, …, 2000-N) may be configured to be larger than the total capacity of the multiple shared memories (400-1, …, 400-N). Thus, by providing multiple main memories outside the system-on-chip (1000), the manufacturing cost of the system-on-chip (1000) can be reduced.

[0210] Multiple shared memories (400-1, …, 400-N) may be composed of multiple memories placed inside the system-on-chip (1000). The total capacity of the multiple shared memories (400-1, …, 400-N) may be configured to be larger than the capacity of the internal memory (140) of the neural processing unit (100). Accordingly, by placing reusable parameters in the multiple shared memories (400-1, …, 400-N), the power consumption of the device (10000) can be reduced by reducing the amount of data read or transmitted from the multiple main memories (2000-1, …, 2000-N).

[0211] When a compiler (not shown) generates executable code for a neural network model to be processed by a system-on-chip (1000), it can be configured to store reusable input parameters, output parameters, and weight parameters in multiple shared memories (400-1, …, 400-N) preferentially over multiple main memories (2000-1, …, 2000-N).

[0212] The device (10000) according to the fifth embodiment of the first example of the present disclosure can improve data reuse while supporting a generative neural network model with a large number of parameters, by combining the features of the second embodiment of the first example of the present disclosure and the fourth embodiment of the first example, with an expandable main memory and an expandable shared memory.

[0213] FIG. 14 is a block diagram illustrating a system-on-chip according to the sixth embodiment of the first example of the present disclosure.

[0214] Referring to FIG. 14, an apparatus (10000) according to the sixth embodiment of the first example of the present disclosure is described. The apparatus (10000) may include a circuit board (BD), a memory (2000), and a system-on-chip (1000). In describing the system-on-chip (1000) according to the sixth embodiment of the first example of the present disclosure, descriptions that are redundant in FIG. 3 through FIG. 13 may be omitted for convenience of explanation. Each example and each embodiment of the present disclosure may be modified and implemented by combining with other examples and other embodiments of the present disclosure.

[0215] The system-on-chip (1000) may refer to a system manufactured from a single semiconductor substrate. The system-on-chip (1000) may be configured to further include a package (not shown) that protects the semiconductor substrate.

[0216] A system-on-chip (1000) according to the sixth embodiment of the first example of the present disclosure is configured to include a plurality of neural processing units (NPU_1, …, NPU_N) (100-1, …, 100-N), a central processing unit (200), a first bus (310), and a second bus (320). A memory (2000) is configured to be electrically connected to the first bus (310).

[0217] Each of the plurality of neural processing units (100-1, …, 100-N) may be configured to communicate with each other with the memory (2000). A first bus (310) may be provided between the plurality of neural processing units (100-1, …, 100-N) and the memory (2000).

[0218] Each of the plurality of neural processing units (100-1, …, 100-N) may be configured to communicate with each other with the central processing unit (200). A second bus (320) may be provided between the plurality of neural processing units (100-1, …, 100-N) and the central processing unit (200).

[0219] Refer to FIGS. 3 to FIGS. 13 for a description of the multiple neural processing units (100-1, …, 100-N). Accordingly, a redundant description of the neural processing unit (100) is omitted.

[0220] The processing performance (TOPS) of each of the multiple neural processing units (100-1, …, 100-N) may be the same or different from one another. Each of the multiple neural processing units (100-1, …, 100-N) may be configured to operate independently. Each of the multiple neural processing units (100-1, …, 100-N) may be configured to process the computation of a specific neural network model in parallel.

[0221] Refer to FIGS. 9 to 13 for a description of the central processing unit (200). Accordingly, a redundant description of the central processing unit (200) is omitted.

[0222] The central processing unit (200) may be configured to control each of the plurality of neural processing units (100-1, …, 100-N) based on the execution code of the neural network model. Here, a compiler (not shown) may be configured to generate respective execution codes corresponding to each of the plurality of neural processing units (100-1, …, 100-N). The central processing unit (200) may be configured to control the plurality of neural processing units (100-1, …, 100-N) by directly setting the register values ​​of the circuits of each of the plurality of neural processing units (100-1, …, 100-N) based on each execution code.

[0223] The memory (2000) may be configured as a main memory placed outside the system-on-chip (1000). The memory (2000) may be configured as a homogeneous memory or a heterogeneous memory. The capacity of the memory (2000) may be determined by considering the parameter size of at least one neural network model to be run on the system-on-chip (1000). The bandwidth of the first bus (310) may be determined according to the processing performance of a plurality of neural processing units (100-1, …, 100-N).

[0224] The bandwidth of the first bus (310) can be determined according to the processing performance of a plurality of neural processing units (100-1, …, 100-N) (e.g., Tera Operations Per Second (TOPS) of the neural processing unit (100)). For example, a bandwidth of 2 GB / s to 4 GB / s of memory (2000) per TOPS may be required. For example, if there are two neural processing units (100) of 10 TOPS, the memory (2000) may be configured to provide a bus bandwidth of 40 GB / s to 80 GB / s. The bandwidth of the first bus (310) can be determined according to the operating frequency of the memory (2000) and the number of communication channels.

[0225] The system-on-chip (1000) according to the sixth embodiment of the first example of the present disclosure provides a plurality of neural processing units (100-1, …, 100-N), thereby improving the parallel processing performance of the neural network model and having the effect of generating a response to an input query more quickly.

[0226] FIG. 15 is a block diagram illustrating a system-on-chip according to the seventh embodiment of the first example of the present disclosure.

[0227] Referring to FIG. 15, an apparatus (10000) according to the seventh embodiment of the first example of the present disclosure is described. The apparatus (10000) may include a circuit board (BD), a plurality of main memories (2000-1, …, 2000-N), and a system-on-chip (1000). In describing the system-on-chip (1000) according to the seventh embodiment of the first example of the present disclosure, descriptions that are redundant in FIG. 3 through FIG. 14 may be omitted for convenience of explanation. Each example and each embodiment of the present disclosure may be modified and implemented by combining with other examples and other embodiments of the present disclosure.

[0228] The system-on-chip (1000) may refer to a system manufactured from a single semiconductor substrate. The system-on-chip (1000) may be configured to further include a package (not shown) that protects the semiconductor substrate.

[0229] A system-on-chip (1000) according to the seventh embodiment of the first example of the present disclosure is configured to include a plurality of neural processing units (100-1, …, 100-N), a central processing unit (200), a first bus (310), a second bus (320), a third bus (330), and a plurality of shared memories (400-1, …, 400-N). The plurality of shared memories (400-1, …, 400-N) are composed of on-chip memories of the system-on-chip (1000). A plurality of main memories (2000-1, …, 2000-N) are configured to be electrically connected to the first bus (310). A plurality of main memories (2000-1, …, 2000-N) are configured to include at least one semiconductor chip. A plurality of shared memories (400-1, …, 400-N) are configured to be electrically connected to the third bus (330).

[0230] A first bus (310) may be provided between the neural processing unit (100) and a plurality of main memories (2000-1, …, 2000-N). A second bus (320) may be provided between the neural processing unit (100) and the central processing unit (200). A third bus (330) may be provided between the neural processing unit (100) and a plurality of shared memories (400-1, …, 400-N). Here, the first bus (310) and the third bus (330) are configured to be electrically connected.

[0231] Refer to FIGS. 3 to FIGS. 14 for a description of the neural processing unit (100). Accordingly, a redundant description of the neural processing unit (100) is omitted. Refer to FIGS. 9 to FIGS. 14 for a description of the central processing unit (200). Accordingly, a redundant description of the central processing unit (200) is omitted. Refer to FIG. 10 for a description of the plurality of main memories (2000-1, …, 2000-N). Accordingly, a redundant description of the plurality of main memories (2000-1, …, 2000-N) is omitted. Refer to FIG. 12 for a description of the plurality of shared memories (400-1, …, 400-N). Accordingly, a redundant description of the plurality of shared memories (400-1, …, 400-N) is omitted. Refer to FIG. 14 for a description of the plurality of neural processing units (100-1, …, 100-N). Accordingly, redundant descriptions of multiple neural processing units (100-1, …, 100-N) are omitted.

[0232] Multiple main memories (2000-1, …, 2000-N) may be composed of multiple memories placed outside the system-on-chip (1000). The total capacity of the multiple main memories (2000-1, …, 2000-N) may be determined by considering the parameter size of at least one neural network model to be run on the system-on-chip (1000). The total capacity of the multiple main memories (2000-1, …, 2000-N) may be configured to be larger than the total capacity of the multiple shared memories (400-1, …, 400-N). Thus, by providing multiple main memories outside the system-on-chip (1000), the manufacturing cost of the system-on-chip (1000) can be reduced.

[0233] Multiple shared memories (400-1, …, 400-N) may be composed of multiple memories placed inside the system-on-chip (1000). The total capacity of the multiple shared memories (400-1, …, 400-N) may be configured to be larger than the capacity of the internal memory (140) of the neural processing unit (100). Accordingly, by placing reusable parameters in the multiple shared memories (400-1, …, 400-N), the amount of data read or transmitted from the multiple main memories (2000-1, …, 2000-N) can be reduced, thereby reducing the power consumption of the device (10000).

[0234] When a compiler (not shown) generates executable code for a neural network model to be processed by a system-on-chip (1000), it can be configured to store reusable input parameters, output parameters, and weight parameters in multiple shared memories (400-1, …, 400-N) preferentially over multiple main memories (2000-1, …, 2000-N).

[0235] The apparatus (10000) according to the seventh embodiment of the first example of the present disclosure provides a plurality of neural processing units, an expandable main memory, and an expandable shared memory by combining the features of the fifth embodiment of the first example and the sixth embodiment of the first example of the present disclosure. Accordingly, data reuse can be improved while the plurality of neural processing units support the parallel processing of a generative neural network model with a large number of parameters.

[0236] FIG. 16 is a block diagram illustrating a system-on-chip according to the eighth embodiment of the first example of the present disclosure.

[0237] A system-on-chip (1000) according to the eighth embodiment of the first example of the present disclosure is described with reference to FIG. 16. In describing the system-on-chip (1000) according to the eighth embodiment of the first example of the present disclosure, descriptions that are redundant in FIG. 3 to FIG. 15 may be omitted for convenience of explanation. Each example and each embodiment of the present disclosure may be modified and implemented by combining with other examples and other embodiments of the present disclosure.

[0238] The system-on-chip (1000) may refer to a system manufactured from a single semiconductor substrate. The system-on-chip (1000) may be configured to further include a package (not shown) that protects the semiconductor substrate.

[0239] A system-on-chip (1000) according to the eighth embodiment of the first example of the present disclosure is configured to include a neural processing unit (100), a central processing unit (200), a first bus (310), and a second bus (320). Herein, the system-on-chip (1000) according to the eighth embodiment of the first example of the present disclosure is configured to further include a digital signal processing unit (500). A memory (2000) is configured to be electrically connected to the first bus (310). The memory (2000) is configured to include at least one semiconductor chip.

[0240] A first bus (310) may be provided between the neural processing unit (100) and the memory (2000). A first bus (310) may be provided between the digital signal processing unit (500) and the memory (2000). A second bus (320) may be provided between the neural processing unit (100) and the central processing unit (200). A second bus (320) may be provided between the digital signal processing unit (500) and the central processing unit (200).

[0241] Refer to FIGS. 3 to 15 for a description of the neural processing unit (100). Accordingly, a redundant description of the neural processing unit (100) is omitted. Refer to FIGS. 9 to 15 for a description of the central processing unit (200). Accordingly, a redundant description of the central processing unit (200) is omitted. Refer to FIGS. 9 to 15 for a description of the memory (2000). Accordingly, a redundant description of the memory (2000) is omitted.

[0242] The digital signal processing unit (500) may be configured to process at least a portion of the computation of a neural network model that is inefficient to process in the neural processing unit (100). The digital signal processing unit (500) may include at least one vector processor (not shown) and at least one scalar processor (not shown).

[0243] A system-on-chip (1000) according to the eighth embodiment of the first example of the present disclosure further provides a digital signal processing unit (500) to offload specific operations of a neural network model that are inefficient to process in the neural processing unit (100). Thus, the system-on-chip (1000) can rapidly process the neural network model in real time.

[0244] In some examples, a dedicated bus or high-speed bus interface operating separately from the system bus may be further included to improve the transfer speed of parameters or computational data between the shared memory(s) and the neural processing unit (100). Here, the system bus is a common data transmission path within the system-on-chip (1000) including the second bus (320), the third bus (330), etc., and may be configured to support communication between modules such as the neural processing unit, the central processing unit, and the on-chip memory. Meanwhile, the dedicated bus may perform the role of connecting model parameters of a specific domain stored in the shared memory in parallel with a plurality of neural processing units to alleviate data transmission bottlenecks and enable high-speed access. This separate bus configuration is designed to reduce the load on the main bus (e.g., the first bus (310)) while simultaneously optimizing the computational path accessing the shared memory.

[0245] Various system-on-chip (SoC) embodiments according to the first example of the present disclosure (see FIGS. 9 through 16) provide a hardware foundation for implementing a scalable on-device AI acceleration platform with high parallel processing capabilities and flexibility, and such a platform may include the following key features optionally or in combination:

[0246] First, according to various examples, a multi-NPU core architecture may be provided that can enhance parallel processing capabilities by integrating multiple NPU cores (each NPU including an integer processing core (P_CORE) and a special function unit (SFU) for handling floating-point (FP) operations) on a single chip, each having optimized power, performance, and area (PPA). (e.g., see the 6th embodiment (Fig. 14) and the 7th embodiment (Fig. 15) of the first example.)

[0247] Second, a hierarchical / distributed memory system may be provided, comprising a plurality of independent on-chip shared memory (SRAM) banks allocated to each NPU core or group of NPU cores to enhance data locality, and a multi-channel main memory (external DRAM) system accessible to all of these NPU cores and providing high total bandwidth. (e.g., see the third, fourth, fifth, and seventh embodiments of the first example (Figs. 11, 12, 13, 15)). This can efficiently manage complex data flows and minimize memory bottlenecks.

[0248] Third, the special function unit (SFU) within the NPU core may include a programmable piecewise affine function (PAF) unit. (e.g., see the configuration of the SFU (150) within the NPU (100) of FIG. 3 and the fourth example (Figs. 24, 25, 27)). This allows for software-based support of various current and future activation functions and non-linear operations without separate hardware changes, thereby ensuring the flexibility and future scalability of the system.

[0249] Fourth, in a system configuration according to the first example of the present disclosure (e.g., FIGS. 9 to 16), a digital signal processing device (DSP, 500) may be optionally integrated to offload data processing before and after neural network computation (e.g., sensor data preprocessing, postprocessing) or specific computations that are inefficient to process in the NPU (Fig. 16 illustrates one example of such DSP integration). This improves the overall efficiency of the system by allowing the NPU to focus on core AI computations.

[0250] The organic combination of the aforementioned components can generate powerful synergy. Multiple NPU cores and distributed / hierarchical memory structures provide extreme parallelism and high data feeding capabilities, enabling the simultaneous or very rapid processing of complex or multiple AI models in an on-device environment. The combination of programmable SFUs further provides flexibility not limited to specific models or computations, as well as adaptability to new future AI algorithms, within the on-device environment. The optional integration of DSPs further expands the scope of application for these platforms.

[0251] In summary, the scalable on-device artificial intelligence (AI) acceleration system according to the present disclosure may include: one or more neural processing unit (NPU) cores, each comprising an integer arithmetic processing core and a floating-point special function unit (SFU); one or more on-chip shared memories (optional) connected to each of the NPU cores or a group of NPU cores to provide data locality; a high-bandwidth main memory interface shared by the NPU cores for storing large amounts of data; and an integrated controller that distributes AI operations to the NPU cores and (optional) the DSP, and controls data flow through the on-chip shared memory and the main memory interface. In this case, the floating-point special function unit (SFU) may include a programmable function approximation (PAF) circuit that approximates various non-linear functions according to input parameters. Additionally, a digital signal processing unit (DSP) that performs specific signal processing operations independently of the operations of the NPU cores may be optionally provided. This enables the parallel and flexible acceleration of various AI models and workloads in an on-device environment, thereby making the realization of high-performance AI applications possible.

[0252] Various system-on-chip (SoC) embodiments according to the first example of the present disclosure (see FIGS. 9 through 16) can create integrated synergy effects beyond simple parallel arrangement of hardware by combining with the unique architecture of the neural processing unit (100) described above. This is a feature of the present disclosure that solves technical challenges of implementing high-performance on-device AI and can be embodied as follows.

[0253] For example, intelligent linkage between a hierarchical / distributed memory system and a hybrid function approximation unit can be implemented. As illustrated in FIGS. 11, 13, 15, etc., a system according to the first example of the present disclosure may adopt a hierarchical memory structure composed of a large-capacity external main memory (2000) and a high-speed on-chip shared memory (400). The effect of such a memory system may be enhanced when it operates in conjunction with a hybrid structure of a special function unit (150) that combines a programmable method and a dedicated hardware method to process various non-linear functions. Specifically, the control unit (110) or the compiler can optimize data placement so that the massive neural network weights are stored in the external main memory (2000), while parameters such as programmable operation coefficients, which the function approximator (FuncApx) repeatedly uses to approximate activation functions such as GELU and SiLU, are resident in the high-speed on-chip shared memory (400) or the internal memory of the neural processing unit (NPU). Since the coefficients are loaded directly from the on-chip memory without needing to access the external memory for every activation function operation, memory bottlenecks in the system can be significantly alleviated and power consumption associated with data movement can be reduced.

[0254] For example, a computation pipeline can be implemented through NPU-DSP heterogeneous computing. As illustrated in FIG. 16, the system of the present disclosure can implement a highly specialized heterogeneous computing architecture by integrating a digital signal processing unit (DSP, 500) together with a neural processing unit (100). This can improve processing efficiency by breaking down the entire AI inference process into the following three-stage specialized pipeline: Signal pre / post-processing (dedicated to DSP): Traditional signal processing operations, such as sensor data filtering and Fourier transform (FFT), can be configured to be dedicated to the DSP (500). Large-scale integer matrix operations (dedicated to P_CORE): Matrix multiplication (MatMul), which accounts for most of the neural network computation, can be configured to be processed by an integer operation-based processing core (P_CORE, 120) specialized for low power. High-precision non-linear function operations (dedicated to SFU / FuncApx): Complex non-linear functions requiring floating-point (FP) precision, such as softmax, negative exponents, and reciprocals, can be configured to be processed at ultra-high speed by a function approximation unit (FuncApx) within a special function unit (SFU, 150) that performs autonomous pipeline operations with a single instruction. This clear division of roles can optimize the overall processing performance and energy efficiency of the system by allocating hardware optimized for each operation.

[0255] FIG. 17 illustrates a device (10000) according to the first embodiment of the second example of the present disclosure.

[0256] The device (10000) illustrated in FIG. 17 may be a top view of the device (10000), but is not limited thereto. The dimensions of each of the hardware components described with reference to FIG. 17 may refer to width (X-axis length) and depth (Y-axis length).

[0257] An apparatus (10000) according to the first embodiment of the second example of the present disclosure includes a circuit board (9000) and a package (5000). The circuit board (9000) may further include a slot (9100) (e.g., an M.2 slot). An apparatus (10000) according to the first embodiment of the second example of the present disclosure may correspond to an apparatus (10000) according to the embodiments of the first example of the present disclosure. An apparatus (9000) according to the first embodiment of the second example of the present disclosure may correspond to an apparatus (BD) according to the embodiments of the first example of the present disclosure. A package (5000) according to the first embodiment of the second example of the present disclosure may correspond to a package (not shown) in the embodiments of the first example of the present disclosure. Redundant content described with reference to FIGS. 3 through 16 is omitted.

[0258] The device (10000) according to the first embodiment of the second example of the present disclosure may be composed of a system-on-chip (SoC). A system-on-chip (SoC) may refer to a semiconductor package that integrates at least one neural processing unit (100) and various components of an electronic system. A system-on-chip (SoC) may integrate digital circuits, analog circuits, mixed-signal and radio frequency processing circuits into a single package. For example, in addition to the neural processing unit (NPU), the system-on-chip (SoC) may further include at least one of a central processing unit (CPU), a digital signal processor (DSP), an image signal processor (ISP), and a graphics processing unit. A system-on-chip (SoC) may include at least one memory (MEM). A system-on-chip (SoC) may include a high-speed data bus for efficient communication between various circuits included in the system-on-chip (SoC). A system-on-chip (SoC) may include at least one interface, such as PCIe, USB, I2C, SPI, UART, and GPIO, for connection with external devices and sensors. The system-on-chip (SoC) may include an on-chip power management device that regulates voltage and power distribution of the semiconductor package. The system-on-chip (SoC) may include a communication interface that integrates wired or wireless communication protocols, such as Ethernet, Wi-Fi, Bluetooth, and cellular connectivity, for data transmission.

[0259] Referring to FIG. 17, the device (10000) is an example implemented in an M.2 form factor. In the first embodiment of the second example of the present disclosure, the width of the circuit board (9000) may be 22 mm and the depth may be 42 mm, 60 mm, 80 mm, or 110 mm, but is not limited thereto.

[0260] The M.2 board is a next-generation interface standard for high-speed data transmission that offers high speed and efficiency. Designed for a compact size, it is being adopted in various devices such as laptops, desktops, and mini PCs. In particular, small form factors like M.2 make it easy to apply on-device.

[0261] A package (5000) mounted on a device (10000) may include a system-on-chip (SoC) and at least one memory (MEM). The system-on-chip (SoC) according to the first embodiment of the second example of the present disclosure may correspond to the system-on-chip (1000) according to the embodiments of the first example of the present disclosure. The memory (MEM) of FIG. 17 may correspond to the memory (MEM) or main memory (M_MEM) described above with reference to FIG. 9 through 16.

[0262] The physical size of the system-on-chip (SoC) may be, for example, 4.50 mm × 4.85 mm. However, the present disclosure is not limited thereto. Considering the minimum separation distance required between the system-on-chip (SoC) and the memories (MEM), the physical size of each memory (MEM) may be limited to 6.43 mm × 5.30 mm. However, the present disclosure is not limited to the size or dimensions of each component.

[0263] Meanwhile, the maximum number of memory (MEM) within the package (5000) can be determined by considering the processing capability of the system-on-chip (SoC) (e.g., TOPS) or the size of a package of a specific size or smaller that satisfies the size of a specific form factor applied to the system-on-chip (SoC).

[0264] According to one embodiment, the process node of the semiconductor manufacturing process of the memory (MEM) may be lower than the process node of the semiconductor manufacturing process of the system-on-chip (SoC). For example, the memory (MEM) may be manufactured using a 14nm semiconductor manufacturing process, and the system-on-chip (SoC) may be manufactured using a 7nm semiconductor manufacturing process. For example, the memory (MEM) may be manufactured using a 5nm semiconductor manufacturing process, and the system-on-chip (SoC) may be manufactured using a 7nm semiconductor manufacturing process. For example, the memory (MEM) may be manufactured using a 2nm semiconductor manufacturing process, and the system-on-chip (SoC) may be manufactured using a 5nm semiconductor manufacturing process. According to the above configuration, the size of the system-on-chip (SoC) can be reduced to fit the specifications of a specific form factor (e.g., M.2). In particular, for on-device applications, a large size of the system-on-chip (SoC) can be a barrier to commercialization.

[0265] Referring to FIG. 17, the package (5000) may include up to four memory (MEMs). Each memory (MEM) may be LPDDR5 (Low Power Double Data Rate 5). By increasing the number of bus channels between the system-on-chip (SoC) and the memory (MEMs) to correspond to the number of memory (MEMs), the memory bandwidth of the first bus (310) of the first exemplary embodiment of the present disclosure may be increased. For example, if the system-on-chip (SoC) of the package (5000) and the four memory (MEMs) are placed closest to each other, the size of the package (5000) may be 21 mm × 31 mm. In this embodiment, the package (5000) may be mounted on a device (10000) having a width of 22 mm and a depth of 42 mm.

[0266] However, the package (5000) of FIG. 17 cannot be mounted on a circuit board (9000) having a depth of 30 mm. Therefore, FIG. 18 describes an embodiment in which a package (5000) including a system-on-chip (SoC) and four memories (MEMs) can be mounted on a circuit board (9000) having a depth of 30 mm.

[0267] The above package (5000) may include at least one semiconductor chip to which a fan-in semiconductor package, a fan-out semiconductor package, a package-on-package (PoP), and / or a 2.5D package method is applied. Additionally, the technical features of different examples of the present disclosure may be combined and modified.

[0268] FIG. 18 illustrates a device (10000) according to a second embodiment of a second example of the present disclosure.

[0269] The device (10000) illustrated in FIG. 18 may be a top view of the device (10000), but is not limited thereto. The dimensions of each of the hardware components described with reference to FIG. 18 may refer to width (X-axis length) and depth (Y-axis length).

[0270] An apparatus (10000) according to the second embodiment of the second example of the present disclosure may include a circuit board (9000) and a package (5100). The circuit board (9000) may further include a slot (9100) (e.g., an M.2 slot). An apparatus (10000) according to the second embodiment of the second example of the present disclosure may correspond to an apparatus (10000) according to the embodiments of the first example of the present disclosure. An apparatus (9000) according to the second embodiment of the second example of the present disclosure may correspond to an apparatus (BD) according to the embodiments of the first example of the present disclosure. An apparatus (5100) according to the second embodiment of the second example of the present disclosure may correspond to an apparatus (not shown) according to the embodiments of the first example of the present disclosure. Duplicate descriptions of the above contents with reference to FIGS. 3 to 17 are omitted. In addition, in describing the second embodiment of the second example of the present disclosure, descriptions that overlap with the first embodiment of the second example may be omitted for the convenience of explanation.

[0271] Referring to FIG. 18, the device (10000) is an example implemented in an M.2 form factor. In the second embodiment of the second example of the present disclosure, the width of the circuit board (9000) may be 22 mm and the depth may be 30 mm, but is not limited thereto.

[0272] A package (5100) mounted on a device (10000) may include a system-on-chip (SoC) and at least one memory (MEM). A system-on-chip (SoC) according to a second embodiment of the second example of the present disclosure may correspond to a system-on-chip (1000) according to embodiments of the first example of the present disclosure. The memory (MEM) of FIG. 18 may correspond to the memory (MEM) or main memory (M_MEM) described above with reference to FIG. 10 through FIG. 17.

[0273] The width of the circuit board (9000) illustrated in FIG. 18 may be 22 mm and the depth may be 30 mm. In order to mount the package (5100) within such a constrained size, the arrangement of the system-on-chip (SoC) and multiple memory (MEM) within the package (5100) must be improved.

[0274] The physical size of the above system-on-chip (SoC) may be 4.50 mm × 4.85 mm, and the physical size of each memory (MEM) may be 6.43 mm × 5.30 mm.

[0275] According to one embodiment, the process node of the semiconductor manufacturing process of the memory (MEM) may be lower than the process node of the semiconductor manufacturing process of the system-on-chip (SoC). For example, the memory (MEM) may be manufactured using a 14nm semiconductor manufacturing process, and the system-on-chip (SoC) may be manufactured using a 7nm semiconductor manufacturing process. For example, the memory (MEM) may be manufactured using a 5nm semiconductor manufacturing process, and the system-on-chip (SoC) may be manufactured using a 7nm semiconductor manufacturing process. For example, the memory (MEM) may be manufactured using a 2nm semiconductor manufacturing process, and the system-on-chip (SoC) may be manufactured using a 5nm semiconductor manufacturing process. According to the above configuration, the size of the system-on-chip (SoC) can be reduced to fit the specifications of a specific form factor (e.g., M.2).

[0276] Referring to FIG. 18, the system-on-chip (SoC) and the plurality of memory (MEM) can be arranged diagonally within the package (5100). In this case, the size of the package (5100) can be reduced to 21 mm × 21 mm. To elaborate, the configuration of the plurality of semiconductor chips arranged within the package (5100) can be referred to as a “rotated multi-chip substrate layout.” The package (5100) of the rotated multi-chip substrate layout has the effect of reducing the size of the package (5100) due to the arrangement of each semiconductor chip rotated at a specific angle. That is, the X-axis of the system-on-chip (SoC) and the X-axis of the substrate (5100) may not be parallel. The Y-axis of the system-on-chip (SoC) and the Y-axis of the substrate (5100) may not be parallel. The X-axis of the memory (MEM) and the X-axis of the substrate (5100) may not be parallel. The Y-axis of the memory (MEM) and the Y-axis of the substrate (5100) may not be parallel.

[0277] Meanwhile, the number of memory (MEM) in the package (5100) may be changed by taking into account the processing capability of the system-on-chip (SoC) (e.g., TOPS), the size of a specific form factor, and the size that can be placed within a package smaller than the specific form factor.

[0278] The above package (5100) may be configured to include at least one semiconductor chip to which a fan-in semiconductor package, a fan-out semiconductor package, a package-on-package (PoP), and / or a 2.5D package method is applied. Additionally, the technical features of different examples of the present disclosure may be combined and modified.

[0279] That is, the package (5100) may include a semiconductor package substrate, a system-on-chip (SoC) disposed on the semiconductor package substrate and rotated at a first angle relative to one side of the semiconductor package substrate with respect to a plane, and one or more memories (MEMs) disposed on the semiconductor package substrate adjacent to the system-on-chip (SoC) and rotated at a second angle relative to one side of the semiconductor package substrate with respect to a plane. Here, the first angle and the second angle may be determined such that the size of the package (5100) is smaller than the size of a specific form factor.

[0280] That is, according to the device (10000) according to the second embodiment of the second example of the present disclosure, a 'rotated multi-chip substrate layout' is illustrated for mounting a system-on-chip (SoC) and a plurality of memory (MEM) in a very limited form factor such as M.2 2230. In order to efficiently implement sophisticated and computationally intensive artificial intelligence (AI) operations in such small devices where physical space is extremely constrained, an optimized hardware foundation is essential. Various embodiments of the present disclosure can provide such hardware support, and their main features are as follows.

[0281] First, a Neural Processing Unit (NPU) core with highly optimized Power, Performance, and Area (PPA) can be provided. This NPU core includes an integer operation-oriented processing core (P_CORE) and a Special Function Unit (SFU) that handles efficient floating-point (FP) operations; in particular, the SFU may adopt methods such as Piecewise Affine Function (PAF) to support complex activation function processing.

[0282] Second, innovative packaging technology can be applied to integrate the NPU chip and essential memory (e.g., LPDDR series) at high density to facilitate mounting on standard small form factors such as M.2 cards. As exemplified in FIG. 18, the overall module size can be minimized through space-optimized layouts, such as rotating the system-on-chip (SoC) and memory chips within the package.

[0283] Third, a main memory of sufficient capacity may be provided, positioned either inside the package or very close to the main board, to store weight and computation data of the target AI model and to exchange data with the NPU at high speed. This reduces the need for memory expansion via an external large printed circuit board (PCB), contributing to the miniaturization of the entire system.

[0284] As such, the organic combination of the PPA-optimized NPU core, high-density integrated packaging technology (see Fig. 18), and proximity memory configuration creates significant synergy. This enables the effective implementation of sophisticated on-device AI functions (e.g., advanced computations such as speculative decoding) even in mobile devices, wearable devices, or small embedded systems with extremely limited space and power budgets, where it was previously difficult to install AI accelerators.

[0285] In conclusion, the small form factor AI acceleration device according to the present disclosure may include an NPU die comprising an integer processing core and a floating-point special function unit (SFU) with optimized power, performance, and area, and one or more memory dies electrically connected to the NPU die to store weights and computational data of an AI model. The NPU die and the memory dies are densely integrated within a single semiconductor package mountable on a standard small form factor substrate, such as the M.2 specification, as illustrated in FIG. 18, and the relative positions and orientations of the dies within the package may be configured to minimize the total package area. Additionally, the SFU may include a programmable function approximation circuit to implement various activation functions in an area-efficient manner. Through this, the AI ​​acceleration device of the present disclosure can be successfully integrated into portable or embedded devices with severe constraints on physical size and power consumption, thereby supporting the efficient execution of advanced AI computations (e.g., computations such as speculative decoding) in an on-device environment.

[0286] FIG. 19 illustrates a tree for explaining speculative decoding according to the third example of the present disclosure.

[0287] The speculative decoding method according to the third example of the present disclosure can be performed in the apparatus (10000) described with reference to FIGS. 3 to 18. Accordingly, in describing the third example of the present disclosure, descriptions that overlap with the first and second examples may be omitted for convenience of explanation.

[0288] A transformer-based generative AI model can receive a query as input and generate a response. For example, a language model can receive a query in text form, obtain an input token, and generate an output token from the input token. Subsequently, the query and the generated output token can be input back into the language model to generate a subsequent output token. This token generation operation can be repeated multiple times until a response to the query is completed. A token may correspond to a word or a part of a word. The response may include output tokens. The final response generated by the language model for the input query data may include a series of output tokens generated by inputting the output tokens, which were generated as part of the response generated by the language model, back into the language model.

[0289] The larger the number of parameters in a language model, the higher the cost of generating a response. Therefore, using a language model with a large number of parameters increases the computational cost required to generate a response. Reducing the number of parameters can lower computational costs and improve response speed, but it may compromise response accuracy. In other words, there is a trade-off relationship between the number of parameters and accuracy (or quality) of a language model and the time required for the processor to compute the language model. In particular, to generate a response of a certain quality or higher, the number of parameters in the language model must be significantly large.

[0290] On the other hand, providing usability services by processing language models on the device itself presents difficulties when considering the device's hardware resource limitations (e.g., energy consumption, battery capacity, memory capacity, AI processing capabilities, etc.).

[0291] Accordingly, the third example of the present disclosure describes speculative decoding (or speculation decoding) for running a heterogeneous language model by utilizing the limited hardware resources of the device (10000).

[0292] Speculative decoding is a technique in which multiple language models each perform a prediction and then combine the results to generate the most reliable answer. For example, the first language model selects candidates for the next token and calculates a probability representing the goodness of fit for each candidate token. Subsequently, the second language model can make a final selection of one of the candidate tokens suggested by the first language model by considering the context. Since the second language model must select one of the candidate tokens by reflecting a broader context, it may be a model with more parameters and superior performance than the first language model.

[0293] An apparatus (10000) according to the examples of the present disclosure may operate heterogeneous language models based on at least one system-on-chip (1000). That is, in an on-device manner, speculative decoding may utilize heterogeneous language models to accelerate the token generation speed of the language models. At least one apparatus (10000) may utilize a speculative decoding technique in which heterogeneous language models cooperate with each other to generate a response to an input query.

[0294] A heterogeneous language model according to the third example of the present disclosure may include a first language model and a second language model. Here, the number of parameters of the first language model is less than the number of parameters of the second language model. That is, the number of parameters of the heterogeneous language models is different from each other. A language model with a relatively small number of parameters and a language model with a large number of parameters can be defined as a supermodel and a submodel, respectively. That is, the first language model may be defined as a submodel of the second language model, and the second language model may be defined as a supermodel of the first language model. For example, the number of parameters of the supermodel may be 7 billion, and the number of parameters of the submodel may be 1 billion.

[0295] A sub-model according to the third example of the present disclosure may generate probabilities used for sampling additional tokens based on additional tokens generated speculatively and a set of currently accepted tokens. A super-model according to the third example of the present disclosure generates output tokens using candidate tokens generated by the sub-model. The super-model may accept or reject candidate tokens by sampling the additional tokens generated speculatively by the sub-model. Here, acceptance or rejection by the super-model may be determined by comparing the probability generated by the super-model corresponding to each token with the probability generated by the sub-model corresponding to each token. The candidate token accepted by the super-model is the output token included in the response. That is, speculative decoding can drastically reduce the total amount of computation by rapidly generating multiple candidates with a lightweight sub-model and having the heavy super-model participate only in verification, instead of performing the entire computation of the super-model every time.

[0296] According to the speculative decoding according to the third example of the present disclosure, a submodel may generate one or more sets of candidate tokens to generate a response corresponding to a query input to the submodel. A set of candidate tokens is a set containing one or more candidate tokens.

[0297] Candidate Token Candidate Token Set apple, is, red, banana, yellow, car, fast set1: [apple, is, red]set2: [banana, is, yellow]set3: [car, is, fast]

[0298] The upper model may sample for one or more candidate token sets and accept or reject the corresponding candidate token sets. Here, if the device (10000) samples for each candidate token set, the device (10000) can generate more tokens faster than if it samples for each token. Additionally, the probability distributions of the lower model and the upper model can be maintained similarly. Here, the token generation speed may be referred to as TPS (tokens per second). The speculative decoding according to the third example of the present disclosure may operate as an automatic regression token generation method. The automatic regression token generation method generates tokens by inputting a series of tokens generated by the lower model back into the lower model. The lower model generates one token per automatic regression. Therefore, if the lower model performs N inferences, N series of tokens are generated. The lower model can generate a conditional probability distribution related to the N series of tokens. Subsequently, the upper model can generate the probability distribution of the upper model by processing the N series of tokens generated by the lower model and the conditional probability distribution related to the N series of tokens. Therefore, the parent model can accept or reject tokens generated by the child model by comparing the probability distribution generated by the parent model with the conditional probability distribution generated by the child model. The parent model may accept tokens based on a threshold. A token is accepted when the similarity between the probability distribution value of the parent model and the conditional probability distribution value of the child model is equal to or greater than the threshold, and the token may be rejected when the similarity is less than the threshold.

[0299] The upper model can accept or reject a series of tokens generated by the lower model based on a threshold. If a specific token among the series is rejected, the tokens preceding the rejected token become the final output tokens. For example, the lower model of the guess decoding can generate tokens in sets. In this case, the upper model can accept or reject multiple sets of tokens generated by the lower model. If the lower model generates candidate tokens in sets, the acceptance or rejection judgment in the upper model can also be performed in sets, which can improve the token generation speed of the guess decoding of the device (10000). Additionally, as the number of token sets generated by the lower model increases, the probability of the token set that the upper model will accept can increase.

[0300] The sub-model may generate one or more candidate token sets corresponding to an input query. Each candidate token set contains a series of tokens, and the sub-model may select a series of tokens with a high probability in a probability distribution as the candidate token set. Each candidate token set may be selected in various ways. For example, each candidate token set may be generated by selecting tokens with the highest probability within a probability distribution, but the present disclosure is not limited thereto. For the super-model to accept a single candidate token set, the probabilities of the tokens included in each candidate token set may be summed. Therefore, the super-model may accept or reject the entire candidate token set based on the summed probability of the candidate token set, rather than the probability of each individual token. To elaborate, if the probability of each individual token is utilized, the probability of rejection by the super-model may increase because it may be difficult to match the probabilities of each token generated by the sub-model and the super-model. If the summed probability of the token set is utilized, the probability of acceptance by the super-model may increase because the matching of the summed probabilities of the token set generated by the sub-model and the super-model may be more similar.

[0301] The sub-model can generate one or more candidate token sets in response to an input query. The one or more candidate token sets can be generated statically or dynamically. For example, the sequence length of each candidate token set can be pre-set. The sub-model can generate one or more candidate token sets and combine each set to generate tree data. The tree data includes a node at each branch point.

[0302] FIG. 19 illustrates tree data (TD) comprising one or more token sets generated in a sub-model according to one embodiment. Referring to FIG. 19, a first token set (S1), a second token set (S2), a third token set (S3), a fourth token set (S4), and a fifth token set (S5) are shown below the leaf nodes of the tree.

[0303] A first candidate token set (S1) may include four tokens (a, b, c, d). A second candidate token set (S2) may include four tokens (a, b, c, e). A third candidate token set (S3) may include four tokens (a, f, g, h). A fourth candidate token set (S4) may include four tokens (a, f, g, i). A fifth candidate token set (S5) may include four tokens (a, f, j, k). Each of the candidate token sets may be selected or rejected by the upper model.

[0304] The token length of each candidate token set can be set. For example, the sentence length of each token set can be set to 4. However, the present disclosure is not limited thereto, and the length of the token set may be changed dynamically.

[0305] Each token may have an order level. An order level may correspond to a sentence length. Specifically, the first order level (L1) corresponds to token (a). The second order level (L2) corresponds to tokens (b, f). The third order level (L3) corresponds to tokens (c, g, j). The fourth order level (L4) corresponds to tokens (d, e, h, i, k). In an example where the sentence length is set to 4, the maximum order level is set to 4. The order of the tokens in each set of tokens may be arranged according to the corresponding order level. Specifically, among the tokens (a, b, c, d) of the first set of tokens (S1), the first token (a) corresponds to the first order level (L1). Among the tokens (a, b, c, d) of the first set of tokens (S1), the second token (b) corresponds to the second order level (L2). The third token (c) among the tokens (a, b, c, d) of the first set of tokens (S1) corresponds to the third order level (L3). The fourth token (d) among the tokens (a, b, c, d) of the first set of tokens (S1) corresponds to the fourth order level (L4).

[0306] Each token is defined as a node of tree data (TD). For example, a token (g) can be a branch node of the third set of tokens (S3) and the fourth set of tokens (S4) at the third order level (L3). Each node contains branch information.

[0307] That is, the tree data (TD) generated by the sub-model may include one or more sets of tokens (S1 to S5), order levels (L1 to L4), information on the nodes (a, b, c, d, e, f, g, h, i, j, k) of each token, and all or at least part of the selection probability of each candidate set of tokens (e.g., the summed probability of the sets of tokens). The super-model may accept or reject at least one token of the tree data (TD).

[0308] Speculative decoding can also be implemented as hybrid speculative decoding. In hybrid speculative decoding, tokens previously generated speculatively are verified by a higher-level model, and a threshold α can be set to determine whether a token is approved or rejected by comparing the probability distributions of the lower-level model and the higher-level model.

[0309] Group guess decoding is configured so that a lower-level model generates tokens in groups and a higher-level model validates them, which can provide high efficiency.

[0310] FIG. 20 schematically illustrates a speculative decoding operation according to a third example of the present disclosure.

[0311] Referring to FIG. 20, a lower model (LM) and a higher model (UM) are illustrated. An apparatus (10000) according to the third example of the present disclosure may be configured to sequentially process the lower model (LM) and the higher model (UM) for speculative decoding.

[0312] Transformer-based AI models can perform specific computational steps (e.g., prefill and generation steps) to generate a response to an input query. A transformer is an AI model based on an attention mechanism. Transformers utilize numerous matrix multiplication operations. Using input values ​​and parameters such as the query (Q), key (K), and value (V), a transformer can obtain an output value, the attention score (Q, K, V). Based on the output value (i.e., attention (Q, K, V)), the transformer can process various inference operations. Transformers are actively utilized in generative language models.

[0313] A tensor refers to a multidimensional matrix parameter processed in an artificial intelligence model (e.g., an artificial neural network). Tensors can refer to various parameters of a neural network model, such as weights, input parameters, output parameters, and attention parameters.

[0314] The prefill stage is an initial step for processing and interpreting input (e.g., a prompt entered by a user). The prefill stage may include a tokenization stage, a context embedding stage, a processing stage through layers, and a preparation stage for decoding. The prefill stage may be processed by one or more neural processing units. In the tokenization stage, the input text is converted into tokens (sub-words, words, or characters), which are numerical representations used by the AI ​​model. In the context embedding stage, tokens can be mapped to high-density embedding vectors that encode semantic and syntactic information. In the prefill stage, the embeddings are processed through multiple layers of the neural network model (e.g., Transformer blocks) to generate contextual representations of the input tokens. In the preparation stage for decoding, the AI ​​model calculates the probability of the next output token for each token based on the input context, but does not yet finally generate the next token. Instead, it can set the context for the generation stage.

[0315] In the generation phase, one or more neural processing units may sequentially generate one or more output tokens based on the context provided by the artificial intelligence model in the prefill phase and previously generated output tokens. The generation phase may include an initial prediction phase, an autoregressive processing phase, and stopping conditions. The generation phase may be processed by one or more neural processing units. In the initial prediction phase, the first token is generated by sampling from the probability distribution calculated in the prefill phase. In the autoregressive processing phase, each subsequent token is generated by integrating the previous token (including the generated token) into the context of the neural network model. Stopping conditions are met, such as reaching a maximum token limit, the occurrence of a special sequence end token (e.g., end of sequence (EOS)), or the achievement of a specific semantic goal, until the generation process continues. The series of tokens generated in the generation phase may correspond to the tree data (TD) of FIG. 19. The tree data may include at least one set of tokens.

[0316] Referring again to FIG. 20, a third example illustrating the steps of speculative decoding in chronological order is illustrated. Speculative decoding according to the examples of the present disclosure may be processed sequentially or in parallel by one or more neural processing units. Speculative decoding may generate a response to an input query by having a higher model (UM) and a lower model (LM) cooperate with each other. A longer horizontal length of the block representing each step indicates that a longer processing time is required.

[0317] Specifically, in speculative decoding, the upper model (UM) first performs a prefill operation. Subsequently, the lower model (LM) performs a prefill operation. Then, the lower model (LM) sequentially performs generation steps, and each token is generated in each generation step. Next, the upper model (UM) can accept or reject the tokens generated by the lower model (LM). The upper model (UM) can accept or reject some or all of the tokens generated by the lower model (LM). The lower model (LM) receives the accepted tokens as input queries, and the lower model (LM) sequentially performs generation steps to generate each token corresponding to each generation step. Next, the upper model (UM) can accept or reject the tokens generated by the lower model (LM). The above steps are repeated until stopping conditions are satisfied.

[0318] The number of parameters in the upper model (UM) is greater than that of the lower model (LM), resulting in a longer processing time. Consequently, the processing time of the upper model (UM)'s prefill phase is longer than that of the lower model (LM). The processing time of the upper model (UM)'s acceptance or rejection phase is longer than that of the lower model (LM)'s generation phase. Therefore, regarding token generation for creating a response, the system's token generation speed can be improved by configuring the system so that the lower model generates as many tokens as possible and the upper model performs only selective decisions.

[0319] The number of parameters for each of the lower model (LM) and the upper model (UM) can be determined by taking into account the hardware resource limitations of the device (10000) (e.g., the processing performance of the system-on-chip (SoC) and the bandwidth of the memory (MEM). The upper model (UM) can utilize a language model in which the number of parameters is determined by taking into account the hardware resource limitations of the device being processed.

[0320] In some examples, the lower model (LM) may be a model obtained by pruning the parameters of the upper model (UM). In such cases, the parameters of the pruned neural network model can be lightweight, and the required memory bandwidth may be smaller than that of the upper model (UM).

[0321] According to a third example of the present disclosure, the device (10000) may be configured to process a prefill operation using an upper model (UM) to process speculative decoding. Subsequently, the device (10000) may process a prefill operation and a generation operation using a lower model (LM) to process speculative decoding.

[0322] A device (10000) according to the third example of the present disclosure may include at least one interface (not shown). The interface may input and / or output data. The device (10000) may receive an input query through the interface (not shown). The interface may be an electronic circuit capable of receiving an input query such as a prompt, gesture, or voice. The input and / or output interface may include, for example, a mouse, keyboard, touchpad, touchscreen, microphone, wireless communication device, wired communication device, camera, speaker, etc.

[0323] The system-on-chip (SoC) of the device (10000) according to the third example of the present disclosure can perform inference operations on an upper model (UM) and a lower model (LM). The upper model (UM) and the lower model (LM) can cooperate with each other to process speculative decoding. The neural processing unit (NPU) of the system-on-chip (SoC) of the device (10000) according to the third example of the present disclosure can process the upper model (UM) and the lower model (LM) alternately. That is, the neural processing unit (NPU) can operate to process the upper model (UM) and the lower model (LM) sequentially. In other words, the neural processing unit (NPU) can operate to process the upper model (UM) and the lower model (LM) in a time-sharing manner. Thus, the upper model (UM) and the lower model (LM) can be processed sequentially while increasing the utilization rate of the computational circuits of the neural processing unit (NPU).

[0324] According to the disclosed embodiment, speculative decoding is performed by sequentially processing the upper model (UM) and the lower model (LM) in the neural processing unit (NPU) of the device (10000), and the parameters of the upper model (UM) and the lower model (LM) reside in memory (MEM), thereby reducing the bottleneck caused by the speculative decoding operation.

[0325] The memory (MEM) of the device (10000) according to the third example of the present disclosure is configured to store all weight parameters of the upper model (UM) and the lower model (LM) when the device (10000) is operating. By storing all first weight parameters of the upper model (UM) and second weight parameters of the lower model (LM) in the memory (MEM), the device (10000) can efficiently process speculative decoding. The memory (MEM) of the device (10000) according to the third example of the present disclosure can provide the parameters of the upper model (UM) to the neural processing unit (NPU) when the neural processing unit (NPU) infers the upper model (UM), and provide the parameters of the lower model (LM) to the neural processing unit (NPU) when the neural processing unit (NPU) infers the lower model (LM).

[0326] For example, the device (10000) can set an address map of memory (MEM) during a speculative decoding operation to allocate a memory bank in which weight parameters of the upper model (UM) and the lower model (LM) are stored. Thus, during a speculative decoding operation, weight parameters of both the upper model (UM) and the lower model (LM) can reside in memory (MEM). That is, the capacity of memory (MEM) of the device (10000) according to the third example of the present disclosure may be large, as the total capacity occupied by the first weight parameters of the upper model (UM) and the second weight parameters of the lower model (LM) may be large. If the memory (MEM) of the device (10000) does not have a memory capacity capable of storing the upper model (UM) and the lower model (LM), it may be difficult to process speculative decoding quickly in real time. That is, the capacity of the memory (MEM) can be determined by considering the size of the parameters of the upper model (UM) and the lower model (LM) to be operated in the device (10000). If the parameters of the upper model (UM) and the lower model (LM) are not all stored in the memory (MEM), the conventional device must read the weight parameters of each model from an external storage device for speculative decoding. In this embodiment, the conventional device must command unnecessary memory read and memory write operations for speculative decoding, so the token generation speed may be lower than the token generation speed of the device (10000) according to the third example of the present disclosure.

[0327] A first memory (MEM1) of a device (200) according to a second embodiment of the first example of the present disclosure is configured to store weight parameters of a higher model (UM) and tokens generated from the higher model (UM) when the device (200) is operating.

[0328] The second memory (MEM2) of the device (200) according to the second embodiment of the first example of the present disclosure is configured to store weight parameters of the lower model (LM) and tokens generated from the lower model when the device (200) is operating. When the weight parameters of the upper model (UM) and the lower model (LM) are all stored in the first memory (MEM1) and the second memory (MEM2), the device (200) can efficiently process speculative decoding. To elaborate, the size of the weight parameters of the upper model (UM) and the lower model (LM) may be considerably large and may be difficult to store in a single memory chip. In such cases, the device (200) may configure multiple memories to increase the memory capacity.

[0329] A first memory (MEM1) of a device (200) according to a second embodiment of a first example of the present disclosure is configured to provide parameters of a higher model (UM) to a neural processing unit (NPU) when the neural processing unit (NPU) infers a higher model (UM), and a second memory (MEM2) is configured to provide parameters of a lower model (LM) to a neural processing unit (NPU) when the lower model (LM) is inferred.

[0330] For example, the device (200) can set the address maps of the first memory (MEM1) and the second memory (MEM2), respectively, during a speculative decoding operation to allocate memory banks where weight parameters of the upper model (UM) and the lower model (LM) are stored. Thus, during a speculative decoding operation, weight parameters of the upper model (UM) and the lower model (LM) can reside in both the first memory (MEM1) and the second memory (MEM2).

[0331] Accordingly, the capacity of the plurality of memories (MEM1, MEM2) of the device (200) according to the second embodiment of the first example of the present disclosure is configured to be greater than the total capacity occupied by the weight parameters of the upper model (UM) and the lower model (LM). Thus, the weight parameters of the upper model (UM) and the lower model (LM) can reside in the plurality of memories (MEM1, MEM2). If the capacity of the upper model (UM) and the lower model (LM) in the plurality of memories (MEM1, MEM2) of the device (200) is insufficient, it may be difficult to process speculative decoding efficiently.

[0332] If the parameters of the upper model (UM) and the lower model (LM) are not all stored in multiple memories (MEM1, MEM2), the conventional device must read the weight parameters of each model from an external storage device for speculative decoding. Consequently, the conventional device commands unnecessary memory read and memory write operations for speculative decoding. As a result, the token generation speed of the conventional device is reduced compared to the token generation speed of the device (200) according to the second embodiment of the first example of the present disclosure.

[0333] In summary, FIGS. 19 and 20 illustrate speculative decoding according to a third example of the present disclosure. Furthermore, the speculative decoding method according to a third example of the present disclosure can be performed in the apparatus (10000) described with reference to FIGS. 3 through 18. To support such efficient on-device speculative decoding, the apparatus (10000) according to the present disclosure may be based on the following optimized hardware architecture and system configuration.

[0334] That is, the device (10000) according to the present disclosure includes a low-power NPU (neural processing unit, see 100 in FIG. 3) core architecture, and the low-power NPU core architecture has a separate structure of an integer operation-centered processing core (P_CORE, see 120 in FIG. 3) and a special function unit (SFU, see 150 in FIG. 3) for floating-point operations and complex function processing. The special function unit (SFU, 150) is configured to include dedicated hardware accelerators for functions such as exponential and reciprocal, which are particularly essential for Softmax and attention operations (e.g., NEXP, RECIP circuits included in the vector function approximation unit (V_FuncApx, see 1537 in FIG. 5) or scalar function approximation unit (S_FuncApx, see 1545 in FIG. 6) within the SFU (150) (see FIG. 27), and a programmable function approximation (PAF) unit that supports various activation functions (e.g., function approximation circuits included in the function approximation unit (see FIG. 24, FIG. 25)).

[0335] Accordingly, the device (10000) includes a large-capacity on-device memory system, said memory system includes a main memory (e.g., external DRAM, see 2000 in FIG. 9 or 2000-1 to 2000-N in FIG. 10) with a capacity sufficient to store all weights of the LLM and even the weights of an auxiliary model that can be used for speculative decoding, and may include an on-chip shared memory (e.g., SRAM, see 400 in FIG. 11 or 400-1 to 400-N in FIG. 12) with a capacity sufficient for high-speed access and reuse of frequently used weights and intermediate activation values. Accordingly, said memory system may have a hierarchical structure.

[0336] Therefore, NPU acceleration-based speculative decoding can be effectively implemented through the optimized hardware configuration of the NPU core (100) and memory system. According to the above configuration, the optimized NPU core (100), rather than being a simple sum of individual technologies, is organically combined with a large-capacity memory system (e.g., 2000, 400) and a speculative decoding algorithm to provide a synergistic effect that enables large-scale language models, which previously required server-level computing power, to be executed in a fully on-device form without response delays while minimizing energy consumption. This can enable the implementation of a practical server-independent AI agent.

[0337] In summary, a low-power on-device large-scale language model (LLM) inference system according to the present disclosure can achieve the above effect by including the following components: a processing core (120) that performs integer-based matrix operations on input data; a special function unit (SFU, 150) that performs floating-point-based special operations (including exponentiation, reciprocal, and programmable function approximation) on the result of the integer operations, wherein the special operations are optimized for the attention and activation function processing of the LLM; a main memory (2000) that stores the total weights of the LLM and at least one auxiliary model weight for speculative decoding; and an on-chip shared memory (400) that temporarily stores data frequently accessed by the processing core (120) or the SFU (150) to increase data reusability. A controller (e.g., a controller (110) within an NPU (100) or a system-level control unit including the same) that controls the processing core (120), the SFU (150), the main memory (2000), and the on-chip shared memory (400) to perform speculative decoding operations including candidate token generation using the auxiliary model and candidate token verification using the LLM. Through this system configuration, it is possible to complete the inference of the LLM at low power without the assistance of an external server.

[0338] The ability of the system according to the present disclosure to efficiently perform speculative decoding is not simply due to the fast computational speed of the NPU (100). This may be an important synergistic effect resulting from the fact that the unique hierarchical / distributed memory system proposed in the present disclosure (see FIG. 11 to FIG. 15) is co-designed to perfectly match the unique data access pattern of the speculative decoding algorithm.

[0339] Conjecture decoding is a technique that accelerates token generation by using a larger model (UM) with many parameters and a smaller model (LM) with few parameters together. In this process, rapid and frequent data exchange between the weights of the two models and intermediate computation results (such as KV cache) is essential. In existing single-memory structures, severe memory bottlenecks can occur during the process of swapping the parameters of the two large models, which can degrade the efficiency of the algorithm.

[0340] The system of the present disclosure may adopt a multilayered memory structure to solve these problems. For example, in an embodiment such as FIG. 13 or FIG. 15, the total weights of a high-level model (UM) that is accessed relatively infrequently but is vast in size may be resident in a large-capacity external main memory (2000-1, …, 2000-N) or in an NPU-dedicated memory (11400) within an AI module (11000). At the same time, data such as the weights of a low-level model (LM) that must be accessed very frequently during the inference process and the KV cache of both models may be cached or stored in an ultra-high-speed on-chip shared memory (SRAM) (400-1, …, 400-N).

[0341] This intelligent data placement strategy is made possible by the hierarchical memory structure provided by the hardware architecture of the present disclosure. This fully supports the data access requirements of speculative decoding algorithms (accessing some data very quickly and others less quickly) at the hardware level, thereby minimizing unnecessary data movement and latency. In other words, the hardware architecture and software algorithms are not optimized independently but are organically combined to enhance each other's strengths, contributing to the improvement of the AI ​​inference performance of the entire system.

[0342] The fourth example of the present disclosure is described below. The fourth example of the present disclosure describes the circuit configuration and operation method of the vector function approximation unit (V_FuncApx) and the scalar function approximation unit (S_FuncApx) illustrated in FIGS. 5 and 6.

[0343] FIG. 21 is a flowchart illustrating an activation function programming method according to the first embodiment of the fourth example of the present disclosure.

[0344] With reference to FIG. 21, the operation of the vector function approximation unit (1537) and the scalar function approximation unit (1545) is described. The vector function approximation unit (1537) and the scalar function approximation unit (1545) are configured to process a programmed activation function to be described later.

[0345] The activation of a neural network model can be determined based on the transmission strength of signals transmitted between layers by an activation function. The inference accuracy of the neural network model may vary depending on the activation function implementation technology of the neural processing unit (100) according to the fourth example of the present disclosure. That is, the inference accuracy and efficiency of the neural network model may vary depending on the hardware implementation characteristics of the processing circuit of the activation function of the neural processing unit (100).

[0346] Accordingly, the activation function programming method according to the first embodiment of the fourth example of the present disclosure may include the step of generating segment data for segmenting an activation function (S210), the step of segmenting the activation function into a plurality of segments using the generated segment data (S220), and the step of approximating at least one of the plurality of segments as a programmable segment (S230).

[0347] The activation function programming method can be performed by an activation function conversion program (not shown). The activation function conversion program (not shown) may be a computer program, software, firmware, application, or executable code stored on a recording medium. The activation function conversion program (not shown) may be a computer program configured to execute instructions that convert an activation function into a programmed activation function expressed by programmable parameters (i.e., operation coefficients). The activation function conversion program (not shown) may be stored on a computer-readable recording medium. The activation function conversion program (not shown) may be included in a compiler (not shown).

[0348] Segment data is generated in step S210. Segment data is data generated to segment a specific activation function into multiple segments. Segment data will be described later.

[0349] In step S220, the activation function can be segmented into multiple segments using the generated segment data. In this disclosure, the term 'segment' refers to a part of an activation function divided into multiple intervals and is distinguished from 'candidate segment' or 'programmable segment,' terms related to the approximation of the activation function.

[0350] Step S220 may include a step of determining the number and width of a plurality of segments based on segment data. In Step S220, the number of segments segmenting the activation function to be transformed and the width of each of the plurality of segments may be determined using the segment data. At least one of the plurality of segments may have a width that is the same as or different from that of another segment.

[0351] The interval of multiple segments can also be represented by the coordinates of the start and end points on the X-axis. On the other hand, it should be understood that once the number of multiple segments and the width of each of the multiple segments are determined, the coordinates of the interval of the multiple segments can be obtained using the number and width of the multiple segments.

[0352] In step S230, at least one of the multiple segments is approximated as a programmable segment. The programmable segment can be programmed according to the circuit design of the vector function approximation unit (1537) and the scalar function approximation unit (1545). That is, the activation function transformation program (not shown) can be configured to program the activation function to be processed by the neural processing unit (100) based on the hardware configuration of the vector function approximation unit (1537) and the scalar function approximation unit (1545). The programmable parameter may be data generated in step S230. The programmable parameter may be configured to have a form compatible with the circuits of the vector function approximation unit (1537) and the scalar function approximation unit (1545) of the neural processing unit (100). For example, each of the vector function approximation unit (1537) and the scalar function approximation unit (1545) may be configured to have hardware configured to perform operations on each programmable segment with a specific gradient and a specific offset. The activation function transformation program (not shown) may be configured to receive configuration information of the vector function approximation unit (1537) and the scalar function approximation unit (1545).

[0353] In this case, an activation function transformation program (not shown) can program a programmable segment in the form of a first-order function or a second-order function or higher having a slope and an offset. For example, the programmable segment may be approximated as a first-order function according to a specific judgment criterion. In this case, the activation function transformation program (not shown) can generate a programmable segment expressed in the form of '(slope a) * (input value x) + (offset b)'. The aforementioned specific slope and specific offset may be programmable parameters. For a programmable segment determined to be approximated as a first-order function, step S230 may include the step of approximating a selected segment with specific slope and specific offset values.

[0354] To elaborate, steps S220 and S230 can effectively be performed as a single step. This is because the step of segmenting the segments and the step of generating programmable parameters for the corresponding programmable segments can be performed simultaneously. To elaborate, steps S220 and S230 can be modified to perform the step of segmenting an activation function into multiple segments using the generated segment data, and approximating at least one of the multiple segments as a programmable segment.

[0355] Meanwhile, the above activation function programming method, in particular the process of generating optimal segment data (e.g., boundary values ​​and widths of segments) and deriving optimal programmable parameters for approximating each segment, can be performed in a systematic and automated manner by substituting it into a learning problem of a deep learning model according to another embodiment of the present disclosure.

[0356] Conventionally, the process of finding optimal approximation parameters relied on empirical or complex numerical analysis, making it difficult to guarantee optimal approximations for all nonlinear functions. However, the present disclosure allows for the use of a small-scale artificial neural network optimized for representing a piecewise linear function. Through this artificial neural network, training can be performed to most precisely mimic the target activation function to be approximated.

[0357] During the learning process, the artificial neural network optimizes internal weights and biases in a direction that minimizes the loss function, and in this process, the optimal segmentation point (bend point) that best represents the non-linear characteristics of the target function and the linear relationship between each segment are automatically determined. Once learning is complete, a set of programmable parameters can be finally extracted from the optimized artificial neural network parameters, including segment boundary values ​​(SB) that can be directly used by the programmable function approximation circuit (PAF), the gradient (A) and offset (B) of each segment. Through this deep learning-based approach, as shown in FIG. 23, it is possible to automatically derive optimal segments with varying widths and numbers depending on the characteristics of the function and generate the most accurate approximation parameters for them.

[0358] FIG. 22 is a graph showing the process of approximating an activation function by an activation function programming method according to the first embodiment of the fourth example of the present disclosure.

[0359] The activation function illustrated in FIG. 22(a) is segmented into multiple segments (s1, s2, s3, s4) using segment data as illustrated in FIG. 22(b). The multiple segments (s1, s2, s3, s4) are approximated into programmable segments (a1x+b1, a2x+b2, a3x+b3, a4x+b4) as illustrated in FIG. 22(c). Here, an example is described in which an activation function transformation program (not shown) generates programmable parameters such that all programmable segments correspond to a linear function.

[0360] Each programmable segment includes a corresponding programmable parameter. In FIG. 22 (c), all of the multiple segments are approximated as programmable segments in the form of a first-order function. However, in various examples, some of the multiple segments may be approximated as programmable segments of a different form. For example, an activation function transformation program (not shown) can program each programmable segment in the form of a first-order function, a second-order function, a third-order function, a logarithmic function, etc.

[0361] For example, only segments (s1, s3, s4) are approximated as programmable segments, and segment (s2) can be approximated using various methods available on the device where the activation function is processed. Specifically, if a predetermined stored lookup table, a non-linear approximation formula, etc., are available in the hardware for the interval of segment (s2), segment (s2) can be approximated using such a predetermined stored lookup table, a non-linear approximation formula, etc.

[0362] In other words, the activation function transformation program (not shown) can be configured to program each segment (s1, s2, s3, s4) independently. In this case, the activation function transformation program (not shown) receives hardware configuration information of the vector function approximation unit (1537) and the scalar function approximation unit (1545). The activation function transformation program (not shown) can be configured to independently determine the approximation method for each of the segments (s1, s2, s3, s4) based on the hardware configuration information of the vector function approximation unit (1537) and the scalar function approximation unit (1545).

[0363] For example, the vector function approximation unit (1537) and the scalar function approximation unit (1545) may each be designed to include at least one of a linear function approximation circuit, a quadratic function approximation circuit, a reciprocal function approximation circuit, a root mean square function approximation circuit, and a negative exponential function approximation circuit. In this case, the activation function transformation program (not shown) may optionally program at least one of each segment (s1, s2, s3, s4) in the form of a linear function, a quadratic function, a reciprocal function, a root mean square function, and a negative exponential function.

[0364] For example, if the vector function approximation unit (1537) and the scalar function approximation unit (1545) are each configured to include a circuit configured to support at least one specific function operation, the activation function transformation program (not shown) can program each segment (s1, s2, s3, s4) into the form of a corresponding specific function.

[0365] For example, the vector function approximation unit (1537) and the scalar function approximation unit (1545) may be configured to include at least one of a first function operation circuit, a second function operation circuit, an inverse function operation circuit, a root mean square function operation circuit, and a negative exponential function operation circuit or a similar function operation circuit designed in hardware.

[0366] For example, an activation function transformation program (not shown) can program the same activation function in different ways.

[0367] For example, an activation function transformation program (not shown) can program a specific activation function as only a first-order function.

[0368] For example, an activation function transformation program (not shown) can program a specific activation function only as a quadratic function.

[0369] For example, an activation function transformation program (not shown) can program a specific activation function only as an inverse function.

[0370] For example, an activation function transformation program (not shown) can program a specific activation function only as a root mean square function.

[0371] For example, an activation function transformation program (not shown) can program a specific activation function as a negative exponent function only.

[0372] For example, an activation function transformation program (not shown) can program each of multiple segments of a specific activation function with a corresponding approximation function.

[0373] For example, an activation function transformation program (not shown) can program multiple segments of a specific activation function into a set of approximation functions of different formulas.

[0374] For example, the activation function programmed to be processed in the vector function approximation unit (1537) and the scalar function approximation unit (1545) according to the first embodiment of the fourth example of the present disclosure may include at least one of Swish, Mish, Sigmoid, Hyperbolic Tangent (Tanh), Scaled Exponential Linear Unit (SELU), Gaussian Error Linear Unit (GELU), Softplus, Rectified Linear Unit (ReLU), Leaky ReLU, Maxout, Exponential Linear Unit (ELU), Hard Sigmoid, Hard Tanh, Bent Identity, ArcTan, Softsign, Sinusoidal Activation, S-shaped Rectified Linear Unit (SReLU), Parametric Rectified Linear Unit (PReLU), Logistic Activation, Radial Basis Function (RBF), Binary Step, Reciprocal, Root mean square, Reciprocal root mean square, and Negative exponential. Each programmed activation function can be converted into a programmable parameter and provided to a neural processing unit (100).

[0375] FIG. 23 is a graph showing various cases of segmenting an activation function into a plurality of segments by an activation function programming method according to the first embodiment of the fourth example of the present disclosure.

[0376] Referring to Fig. 23 (a), a specific activation function can be segmented with a uniform width and four segments.

[0377] Referring to Fig. 23(b), a specific activation function can be segmented with four segments and different widths.

[0378] Referring to Fig. 23 (c), a specific activation function can be segmented with four segments and different widths.

[0379] Referring to Fig. 23 (d), a specific activation function can be segmented with different widths and six segment numbers.

[0380] The number of multiple segments and the width of each of the multiple segments can be determined using segment data.

[0381] An activation function transformation program (not shown) may be configured to analyze the nonlinearity of an activation function so that multiple segments are segmented with different widths. However, the present disclosure is not limited thereto.

[0382] An activation function transformation program (not shown) may be configured to analyze the nonlinearity of the activation function so that each of the multiple segments is segmented to an optimal width. However, the present disclosure is not limited thereto.

[0383] In the present disclosure, the activation function may be implemented in various forms including characteristic intervals. When segmenting the activation function into multiple segments, the number and width of the multiple segments may be determined in various ways depending on the various forms of the activation function.

[0384] For example, various activation functions such as the swish function, Mish function, sigmoid function, hyperbolic tangent (tanh) function, SELU function, GELU (Gaussian Error Linear Unit) function, SOFTPLUS function, ReLU function, Leaky ReLU function, Maxout function, ELU function, etc., have various forms that are divided into multiple characteristic intervals including (actually) linear intervals and / or non-linear intervals. Accordingly, when approximating a non-linear activation function to be processable in hardware, segmenting by considering these characteristic intervals—that is, determining the number and width of segments by considering (actually) linear intervals, non-linear intervals, etc.—allows for more efficient approximation of the activation function corresponding to the characteristics of each activation function.

[0385] FIG. 24 is a diagram showing a function approximation circuit of a neural processing unit configured to process a programmed activation function according to the second embodiment of the fourth example of the present disclosure.

[0386] The programmable function approximation circuit (PAF) according to the second embodiment of the fourth example of the present disclosure illustrated in FIG. 24 may correspond to the vector function approximation circuit (V_PAF) of the vector function approximation unit (1537) of FIG. 5 and the scalar function approximation circuit (S_PAF) of the scalar function approximation unit (1545) of FIG. 6. For convenience of explanation, the vector function approximation circuit (V_PAF) and the scalar function approximation circuit (S_PAF) will be collectively referred to as the programmable function approximation circuit (PAF) to describe the second embodiment of the fourth example of the present disclosure.

[0387] Specifically, the programmable function approximation circuit (PAF) according to the second embodiment of the fourth example of the present disclosure illustrated in FIG. 24 is an example of a circuit configured to process an activation function programmed as a first-order function.

[0388] A programmable function approximation circuit (PAF) configured to process an activation function programmed as a first-order function may be configured to include a plurality of comparators (COMP0 to COMP(N-2)), a selector (SLT), a multiplier (PAF MUL), and an adder (PAF ADD). However, the present disclosure is not limited thereto, and it is also possible to configure the circuit in various ways to distinguish the regions of each segment. Furthermore, the programmable function approximation circuit (PAF) may be modified to include additional circuit configurations to process the activation function using programming methods other than a first-order function.

[0389] The coordinates of the start and end points of the intervals of multiple segments are defined by segment boundary values ​​(SB). That is, the intervals of multiple segments can each be determined by segment boundary values ​​(SB0 to SB(N-2)). For example, to define the intervals of N segments, N-1 segment boundary values ​​(SB0 to SB(N-2)) may be required.

[0390] Each of the multiple comparators (COMP0 to COMP(N-2)) compares the input value (X) with each of the multiple segment boundary values ​​(SB0 to SB(N-2)). Accordingly, the interval of the segment to which the input value (X) belongs among the intervals of the multiple segments can be determined through the comparison values ​​output from each of the multiple comparators (COMP0 to COMP(N-2)). The output values ​​output from each of the above-described multiple comparators (COMP0 to COMP(N-2)) can be referred to as interval determination data (SDD). The selector (SLT) outputs the slope (A) for the programmable segment corresponding to the interval of the segment to which the input value (X) belongs among the multiple slopes (A0 to A(N-1)) for the multiple programmable segments according to the interval determination data (SDD0 to SDD(N-2)). The selector (SLT) outputs, according to interval determination data (SDD0 to SDD(N-2)), an offset (B) for a programmable segment corresponding to the interval of the segment to which the input value (X) belongs among a plurality of offsets (B0 to B(N-1)) for a plurality of programmable segments.

[0391] When a specific segment is determined based on interval determination data (SDD), the slope (A) and offset (B) of the determined segment can be output from the selector (SLT). That is, the programmable parameter includes the slope (A) and offset (B) for each of the programmable segments. The slope (A) is multiplied with the input value (X) in the multiplier (PAF MUL), and the offset (B) is added to the output value of the multiplier (PAF MUL) in the adder (PAF ADD). That is, the output of the multiplier (PAF MUL) is It can be expressed as, and the output of the adder (PAF ADD) is It can be expressed as.

[0392] Meanwhile, the operation of each of the multiple comparators (COMP0 to COMP(N-2)) can be determined by each of the multiple comparator enable signals (Comp En 1 to Comp En (N-2)). For example, even when an activation function transformation program (not shown) programs the same activation function, the first programmed activation function can be programmed to have 10 segments, and the second programmed activation function can be programmed to have 5 segments. Accordingly, the programmable function approximation circuit (PAF) can control the number of comparators activated in the programmable function approximation circuit (PAF) differently according to each programmed activation function data, even for the same activation function. Accordingly, the accuracy of the artificial neural network operation and the power consumption of the neural processing unit (100) can also vary depending on the programming. That is, depending on the user's requirements, it is possible to provide a high-performance activation function operation function or a low-power activation function operation function, even for the same activation function.

[0393] Meanwhile, the number of multiple comparators that take the segment boundary values ​​(SB) as input must vary depending on the maximum number of segment boundary values ​​(SB). For example, if the maximum number of segment boundary values ​​(SB) is 10, at least 10 comparators may be provided. That is, the minimum number of comparators can be the maximum number of segment boundary values.

[0394] Accordingly, each of the multiple comparators (COMP0 to COMP(N-2)) can determine whether to operate based on each of the multiple comparator enable signals (Comp En 1 to Comp En (N-2)). Thus, by controlling the operation of unnecessary comparators according to the number of multiple segments, the power consumption of the neural processing unit (100) can be reduced.

[0395] However, due to hardware limitations, the number of comparators may be limited. Accordingly, the number of multiple segments segmenting the activation function may be limited depending on the number of comparators of the programmable function approximation circuit (PAF).

[0396] Meanwhile, according to the first embodiment of the fourth example of the present disclosure, it is possible to distinguish between linear and non-linear sections of the activation function, and to minimize the number of segments by providing a variable segment width while minimizing the error value. Therefore, there is an advantage in that the number of gate counts of the hardware of the programmable function approximation circuit (PAF) of the neural processing unit (100) can be minimized by minimizing the number of comparators.

[0397] Additionally, the activation function programming method according to the first embodiment of the fourth example of the present disclosure may be configured to program a specific activation function based on information of a maximum comparator that can be supported.

[0398] Meanwhile, the selector (SLT) may be a multiplexer composed of multiple switching elements, but the configuration of the selector (SLT) can be varied. The programmed activation function operation unit of the programmable function approximation circuit (PAF) may refer to a circuit configured to receive input values ​​(X), slope (A), and offset (B) and calculate an output value (Y). The programmed activation function operation unit of the programmable function approximation circuit (PAF) may be configured to include at least one multiplier (PAF MUL) and one adder (PAF ADD). The programmed activation function operation unit of the programmable function approximation circuit (PAF) may be a hard-wired circuit. In particular, if the programmable function approximation circuit (PAF) is a hard-wired circuit, the programmable function approximation circuit (PAF) can process approximation operations in clock units by inputting only programmable parameters without complex instructions.

[0399] Meanwhile, the configuration in which the programmable function approximation circuit (PAF) according to the present disclosure stores programmable parameters (A, B, C, etc.) and uses them through a selector (SLT) may appear outwardly similar to the conventional lookup table (LUT) method.

[0400] However, the parameter storage and utilization method of the present disclosure has the following fundamental differences from the conventional LUT method in its technical concept and operating principle.

[0401] First, the nature of the stored data is different. Conventional LUTs store 'final approximate result values' or 'representative values ​​for interpolation' for specific input ranges. This is a passive data table intended to retrieve pre-calculated result values ​​by looking up based on input addresses. In contrast, the programmable parameters stored in this disclosure are not result values, but 'operation coefficients' (i.e., programmable parameters) for controlling a hard-wired operation unit in real time.

[0402] Second, the role and operation method of the data are different. Conventional LUT methods are 'lookup-centric' operations that read values ​​from memory. In contrast, the present disclosure is a 'computation-centric' architecture that directly performs polynomial operations such as Ax+B by supplying operation coefficients selected through a selector (SLT) in real time to a 'Direct Computation Engine' composed of a multiplier (PAF_MUL) and an adder (PAF_ADD). That is, the parameter storage unit of the present disclosure functions as an active 'Coefficient Register File' that dynamically configures the operation circuit, and this is one of the technical features of the present disclosure that is clearly distinguished from conventional LUTs.

[0403] In particular, the above programmable parameters (A0…A(N-1), B0…B(N-1), etc.) have an important technical feature in that they are constant values ​​generated offline through the deep learning-based optimization method described above.

[0404] In other words, these parameters, finally determined through learning, are not changed at runtime when the Neural Processing Unit (NPU) performs the actual inference operation, and are stored in advance in designated memory or coefficient registers within the Programmable Function Approximation (PAF). Therefore, when performing the operation, the NPU immediately loads and processes these pre-optimized constant values ​​without complex calculations, thereby eliminating additional computational overhead and improving the speed of function approximation operations.

[0405] In addition, the function approximation unit (FuncApx) of the present disclosure has a structure that allows a single programmable function approximation circuit (PAF) to be commonly reused for approximating various different types of dedicated functions (e.g., reciprocal, square root reciprocal, negative exponent, etc.). The effect of this structure can be increased when combined with the learning-based parameter generation method. By having only an area-efficient single programmable function approximation circuit (PAF) and selectively loading and using a pre-learned optimal constant parameter set (i.e., programmable parameter set) according to the function to be processed, the inefficiency of designing a separate approximation circuit for each function is eliminated, and the utilization of hardware resources is increased while simultaneously ensuring high accuracy and flexibility.

[0406] Meanwhile, the means for storing programmable parameters (slope A, offset B, etc.) used by the PAF circuit of the present disclosure has a fundamental difference from a conventional lookup table (LUT) in its purpose and method of operation.

[0407] Conventional LUTs are passive memory tables that store 'final approximate result values' for specific input intervals and look up the corresponding result values ​​according to input addresses. In contrast, the means for storing parameters in the present disclosure may function as an active coefficient register file or an equivalent on-chip memory.

[0408] In other words, the values ​​stored here are not the final result, but 'coefficients' used to control hard-wired operations (multipliers, adders) in real time. The Selector (SLT) can read the coefficients corresponding to the determined segment from this register file and supply them to the 'Direct Computation Engine'. Therefore, this circuit is a 'computation-centric' architecture that calculates results by directly performing arithmetic operations, such as Ax+B, on input values, which is clearly distinct from the conventional 'lookup-centric' LUT method.

[0409] FIG. 25 is a diagram showing a function approximation circuit of a neural processing unit configured to process a programmed activation function according to the third embodiment of the fourth example of the present disclosure.

[0410] The programmable function approximation circuit (PAF) according to the third embodiment of the fourth example of the present disclosure illustrated in FIG. 25 may correspond to the vector function approximation circuit (V_PAF) of the vector function approximation unit (1537) of FIG. 5 and the scalar function approximation circuit (S_PAF) of the scalar function approximation unit (1545) of FIG. 6. For convenience of explanation, the vector function approximation circuit (V_PAF) and the scalar function approximation circuit (S_PAF) are collectively referred to as the programmable function approximation circuit (PAF) to describe the third embodiment of the fourth example of the present disclosure.

[0411] In describing the programmable function approximation circuit (PAF) of FIG. 25, descriptions that overlap with the programmable function approximation circuit (PAF) of FIG. 24 may be omitted for the convenience of explanation.

[0412] Specifically, the programmable function approximation circuit (PAF) according to the third embodiment of the fourth example of the present disclosure illustrated in FIG. 25 is an example of a circuit configured to process an activation function programmed as a quadratic function.

[0413] Accordingly, the coefficients (A, B, C) for the programmable segment described above may include the coefficient of quadratic term (A), the coefficient of linear term (B), and the offset (C).

[0414] A programmable function approximation circuit (PAF) according to the third embodiment of the fourth example of the present disclosure may be configured to include a plurality of comparators (COMP0 to COMP(N-2)), a selector (SLT), a plurality of multipliers (PAF_MUL1, PAF_MUL2, PAF_MUL3), and a plurality of adders (PAF_ADD1, PAF_ADD2).

[0415] Each of the multiple comparators (COMP0 to COMP(N-2)) compares an input value (X) calculated in at least one processing core (P_CORE) with each of the multiple segment boundary values ​​(SB0 to SB(N-2)).

[0416] Accordingly, the interval of the segment to which the input value (X) belongs among the intervals of the plurality of segments can be determined through the output values ​​output from each of the plurality of comparators (COMP0 to COMP(N-2)).

[0417] Meanwhile, the operation of each of the multiple comparators (COMP0 to COMP(N-2)) can be determined by each of the multiple comparator enable signals (Comp En 1 to Comp En (N-2)).

[0418] And, the selector (SLT) can output coefficients (A, B, C) for a programmable segment corresponding to the interval of the segment to which the input value (X) belongs, among coefficients (A0 to A(N-1), (B0 to B(N-1)), (C0 to C(N-1)) for a plurality of programmable segments.

[0419] Specifically, a pre-stored set of coefficients or coefficient registers can provide the coefficients of the second term (A0 to A(N-1)), the coefficients of the first term (B0 to B(N-1)), and the offset (C0 to C(N-1)) for each of the plurality of programmable segments to the selector (SLT).

[0420] And, the selector (SLT) can determine the segment to which the input value (X) belongs among the segments of the plurality of segments according to the segment determination data (SDD0 to SDD(N-2)) output from each of the plurality of comparators (COMP0 to COMP(N-2)).

[0421] And, the selector (SLT) can output the coefficients of the second term (A0 to A(N-1)), the coefficients of the first term (B0 to B(N-1)), and the offset (C0 to C(N-1)) for a plurality of programmable segments, the coefficients of the second term (A), the coefficients of the first term (B), and the offset (C) for a programmable segment corresponding to the interval of the determined segment.

[0422] Accordingly, the selector (SLT) can output the coefficient of the second term (A), the coefficient of the first term (B), and the offset (C) for the programmable segment corresponding to the interval of the segment to which the input value (X) belongs.

[0423] The programmed activation function operation part of the programmable function approximation circuit (PAF) may refer to a circuit part configured to receive input values ​​(X), coefficients of the second term (A), coefficients of the first term (B), and an offset (C) to calculate an output value (Y).

[0424] The programmed activation function operation unit of the programmable function approximation circuit (PAF) may be configured to include a plurality of multipliers (PAF_MUL1, PAF_MUL2, PAF_MUL3) and a plurality of adders (PAF_ADD1, PAF_ADD2) to process a second-order function or a first-order function.

[0425] The programmed activation function operation part of a programmable function approximation circuit (PAF) can be a hard-wired circuit.

[0426] A plurality of multipliers in the programmed activation function operation unit may include a first multiplier (PAF_MUL1), a second multiplier (PAF_MUL2), and a third multiplier (PAF_MUL3).

[0427] The first multiplier (PAF_MUL1) multiplies the input value (X) with the coefficient (A) of the quadratic term for the programmable segment corresponding to the interval of the segment to which the input value (X) belongs.

[0428] Specifically, the first multiplier (PAF_MUL1) multiplies the input value (X) calculated in at least one processing core (P_CORE) by the coefficient (A) of the second term for the programmable segment output from the selector (SLT).

[0429] Accordingly, the first multiplier (PAF_MUL1) can output the coefficient (A) of the quadratic term for the programmable segment by multiplying the input value (X). That is, the output of the first multiplier (PAF_MUL1) is It can be expressed as.

[0430] Then, the second multiplier (PAF_MUL2) multiplies the output value from the first multiplier (PAF_MUL1) with the input value (X).

[0431] Specifically, the second multiplier (PAF_MUL2) multiplies the input value (X) calculated by at least one processing core (P_CORE) by the output value output from the second multiplier (PAF_MUL2).

[0432] Accordingly, the output of the second multiplier (PAF_MUL2) is It can be expressed as. However, the above-described configuration is This is merely an example for implementing it, and variations can be implemented through various circuit combinations.

[0433] The third multiplier (PAF_MUL3) multiplies the input value (X) with the coefficient (B) of the first term for the programmable segment corresponding to the interval of the segment to which the input value (X) belongs.

[0434] Specifically, the third multiplier (PAF_MUL3) multiplies the input value (X) calculated in at least one processing core (P_CORE) by the coefficient (B) of the first term for the programmable segment output from the selector (SLT).

[0435] Accordingly, the third multiplier (PAF_MUL3) can output the coefficient (B) of the first-order term for the programmable segment by multiplying the input value (X). That is, the output of the third multiplier (PAF_MUL3) is It can be expressed as.

[0436] Multiple adders may include a first adder (PAF_ADD1) and a second adder (PAF_ADD2).

[0437] The first adder (PAF_ADD1) adds the output value of the third multiplier (PAF_MUL3) to the output value of the second multiplier (PAF_MUL2).

[0438] Specifically, the first adder (PAF_ADD1) can output the sum of the quadratic term and the linear term of each of a plurality of programmable segments composed of quadratic terms. That is, the output of the first adder (PAF_ADD1) is It can be expressed as.

[0439] Then, the second adder (PAF_ADD2) adds the offset (C) for the programmable segment corresponding to the interval of the segment to which the input value (X) belongs to the output value of the first adder (PAF_ADD1).

[0440] Specifically, the adder (PAF_ADD) adds the offset (C) for the programmable segment to the sum of the quadratic and linear terms of the programmable segment composed of quadratic terms. That is, the output of the second adder (PAF_ADD2) is It can be expressed as.

[0441] Accordingly, the adder (PAF_ADD) can output an activation value to which an activation function programmed as a quadratic function is applied to the operation value of the input value (X).

[0442] According to the configuration described above, the programmable function approximation circuit (PAF) operates to process second-order polynomials.

[0443] Meanwhile, the operation of the second multiplier (PAF_MUL2), the third multiplier (PAF_MUL3), and the second adder (PAF_ADD2) can be controlled by the first enable signal (EN1).

[0444] Specifically, when the second multiplier (PAF_MUL2), the third multiplier (PAF_MUL3), and the second adder (PAF_ADD2) are not operated by the first enable signal (EN1), the operation is as described below.

[0445] The first multiplier (PAF_MUL1) multiplies the input value (X) with the coefficient (A) of the quadratic term for the programmable segment corresponding to the interval of the segment to which the input value (X) belongs.

[0446] Specifically, the first multiplier (PAF_MUL1) multiplies the input value (X) calculated in at least one processing core (120) by the coefficient (A) of the second term for the programmable segment output from the selector (SLT).

[0447] Accordingly, the first multiplier (PAF_MUL1) can output the coefficient (A) of the quadratic term for the programmable segment by multiplying the input value (X). That is, the output of the first multiplier (PAF_MUL1) is It can be expressed as.

[0448] And, since the second multiplier (PAF_MUL2) and the third multiplier (PAF_MUL3) do not operate, the output of the first multiplier (PAF_MUL1) is input directly to the first adder (PAF_ADD1). That is, the arithmetic unit disabled by the first enable signal (EN1) can be bypassed.

[0449] Then, the first adder (PAF_ADD1) adds the coefficient (B) of the first term for the programmable segment corresponding to the interval of the segment to which the input value (X) belongs to the output value of the first multiplier (PAF_MUL1).

[0450] Specifically, the first adder (PAF_ADD1) adds the coefficient of the first term (B) for the programmable segment to the value obtained by multiplying the input value (X) by the coefficient (A) of the second term for the programmable segment. That is, the output of the first adder (PAF_ADD1) is It can be expressed as.

[0451] And, since the second adder (PAF_ADD2) does not operate, the output of the first adder (PAF_ADD1) is output as is. That is, the arithmetic unit disabled by the first enable signal (EN1) can be bypassed.

[0452] That is, the first adder (PAF_ADD1) can output an activation value to which an activation function programmed as a first-order function is applied to the operation value, which is the input value (X).

[0453] According to the configuration described above, the programmable function approximation circuit (PAF) operates to process first-order polynomials.

[0454] As described above, some configurations of a plurality of multipliers and a plurality of adders can be controlled by a first enable signal (EN1). Accordingly, depending on the first enable signal (EN1), the programmable function approximation circuit (PAF) can be driven not only when each programmable segment is a second-order polynomial, but also when each programmable segment is a first-order polynomial.

[0455] To elaborate, according to one example of the present disclosure, at least one processing core (120) and a pipelined programmable function approximation circuit (PAF) may be composed of a hard-wired circuit configured to implement both a second-order function and a first-order function activation function.

[0456] Therefore, there is an advantage in being able to handle various cases of a Programmable Function Approximation (PAF) with a single PAF.

[0457] Meanwhile, the programmable function approximation circuit (PAF) includes a plurality of comparators (COMP0 to COMP(N-2)) (COMP0 to COMP(N-2)), a selector (SLT), a plurality of multipliers (PAF_MUL1, PAF_MUL2, PAF_MUL3) and a plurality of adders (PAF_ADD1, PAF_ADD2), and may further include a logarithm operator.

[0458] FIG. 26 is a drawing showing an example of a number system processed in a special function unit of a neural processing unit that can be applied to the examples of the present disclosure.

[0459] The special function unit (150) of the neural processing unit (100) according to the examples of the present disclosure is designed to process at least one floating-point number system.

[0460] For example, the special function unit (150) may be designed to handle floating-point numbers. Referring to FIG. 26, the floating-point number is represented in the brain floating-point 16 format. Referring to FIG. 26, the floating-point number system of the BF16 format includes a sign bit (S), an exponent bit (E), and a mantissa bit (M).

[0461] However, the number system of the special function unit (150) according to the examples of the present disclosure may use IEEE standard 754, and other number systems may be applied to improve the computational efficiency, reduce memory usage, and reduce power consumption of the special function unit (150). However, the examples of the present disclosure are not limited thereto, and Dynamic Floating-Point, Variable Precision Floating-Point (VPFP), Custom Floating-Point (CFP), Flexible Floating-Point (FFP), etc. may be applied. Accordingly, the bit width allocation of the exponent part and the bit width allocation of the mantissa part may vary depending on the number system supported by the special function unit (150) of the neural processing unit (100).

[0462] For example, when a special function unit (150) according to the examples of the present disclosure is configured to use the FP32 number system, the special function unit (150) has 31 of weight parameters, input parameters, and output parameters st Recognize the bit as the sign bit (S), and 23 rd bit to 30 th bit is recognized as an exponent bit (E), and 0 th bit to 22 nd It is configured to recognize bit as a mantissa bit (M).

[0463] For example, when a special function unit (150) according to the examples of the present disclosure is configured to use the FP16 number system, the special function unit (150) has 15 of a weight parameter, an input parameter, and an output parameter.th Recognize the bit as the sign bit (S), and 10 th bit to 14 th bit is recognized as an exponent bit (E), and 0 th bit to 9 th It is configured to recognize bit as a mantissa bit (M).

[0464] For example, when a special function unit (150) according to the examples of the present disclosure is configured to use the Brain Floating 16 (BF16) number system, the special function unit (150) has 15 of a weight parameter, an input parameter, and an output parameter. th Recognize the bit as the sign bit (S), and 7 th bit to 14 th bit is recognized as an exponent bit (E), and 0 th bit to 6 th It is configured to recognize bit as a mantissa bit (M).

[0465] This can be expressed as a mathematical formula, as shown in <Mathematical Formula 5>.

[0466]

[0467] Here, x is the input in floating-point form, s represents the sign of the input (x), e represents the exponent of the input (x), and m represents the mantissa of the input (x). The bias of the BF16 is 127, and the actual value range of the exponent is -126 to 127. The actual exponent value can be obtained by subtracting the bias from the stored exponent.

[0468] FIG. 27 is a block diagram illustrating a function approximation part according to the fourth embodiment of the fourth example of the present disclosure.

[0469] With reference to FIGS. 5, 6, 24, 25, and 27, a function approximation unit (FuncApx) according to the fourth embodiment of the fourth example of the present disclosure will be described.

[0470] Referring to FIG. 27, a function approximation unit (FuncApx) according to the fourth embodiment of the fourth example of the present disclosure may include a programmable function approximation circuit (PAF), an inverse function circuit (RECIP), an inverse of a square root function circuit (RSQRT), and a negative exponential function circuit (NEXP). These circuits are individual circuits configured to independently process the operations of functions required for the attention score calculation of an artificial intelligence model.

[0471] According to the fourth embodiment of the fourth example of the present disclosure, the function approximation unit (FuncApx) of FIG. 27 may correspond to the vector function approximation unit (V_FuncApx) of FIG. 5 or the scalar function approximation unit (S_FuncApx) of FIG. 6. According to the fourth embodiment of the fourth example of the present disclosure, the programmable function approximation circuit (PAF) of FIG. 24 may correspond to the programmable function approximation circuit (PAF) of FIG. 25 or the programmable function approximation circuit (PAF) of FIG. 27. According to the fourth embodiment of the fourth example of the present disclosure, the reciprocal function circuit (RECIP) of FIG. 27 may correspond to the vector reciprocal function circuit (V_RECIP) of FIG. 5 or the scalar reciprocal function circuit (S_RECIP) of FIG. 6. According to the fourth embodiment of the fourth example of the present disclosure, the inverse square root function circuit (RSQRT) of FIG. 27 may correspond to the vector inverse square root function circuit (V_RSQRT) of FIG. 5 or the scalar inverse square root function circuit (S_RSQRT) of FIG. 6. According to the fourth embodiment of the fourth example of the present disclosure, the negative exponential function circuit (NEXP) of FIG. 27 may correspond to the vector negative exponential function circuit (V_NEXP) of FIG. 5 or the scalar negative exponential function circuit (S_NEXP) of FIG. 6.

[0472] Each of the programmable function approximation (PAF), reciprocal function (RECIP), reciprocal of the square root (RSQRT), and negative exponential function (NEXP) circuits is a hard-wired operation circuit. That is, when each of the above circuits receives input parameters, they can perform operations very quickly without complex instructions through the hard-wired operation circuits. That is, the function approximation unit (FuncApx) activates at least one circuit corresponding to the function to be processed. Subsequently, when the function approximation unit (FuncApx) provides input parameters to the activated circuit, the function is calculated along the pipeline of the calculation circuit without complex instruction control. That is, each of the programmable function approximation (PAF), reciprocal function (RECIP), reciprocal of the square root (RSQRT), and negative exponential function (NEXP) circuits is a dedicated operation circuit for the function corresponding to each circuit. To elaborate, hard-wired computational circuits can only process predefined operations and, due to their lack of versatility, cannot process other operations. However, each of the Programmable Function Approximation (PAF), Reciprocal Function (RECIP), Reciprocal of the Square Root (RSQRT), and Negative Exponential Function (NEXP) circuits can process predefined operations quickly and at low power. In particular, since all of the above computational circuits can be used for attention score calculation, transformer-based artificial intelligence models can be computed quickly.

[0473] For example, the function approximation unit (FuncApx) can be designed to process function approximation operations by activating only the programmable function approximation circuit (PAF). For example, the programmable function approximation circuit (PAF) can process at least one of Swish, Mish, Sigmoid, Hyperbolic Tangent (Tanh), Scaled Exponential Linear Unit (SELU), Gaussian Error Linear Unit (GELU), Softplus, Rectified Linear Unit (ReLU), Leaky ReLU, Maxout, Exponential Linear Unit (ELU), Hard Sigmoid, Hard Tanh, Bent Identity, ArcTan, Softsign, Sinusoidal Activation, S-shaped Rectified Linear Unit (SReLU), Parametric Rectified Linear Unit (PReLU), Logistic Activation, Radial Basis Function (RBF), Binary Step, Reciprocal, root mean square, and negative exponential as a programmable parameter input.

[0474] Furthermore, the function approximation unit (FuncApx) can be designed to process the corresponding reciprocal function, square root function, and negative exponential function more precisely by activating one of the reciprocal function circuit (RECIP), the reciprocal of the square root function circuit (RSQRT), and the negative exponential function (NEXP) circuit while activating the programmable function approximation circuit (PAF).

[0475] The feature of the aforementioned function approximation unit (FuncApx) lies in its hybrid architecture designed to achieve both generality and efficiency simultaneously.

[0476] That is, the function approximation unit (FuncApx) may be equipped with a programmable function approximation circuit (PAF) to flexibly respond to new non-linear functions of various forms or that may appear in the future, such as GELU and Swish, and a dedicated function circuit to process functions that are frequently used in neural network operations and have a fixed form, such as RECIP, RSQRT, and NEXP, at the highest speed and efficiency.

[0477] When the control unit (110) receives a command indicating the type of function to be processed, it can selectively activate either a PAF circuit or a specific dedicated function circuit accordingly. At this time, power supply to other circuit blocks that are not activated is cut off through power gating or clock gating to minimize unnecessary power consumption.

[0478] In addition, instead of each dedicated function circuit (RECIP, RSQRT, NEXP) having its own approximation circuit, a single PAF circuit can be reused as a Shared Operation Unit. Each dedicated function circuit can preprocess input values ​​according to its function characteristics (e.g., mantissa normalization) and then pass the result to the shared PAF circuit to delegate the approximation operation. In this case, the control unit can be configured to provide the PAF circuit with an optimal set of pre-learned programmable parameters (P_PARA-1, P_PARA-2, etc.) tailored to the corresponding function.

[0479] This structure can provide the effects of the present disclosure, which drastically reduce the hardware area compared to a method of implementing a separate complex approximation circuit for each function, while maintaining high accuracy and performance through preprocessing circuits and parameters optimized for each function.

[0480] The programmable function approximation (PAF) circuit (see FIG. 24, 25, 27) included in the special function unit (150) of the present disclosure may be a strategic component that goes beyond merely a means of implementing a specific activation function and provides the significant value of 'future-proofing' to the entire system. This can lead to a synergistic effect where hardware flexibility ensures the system's long-term lifecycle and investment value.

[0481] Artificial intelligence technology, particularly neural network models, is evolving very rapidly, and new, more efficient activation functions such as GELU, Swish, and Mish are continuously being proposed even after ReLU. If an AI accelerator is designed as fixed-function hardware that processes only a specific activation function (e.g., ReLU), that hardware may become technically obsolete and lose its value when a new standard activation function emerges. This poses a significant risk for users who invest massive amounts of money in AI infrastructure.

[0482] The PAF circuit of the present disclosure can fundamentally solve these problems. The PAF circuit is not fixed to a specific function but can approximate various forms of non-linear functions as polynomials by loading a set of programmable parameters (e.g., slope, offset coefficient) that are pre-calculated offline. This means that even if a new activation function, such as 'Super-GELU', is developed in the future, there is no need to design and fabricate a new chip; instead, a new set of parameters to approximate the function can simply be calculated and applied to the system via a software or firmware update.

[0483] Consequently, the hardware-level flexibility of the PAF circuit can transform the entire AI system from a static hardware asset into a dynamic and adaptable platform. This can create a powerful synergy effect by eliminating the risk of premature hardware obsolescence due to the rapid changes in AI technology and protecting the user's initial investment. This can be one of the technical features of the present disclosure that goes beyond simply providing high computational speeds and contributes to enhancing the sustainability and economic efficiency of the system from a long-term perspective.

[0484] FIG. 28 is a diagram schematically illustrating the basic architecture of an AI system according to the fifth example of the present disclosure.

[0485] <Example 5: Scalable AI System Architecture>

[0486] The fifth example of the present disclosure is described below. This example relates to a scalable AI system (10000) that can be configured to efficiently infer specific data by utilizing the advantages of the previously described NPU architecture (see FIG. 3). This system (10000) can aim to overcome limitations such as high power consumption, massive cost, and large physical size that a GPU-based system may have. To this end, it can aim to provide performance optimized for the requirements of various AI applications by actively utilizing the architectural advantages provided by an NPU (100) specialized for AI computation, namely a structure optimized for specific computation, efficient data flow, and an optimized memory hierarchy.

[0487] The most significant feature of the system (10000) is its modular expansion structure, which allows the overall AI computational performance of the system to be flexibly expanded or configured to meet user requirements through multiple slots and AI modules mounted thereon.

[0488] <AI 시스템의 구성 요소 및 유기적 연동>

[0489] FIG. 28 schematically illustrates the basic architecture of an AI system (10000) according to the present disclosure. The AI ​​system (10000) can be implemented in various forms, such as a tower case with a desktop structure or a case with a server rack structure, and a main board (10100) may be located at the center, on which major components are mounted and a bus (10150) is arranged to enable mutual communication. On the main board (10100), a main processor (10200) capable of acting as the brain of the system, a main memory (10300) utilized as a main workspace, a main storage device (10400) for permanent data storage, a communication module (10500) responsible for communication with the outside, and a plurality of slots (10600) which are elements for implementing the scalability of the system may be included.

[0490] The main processor (10200) may be composed of a CPU (Central Processing Unit) or an AP (Application Processor), and can perform a role similar to a control tower that executes the operating system (OS) of the AI ​​system (10000) and oversees the overall operation of the system. In particular, the main processor (10200) can analyze AI computation requests (queries) received through the communication module (10500), efficiently allocate and control tasks to each AI module (11000) mounted in multiple slots (10600), and direct the process of delivering the results of the computation back to the user. If the number of AI modules (11000) increases and the system load increases, a configuration that increases the number of main processors (10200) to flexibly respond to this can also be considered.

[0491] The main memory (10300) and the main storage device (10400) can serve as important infrastructure for storing data essential for the operation of the AI ​​system (10000). The main memory (10300) can be typically DRAM, but can be implemented with various types of memory such as ROM, SRAM, and HBM as needed. The main storage device (10400) is composed of a large-capacity storage medium such as a Hard Disk Drive (HDD) or a Solid State Drive (SSD) and can permanently store operating systems, applications, as well as specific neural network models required for AI computation and compiler software that generates machine code optimized for each AI module (11000). This main storage device (10400) can be connected to the main board (10100) via standard interfaces such as SATA, PCIe, and NVMe. When an AI application is executed, data from the main storage device (10400) is loaded into the high-speed main memory (10300) and rapidly accessed by the main processor (10200) and the AI ​​module (11000), thereby enabling a smooth data flow throughout the entire system.

[0492] The communication module (10500) can serve as a channel connected to the outside world through a wired or wireless network. Through this, the AI ​​system (10000) can receive various forms of AI service requests from the user and support smooth interaction in transmitting the sophisticated inference results processed by the AI ​​module (11000) back to the user.

[0493] The multiple slots (10600) and the AI ​​modules (11000) mounted thereon, which are the configuration of this example, can be considered the heart of the system, determining its performance and scalability. The number of slots provided on the main board (10100) can be configured to vary from 4 to 32, or from 16 to 64 as needed. Each slot supports a standard interface such as M.2 or E1.S, thereby providing a physical basis for easily mounting or removing standardized AI modules (11000). Additionally, each slot can be designed to operate at a low power of 30W or less, preferably 25W or 20W or less, which can contribute to efficiently managing the overall power consumption of the system even when multiple modules are mounted.

[0494] Each AI module (11000) may include at least one NPU (100) described in detail in the preceding examples of the present disclosure, thereby providing high-efficiency inference performance specialized for AI computation. The computational performance of this NPU (100) can be designed at various levels, such as 5 TOPS, 25 TOPS, and 50 TOPS, by applying existing processes such as 28nm and 14nm to the latest semiconductor microfabrication processes such as 7nm, 5nm, and further 2nm, and by adjusting the number of processing elements or MAC calculators included internally. This configuration can provide a basis for the total AI computational performance of the AI ​​system (10000) to be expanded in conjunction with the number of AI modules (11000) installed in multiple slots (10600). For example, if AI modules (11000) providing 25 TOPS of computational performance are installed in all eight slots, the AI ​​system (10000) can secure powerful computational performance of a total of 200 TOPS. Users can have the flexibility to easily expand and extend the system without interruption by simply adding an AI module (11000) after initially building the system with a single module, if higher performance is needed in the future.

[0495] <NPU 기반 아키텍처의 우수성과 GPU 아키텍처와의 본질적 차이>

[0496] The NPU (100) included in the AI ​​module (11000) of the present disclosure may appear similar to a GPU in that it processes AI computations, but due to fundamental differences in its architecture, it may exhibit essentially different characteristics in terms of processing speed and power efficiency in AI inference computations. A GPU is a general-purpose parallel processor designed to process numerous independent pixels simultaneously for graphic rendering, and due to this characteristic, it has a vast number of computational cores and has strengths in high-precision floating-point (FP32, FP64) computations.

[0497] However, AI inference, particularly neural network computations, is not merely a collection of parallel calculations but involves a structured data flow where formalized operations—such as matrix multiplication, convolution, and the application of activation functions—are processed sequentially according to the layer order. The general-purpose SIMT (Single Instruction, Multiple Thread) architecture of GPUs may not always optimally accommodate this data flow unique to AI inference, which can lead to unnecessary control overhead and inefficient data movement. Furthermore, GPUs may have a structure where data is constantly exchanged between computation cores and large, power-hungry external high-bandwidth memory (HBM, GDDR). This makes it difficult to utilize data locality during the AI ​​inference process, causing memory bottlenecks and consequently increasing the time computation cores spend idle while waiting for data. Consequently, inefficiencies may arise where the high theoretical computational performance (TOPS) figures of GPUs do not fully translate into actual AI inference speeds.

[0498] On the other hand, the NPU (100) architecture according to the present disclosure may aim to fundamentally resolve this inefficiency. The NPU (100) may be designed to be specialized only for AI inference operations. For example, it has a heterogeneous pipeline structure in which operations such as matrix multiplication, which account for most of the computational load, are handled by a processing core (120) optimized for low-power integer (INT8, INT4) operations, and complex non-linear operations requiring floating-point precision, such as activation functions and normalization, are handled by a separate special function unit (150). Compared to the GPU method of processing all operations with general-purpose cores, this eliminates unnecessary control overhead and improves computational efficiency by allocating hardware optimized for each operation. This can be much more efficient than the GPU method of processing all operations with general-purpose cores. In terms of data flow, the NPU (100) can improve data reusability through a hierarchical memory structure such as on-chip SRAM or dedicated LPDDR memory within the module, and minimize unnecessary external memory access by placing the data required for computation as close as possible to the computation unit.

[0499] This drastically reduces the time and power required for data movement, providing the potential to achieve superior power efficiency (TOPS / W) compared to GPUs when processing the same amount of computation. Consequently, it may not be certain that a GPU's TOPS is higher than that of an NPU (100), and that the GPU is guaranteed better performance in the same neural network model inference. This is because the architectural efficiency of the NPU (100), which optimizes data flow, memory access, and even the type of computational unit for the specific purpose of AI inference, may outweigh the inherent inefficiencies arising from the general-purpose structure of the GPU, thereby providing the potential to offer actually faster and more efficient performance.

[0500] Software Support and Application Scenarios

[0501] The AI ​​system (10000) can provide high utility through not only optimized hardware but also powerful software support. A compiler that may be included in the system can receive a neural network model developed with widely used AI frameworks such as TensorFlow, PyTorch, and ONNX, and convert it into optimized machine code that can be executed most efficiently on the NPU (100) mounted on the AI ​​module (11000) of the system. In this process, the compiler can improve processing efficiency on the NPU (100) by applying model lightweighting techniques such as quantization and pruning. Additionally, if the machine code compiled once is stored in the main storage device (10400), it can be loaded immediately without a compilation process when the same AI application is executed later, enabling fast and stable execution.

[0502] Through the organic combination of such hardware and software, the AI ​​system of the present disclosure can provide a powerful and flexible AI infrastructure platform that customers can consider applying to specific application fields.

[0503] Customers can utilize this AI system to envision various services based on AI video analysis, such as intelligent control, smart factories, and retail analytics. For instance, scenarios are possible where facial or fingerprint recognition-based systems are implemented for intelligent access control and security using various image classification models like Mobilenet, ResNet, VGG, and Inception. Furthermore, for smart city traffic analysis or in-store customer movement analysis, object detection models such as YOLO, SSD, and Faster R-CNN can be run to identify vehicles or people within images in real time, and integrated with DeepSORT or Siamese network-based tracking models to continuously monitor the movement of specific objects. In the medical field, U-Net, Mask R-CNN, and DeepLab models can be used to precisely segment specific organs or lesions in CT or MRI images at the pixel level to assist doctors in diagnosis, or to contribute to enhancing driving safety in autonomous driving systems by precisely segmenting various objects such as roads, lanes, and pedestrians. Furthermore, it is possible to develop smart fitness solutions that analyze a user's exercise posture and provide real-time feedback using posture estimation models such as OpenPose and HRNet, or to build a system that automatically detects specific events in surveillance footage, such as a pedestrian falling or a goal being scored in a soccer match, and notifies the manager using an event detection model that combines CNN with RNN or 3D CNN. Furthermore, it is possible to provide assistive devices for the visually impaired that describe the content of an image captured by a camera in speech, such as "a group of people playing soccer on the field," by running an Image-to-Text model that combines CNN with Transformer or RNN.

[0504] Furthermore, through AI sound analysis, customers can consider implementing next-generation voice-based interfaces and intelligent auditory analysis solutions. By utilizing models such as DeepSpeech and WaveNet, or high-performance transformer-based speech recognition models like Whisper and wav2vec, it is possible to provide AI assistants that control devices via voice commands, or automated meeting minutes systems that convert meeting content into text in real time and identify and organize speakers. By leveraging CNN and RNN-based sound detection models and acoustic feature analysis such as the Mel Frequency Cepstral Coefficient (MFCC), it is possible to create solutions that integrate with security systems by instantly detecting abnormal sound sources—such as gunshots, glass breaking, or crying babies—occurring in public places or sensitive facilities. Voice biometric recognition models like i-vector and x-vector can enable safe and convenient financial services by performing identity verification based on a user's unique voice characteristics (pitch, tone, rhythm), while applying DNN or GAN-based noise reduction models can help build communication systems that guarantee clear call quality even in noisy environments. In addition to this, long audio recordings such as podcasts can be automatically segmented by speaker or topic using Bayesian Information Criterion (BIC) or deep learning-based audio segmentation models to facilitate navigation.

[0505] Particularly in the field of natural language processing, this system leverages its powerful performance to support customers in developing innovative language-based AI services by efficiently utilizing Large Language Models (LLMs) such as LLaMA and Chat-GPT. Examples include automating customer support tasks by developing intelligent chatbots capable of 24 / 7 customer service, or implementing virtual assistants equipped with specialized knowledge in specific fields. Furthermore, it can be utilized to improve content creation efficiency by automatically generating various types of text—such as marketing copy, blog posts, and social media posts—based on given keywords or topics. Additionally, by using Transformer models trained on parallel corpora of multiple languages, it can provide real-time translation applications that enable communication without language barriers during travel or business meetings. It is also possible to build knowledge management solutions that drastically reduce information acquisition time by summarizing the core content of vast amounts of news articles, research papers, and legal documents. Going further, it can be used to develop next-generation search engines that accurately grasp the user's intent to go beyond simple information retrieval—understanding document content to generate direct responses—or provide text completion features that recommend contextually appropriate follow-up sentences when composing emails. Text can be classified into predefined labels such as sentiment and topic, and can also be utilized in services like social media monitoring or email spam filtering.

[0506] Finally, clients in the creative industry can consider innovating the way they visualize ideas through image generation scenarios. By combining image generation models such as Stable Diffusion, transformer-based text encoders such as CLIP, and the U-Net architecture, it is possible to instantly generate high-quality concept art, illustrations, and visual designs using only simple text prompts, such as "a silver robot horse running across a green meadow," thereby contributing to expanding the efficiency and possibilities of creative work.

[0507] <Example 6: Implementation of an AI Module with Standard Form Factor and High-Density Packaging>

[0508] The following sixth example of the disclosure relates to a specific implementation form of an AI module (11000) that can be mounted on the scalable AI system (10000) described in the fifth example. This example may include various embodiments applying industry standard form factors such as M.2 and E1.S, and further, high-density integrated packaging technology for physically mounting SoCs and memory. The adoption of such standard form factors and packaging technology can contribute to achieving economies of scale by increasing the mass producibility of the AI ​​module (11000) and improving AI performance under various physical constraints. Each embodiment of this sixth example may have an independent configuration in itself, and redundant content may be omitted for convenience of explanation. Additionally, each embodiment may be modified to suit various purposes and performance by selectively combining or referencing the technical features of each other.

[0509] The modular scalability of the AI ​​system according to the present disclosure has meaning beyond simply providing multiple slots. This flexible and economical scalability is realized only because of the fundamental architectural efficiency of the NPU (100) mounted in the individual AI module (11000), which may correspond to a significant synergistic effect between the two technical features.

[0510] Specifically, the NPU (100) has a heterogeneous pipeline structure in which matrix multiplication, which accounts for most of the computational load, is handled by a processing core (120) optimized for low-power integer operations, and complex non-linear operations, such as activation functions, are processed by a separate special function unit (150). This computationally specialized design can eliminate unnecessary control overhead compared to a GPU method that processes all computations with a general-purpose core and improve performance per power (TOPS / W). The AI ​​module (11000) of the present disclosure can be designed to operate at very low power of 25W or less, preferably 20W or less, based on such an NPU (100).

[0511] The 'low power, high efficiency' characteristics of these individual modules are a key prerequisite for enabling 'high-density scalability' of the entire system. For example, integrating 16 high-power GPUs of the 1000W class into a single chassis is a very difficult engineering task that requires a massive power supply of up to 16kW and a complex liquid cooling system. On the other hand, when 16 AI modules of the 25W class of the present disclosure are mounted in the same chassis, the total power consumption is only 400W, which can be sufficiently managed by standard air cooling.

[0512] Therefore, the scalability of the present disclosure may not be merely the result of a physical configuration with a large number of slots, but rather the result of the NPU’s architectural efficiency solving the power and heat generation issues of individual modules, thereby making high-density module integration physically and economically feasible. In other words, chip-level architectural innovation can provide a clear synergistic effect that creates new value in the form of flexible scalability at the system level.

[0513] < First Embodiment: AI Module Based on M.2 Form Factor >

[0514] FIG. 29 is a drawing illustrating an M.2 AI module (11000) according to the first embodiment of the sixth example of the present disclosure.

[0515] The present embodiment may aim to demonstrate the possibility of AI acceleration in a small form factor by presenting the most basic implementation form of the AI ​​module (11000). Referring to FIG. 29, the AI ​​module (11000) of the present embodiment may be configured in an M.2 form factor, and may include a printed circuit board (11100), an M.2 connector (11200) formed on one side of the board and responsible for physical and electrical connection with a main board slot, and an NPU (100), which is a core computational unit mounted on the printed circuit board (11100). The M.2 interface supports various interfaces including Serial ATA (SATA) and Peripheral Component Interconnect Express (PCIe), and in the case of PCIe, high-speed data transmission is possible by utilizing multiple lanes such as x2, x4, etc. In addition, it is available in various sizes (e.g., 2230, 2242, 2280, 22110) and can be flexibly applied to a wide range of devices such as laptops, desktops, and mini PCs.

[0516] The effect of this embodiment is to provide AI computation capabilities with low power consumption of about 10W or less. This low-power characteristic can create significant synergy by being directly linked to the scalability of the AI ​​system (10000) described in the fifth example. For example, if the AI ​​system (10000) provides 20 M.2 slots and 15 M.2 AI modules (11000) are installed therein, the AI ​​system (10000) can perform AI computation with relatively low power consumption of about 150W. If one AI module (11000) has a computational performance of 25 TOPS, the AI ​​system (10000) can secure a total AI computational performance of 375 TOPS, which means that it is possible to flexibly expand AI computational performance with superior power efficiency compared to a GPU-based system.

[0517] <Second Embodiment: M.2 AI Module Including NPU-Dedicated Memory>

[0518] FIG. 30 is a drawing illustrating an M.2 AI module (11000) according to the second embodiment of the sixth example of the present disclosure.

[0519] The second embodiment of the sixth example of the present disclosure is a modified embodiment, and for convenience of explanation, descriptions of redundant content may be omitted. The second embodiment aims to improve the efficiency and autonomy of AI computation by going further than the configuration of the first embodiment.

[0520] As illustrated in FIG. 30, the AI ​​module (11000) may be mounted on a printed circuit board (11100) along with an NPU (100) and NPU-dedicated memory (11400). For example, the dedicated memory may be LPDDR. Integrating large-capacity on-chip memory inside the NPU (100) may have limitations that drastically increase the physical size and manufacturing cost of the chip. On the other hand, the parameter size of the latest neural network models is rapidly increasing to hundreds of megabytes or more, making it difficult to process with only the limited on-chip memory of the NPU (100), so the main memory (10300) of the AI ​​system (10000) may need to be used frequently. However, since the main memory (10300) is shared with other components of the system, it may be difficult to always reliably guarantee the data bandwidth required by the NPU (100).

[0521] This embodiment can solve this problem by providing the AI ​​module (11000) with NPU-dedicated memory (11400), such as LPDDR memory with low-power characteristics. As a result, the NPU (100) can stably secure independent and high memory bandwidth without bus contention with other devices. By storing all parameters (weights, activation values, etc.) of a specific neural network model in this dedicated memory (11400), data exchange with the main memory (10300) of the AI ​​system (10000) can be minimized, and the computational performance of the AI ​​module (11000) can be stably maintained regardless of external factors. This can also contribute to making service switching between various AI applications fast and easy.

[0522] <Third Embodiment: High-performance M.2 AI module with multiple channel memories>

[0523] FIG. 31 is a drawing illustrating an M.2 AI module (11000) according to the third embodiment of the sixth example of the present disclosure.

[0524] The third embodiment of the sixth example of the present disclosure is a modified embodiment, and for convenience of explanation, descriptions of redundant content may be omitted.

[0525] The third embodiment aims to provide a high-performance module optimized for specific applications where memory bandwidth has a significant impact on performance. As shown in FIG. 31, a plurality of NPU-dedicated memories (11400) may be included corresponding to a single NPU (100). The two NPU-dedicated memories (11400) can improve the total memory bandwidth supplied to the NPU (100) by forming a plurality (e.g., dual-channel) memory configuration in which, for example, the four lanes provided by the PCIe interface are each divided into two.

[0526] The effect of such a multi-channel memory configuration can be enhanced, particularly in cases where memory-bound phenomena determine performance due to the massive size of parameters to be processed, such as in large-scale language models (LLM) or high-resolution image processing. Even with the same NPU (100) computational performance (TOPS), data can be processed faster than in a single-memory configuration as the memory bandwidth is doubled. As a result, this can lead to a significant improvement in actual perceived performance, such as the token generation speed of small-scale language models (SLM).

[0527] <Fourth Embodiment: E1.S AI Module Including Multiple NPUs>

[0528] FIG. 32 is a drawing illustrating an E1.S AI module (11000) according to the fourth embodiment of the sixth example of the present disclosure.

[0529] The fourth embodiment of the sixth example of the present disclosure is a modified embodiment, and for convenience of explanation, descriptions of redundant content may be omitted.

[0530] The fourth embodiment aims to provide an AI module (11000) suitable for data center and enterprise environments that require higher performance and density than M.2. To this end, an E1.S form factor may be utilized. The E1.S interface is a form factor designed as a standardized high-density solution for data center and enterprise storage systems, and can support high-speed data transfer by utilizing multiple lanes (x2, x4, x8, etc.) of the PCIe interface. The E1.S module generally has a rectangular shape with a height of 1U (approx. 44.45 mm), and can be configured with a width of up to 14 mm and a length of up to 300 mm or more. As such, it can have characteristics that allow for a physically larger size than M.2 and higher power consumption (e.g., 20 W or less), making it possible to integrate a larger number of NPUs (100) into a single module.

[0531] Referring to FIG. 32, the E1.S AI module (11000) of the present embodiment may include a plurality of NPUs (100) (e.g., two NPUs) along with an E1.S connector (11300) on a printed circuit board (11100). The effect of this configuration is that the computational performance of a single AI module can theoretically be expanded up to twice that of an M.2-based module, even when using the same NPU chip. For example, it is possible to configure two NPUs (100) with power consumption of 10W or less to be mounted on a 20W-class E1.S module (11000). This presents an effective method to standardize and systematically expand the performance of an AI module simply by changing the form factor.

[0532] <Fifth Embodiment: Multiple NPU E1.S AI Module with Dedicated Memory>

[0533] FIG. 33 is a drawing illustrating an E1.S AI module (11000) according to the fifth embodiment of the sixth example of the present disclosure.

[0534] The fifth embodiment of the sixth example of the present disclosure is a modified embodiment, and for convenience of explanation, descriptions of redundant content may be omitted.

[0535] The present embodiment is based on the configuration of the fourth embodiment and aims to prevent potential performance degradation that may occur when multiple NPUs (100) operate at maximum performance simultaneously and to ensure stable parallel processing. As shown in FIG. 33, each of the multiple NPUs (100) within the E1.S module (11000) may include one NPU-dedicated memory (11400) corresponding to each of the multiple NPUs.

[0536] Referring to FIG. 33, through a structural feature in which each NPU (100) has its own independent dedicated memory (11400), the effect of preventing interference in each other's memory access paths can be achieved even when two NPUs (100) perform computations simultaneously. This structure fundamentally prevents memory contention between NPUs and ensures that each NPU (100) can stably achieve maximum processing performance. This can contribute to maintaining the processing performance of the entire module stably. This can provide a particularly important advantage in a scenario where each NPU (100) processes independent AI applications in parallel.

[0537] <6th Embodiment: E1.S AI Module with Improved Performance>

[0538] FIG. 34 is a drawing illustrating an E1.S AI module (11000) according to the sixth embodiment of the sixth example of the present disclosure.

[0539] The sixth embodiment of the sixth example of the present disclosure is a modified embodiment, and for convenience of explanation, descriptions of redundant content may be omitted.

[0540] The sixth embodiment is a configuration intended to improve module-level performance by making full use of the space and power advantages provided by the E1.S form factor. As illustrated in FIG. 34, a plurality of NPU-dedicated memories (11400) may be included corresponding to each of the two NPUs (100) within the E1.S module (11000). That is, each NPU (100) has a plurality of channel memory configurations as described in the third embodiment. For example, it is possible to configure a plurality of channel memories by dividing the total of eight lanes of the PCIe interface into four for each NPU (100).

[0541] This configuration has significant effects in that it maximizes both computational performance (multiple NPUs) and memory bandwidth (dual channels per NPU) within the AI ​​module (11000). This can provide a solution suitable for high-performance AI systems that need to process large-scale AI models in a single module or process multiple AI models simultaneously with high performance.

[0542] <7th Embodiment: High-density integrated packaging based on standard layout>

[0543] The seventh embodiment of the sixth example of the present disclosure is a modified embodiment, and for convenience of explanation, descriptions of redundant content may be omitted.

[0544] The seventh embodiment aims to present a packaging method that efficiently implements a high-performance AI module through a standard component arrangement in a form factor with relatively ample space, such as E1.S. Referring to FIG. 17, a system-on-chip (SoC) that performs AI computations and a plurality of memory (MEM) for data storage can be arranged in a standard grid pattern within a package (5000). Since the E1.S form factor provides a larger mounting area compared to M.2, it is possible to arrange the SoC and up to four or more memory chips without difficulty. For example, when the size of the SoC is 4.50mm × 4.85mm and the size of the memory is 6.43mm × 5.30mm, mounting them on an E1.S substrate is spatially feasible. The advantage of this embodiment is that by utilizing the physical advantages of E1.S, the performance of the AI ​​module (especially memory capacity and bandwidth) can be easily improved by integrating multiple memories in a standard manner without complex layout technology. Various semiconductor packaging technologies, such as fan-in, fan-out, package-on-package (PoP), and 2.5D packaging methods, can be applied, which can affect the integration density and performance of the system.

[0545] <Eighth Embodiment: Space-Optimized Packaging via Rotated Multi-Chip Layout>

[0546] The eighth embodiment of the sixth example of the present disclosure is a modified embodiment, and for convenience of explanation, descriptions of redundant content may be omitted.

[0547] The eighth embodiment aims to overcome mounting limitations that occur in small form factors with extremely limited physical space, such as the M.2 2230, and to integrate high-performance AI functions. The standard layout described in the seventh embodiment may encounter physical limitations when placing multiple chips on a small substrate such as the M.2. To solve this, a placement method called a "rotated multi-chip substrate layout" as shown in FIG. 18 can be applied. Referring to FIG. 18, an SoC and a plurality of memory (MEM) can be placed diagonally within a package (5100) by rotating them at a specific angle.

[0548] The progressive effects provided by this layout optimization are clear. For example, if an SoC of the same size as in the seventh embodiment and four memory modules are rearranged in this manner, the overall package size is reduced to approximately 21 mm × 21 mm. This miniaturized package makes it possible to mount it sufficiently on an M.2 2230 circuit board with a width of 22 mm and a depth of 30 mm. Consequently, this embodiment can provide a solution for effectively implementing sophisticated and computationally intensive on-device AI functions even in devices with power budgets where the installation of AI accelerators was constrained by existing technology.

[0549] <Example 7: Extension and Integration of AI Systems for Specific Applications>

[0550] The seventh example of the present disclosure below relates to various embodiments for integrating with other functional units or expanding the system to build a system optimized for a specific application field, based on the AI ​​system (10000) of the fifth example and the AI ​​module (11000) of the sixth example. Each embodiment according to the present example may have an independent configuration in itself, and descriptions of redundant content may be omitted for convenience of explanation. Furthermore, each embodiment may be modified and implemented to suit various purposes and performance by selectively combining with or referencing the technical features of other examples.

[0551] <First Embodiment: AI System with Integrated Storage Unit>

[0552] FIG. 35 is a diagram schematically illustrating an AI system (10000) including a storage unit according to the first embodiment of the seventh example of the present disclosure.

[0553] The objective of the present embodiment is to efficiently integrate AI computing functions and large-capacity storage functions within a single system. Referring to FIG. 35, at least one storage unit (12000) having the same form factor (e.g., M.2 or E1.S) can be mounted in a plurality of slots (10600) provided on the main board (10100) of the AI ​​system (10000), along with at least one AI module (11000).

[0554] This configuration can be particularly effective for implementing a system that integrates AI analysis server and network video recorder (NVR) functions. As a result, it is possible to analyze multiple CCTV videos in real time using an AI module (11000) to perform operations such as object detection and event detection, and to immediately store the analysis results (metadata) along with the original videos in a storage unit (12000) within the same system. Compared to conventional technology where the AI ​​server and NVR server are separated, this can reduce network load caused by data transmission between systems and provide the effect of reducing system construction and operation costs.

[0555] The adoption of industry standard specifications, such as M.2 or E1.S, as the form factor of the AI ​​module (11000) in this disclosure is not merely a design choice, but can serve as a technological catalyst enabling the 'hyper-convergence' of AI, storage, and data processing functions. This is a significant synergistic effect that creates new system architectures and operational efficiencies that were impossible in existing GPU-based systems using proprietary form factors.

[0556] Traditional systems are configured with functions separated by category. For example, AI computations are handled by GPU servers, video recording and storage by NVR (Network Video Recorder) servers, and databases by storage servers, all of which are connected via a network. This increases the complexity of the system configuration and can cause network latency and bandwidth issues during data transmission. Furthermore, due to their proprietary form factors, large, high-power GPU cards find it difficult to share the same physical interfaces with storage devices (e.g., NVMe SSDs) or other peripherals.

[0557] The present disclosure can shift this paradigm by applying an M.2 or E1.S form factor to the AI ​​module (11000). This form factor is a widely used standard in modern high-speed NVMe SSDs. As the AI ​​accelerator and the storage device share the same physical and electrical interface, it becomes possible to place multiple identical slots (10600) on the main board (10100) and flexibly mount the AI ​​module (11000) in one slot and the storage unit (12000) in another slot according to the user's needs (see FIG. 35).

[0558] This configuration enables the implementation of a completely new 'integrated data processing pipeline' within a single chassis. For example, the entire process of decoding an externally input video stream by a dedicated decoder unit (13000, see FIG. 37) within the system, analyzing the results in real time by an AI module mounted in slot 1, and immediately storing the analyzed video and metadata in a storage unit mounted in slot 2 can be carried out via the high-speed bus (10150) of the main board. This minimizes network communication between servers to minimize latency and can provide a powerful synergistic effect that drastically reduces the physical space, power consumption, and management costs of the system.

[0559] < Second Embodiment: Main processor-based system including encoder / decoder >

[0560] FIG. 36 is a drawing illustrating an AI system (10000) according to the second embodiment of the seventh example of the present disclosure.

[0561] The present embodiment aims to increase the efficiency of data preprocessing and postprocessing required for AI computation, and to this end, the main processor (10200) is characterized by having built-in encoder (10210) and decoder (10220) functions in the form of a hardware accelerator. As shown in FIG. 36, the main processor (10200) can support efficient data processing while simplifying the system through these built-in functions.

[0562] For example, when the AI ​​system (10000) receives a video stream encoded with a specific codec, the main processor (10200) can rapidly decode the video frames using a built-in decoder (10220) and supply them to the AI ​​module (11000). At this time, the codecs supported by the decoder (10220) may be, for example, at least one of H.264, HEVC, JPEG, YUV, AV1, MP3, AAC-LC, HE-AAC, etc. Such a configuration allows the system to be built without a separate decoding device, thereby providing a cost-effective solution for systems with a small number of AI modules (11000) or a small number of video channels to process.

[0563] < Third Embodiment: System including a dedicated decoder >

[0564] FIG. 37 is a drawing illustrating an AI system (10000) according to the third embodiment of the seventh example of the present disclosure.

[0565] This embodiment is intended for a high-performance configuration for cases where a decoding load exceeding the processing capacity of the main processor (10200) occurs, such as when multiple high-resolution video channels must be processed simultaneously. As illustrated in FIG. 37, the AI ​​system (10000) may further include a separate decoder (13000) chip on the main board (10100).

[0566] This dedicated decoder chip (13000) is specialized for hardware processing of various video codecs such as H.264, HEVC, and AV1. As a result, it can efficiently decode multiple high-resolution video streams, thereby efficiently allocating system resources so that the AI ​​module (11000) can focus only on core AI operations. Additionally, the number of decoder chips (13000) can be expanded in correspondence with the number of AI modules (11000) installed in the AI ​​system (10000) and the total processing capacity, thus providing th...

Claims

Claim 1 An intelligent control server system comprises: a plurality of slots provided on a main board; at least one AI module detachably mounted in at least one of the plurality of slots and performing AI (artificial intelligence) operations; and at least one main processor controlling the main board, wherein each of the at least one AI module comprises a printed circuit board; and at least one NPU (neural network processing unit) performing the AI ​​operations, wherein each of the at least one NPU comprises a processing core for integer-based matrix operations and a special function unit (SFU) for floating-point-based non-linear function operations, wherein the special function unit (SFU) comprises a programmable function approximation (PAF) circuit that approximates a plurality of different non-linear functions without hardware modification by loading different programmable parameter sets learned offline in advance, and wherein the overall AI operation performance of the intelligent control server system is configured to be expandable based on the number of at least one AI modules mounted in the plurality of slots. Claim 2 delete Claim 3 An intelligent control server system according to claim 1, characterized in that the intelligent control server system further includes a dedicated decoder configured to decode encoded video or voice data and provide it to each of the AI ​​modules. Claim 4 The intelligent control server system according to claim 1, wherein the intelligent control server system further includes a GPU configured to perform general-purpose parallel processing operations or 3D graphics rendering, and the main processor selectively assigns tasks to the at least one AI module or the GPU according to the characteristics of the operations. Claim 5 An intelligent control server system according to claim 1, characterized in that at least one other slot among the plurality of slots where the AI ​​module is not installed is equipped with a storage unit for permanently storing data. Claim 6 An intelligent control server system according to claim 1, wherein the intelligent control server system has a hierarchical memory structure including a main memory located outside of at least one AI module and storing large-capacity data, and an on-chip shared memory integrated inside a system-on-chip (SoC) containing the main processor and temporarily storing frequently used data at high speed. Claim 7 An intelligent control server system according to claim 1, wherein the main processor controls the NPU included in the AI ​​module to perform speculative decoding using a first language model having a first number of parameters and a second language model having a second number of parameters. Claim 8 An intelligent control server system according to claim 1, wherein each of the at least one AI module further comprises a connector formed on one side of the printed circuit board and electrically connected to at least one of the plurality of slots. Claim 9 An intelligent control server system according to claim 8, wherein each of the above-mentioned at least one AI module is mounted on the above-mentioned printed circuit board and further includes an NPU-dedicated memory that the NPU uses independently during AI computation. Claim 10 An intelligent control server system according to claim 9, characterized in that the NPU dedicated memory is composed of a plurality of units to provide dual-channel memory bandwidth to the NPU. Claim 11 An intelligent control server system according to claim 8, wherein each of the above-mentioned at least one NPU has an independent NPU-dedicated memory. Claim 12 delete Claim 13 delete Claim 14 An intelligent control server system according to claim 8, wherein the printed circuit board is formed according to the standard specifications of M.2 or E1.S. Claim 15 An intelligent control server system according to claim 1, wherein each of the at least one AI module further comprises a controller that controls the operation of the processing core and the special function unit based on compiled executable code. Claim 16 An intelligent control server system according to claim 15, wherein the programmable function approximation (PAF) circuit approximates various activation functions using a plurality of preset programmable parameters. Claim 17 An intelligent control server system according to claim 16, wherein the programmable function approximation (PAF) circuit comprises: a plurality of comparators for determining the interval to which an input value belongs; a selector for selecting a programmable parameter corresponding to the determined interval; and an operation unit for performing a polynomial operation using the selected programmable parameter. Claim 18 An intelligent control server system according to claim 15, wherein the controller controls the processing core and the special function unit to perform speculative decoding by processing a first language model having a first number of parameters and a second language model having a second number of parameters in a time-division manner. Claim 19 An intelligent control server system according to claim 18, characterized in that the weights of the first language model and the second language model reside together in the memory included in the intelligent control server system. Claim 20 An intelligent control server system according to claim 15, wherein the controller determines the validity of operations processed by the processing core or the special function unit and controls the operation of the unit through clock gating or power gating for unnecessary operations. Claim 21 delete

Images