System for processing heterogeneous generative artificial intelligence model

Abstract

According to the present disclosure, a device is provided. The device may include: a first memory of a first capacity configured to store a first generative neural network model including a first number of parameters; and a first neural processing unit configured to generate a response corresponding to an input query by utilizing the first generative neural network model stored in the first memory, wherein the first neural processing unit is configured to store a first executable code of the first generative neural network model compiled to process speculative decoding.

Description

A system for processing heterogeneous generative artificial intelligence models

[0001] The present disclosure relates to a system for processing heterogeneous generative artificial intelligence models. More specifically, it relates to a federated processing system for processing heterogeneous language models.

[0002] Humans possess the intelligence to recognize, classify, infer, predict, and control / decision-making. Artificial Intelligence (AI) refers to the artificial imitation of human intelligence.

[0003] The human brain is composed of countless nerve cells called neurons. Each neuron is connected to hundreds or thousands of other neurons through connections called synapses. To mimic human intelligence, a neural network (NN) model models the operating principles of biological neurons and the connections between them. In other words, a neural network model is a system that connects nodes that mimic neurons in layers.

[0004] As neural network technology advances, the application of neural network inference services leveraging big data-based training is becoming increasingly diverse. These inference services train neural networks on massive datasets, enabling them to infer diverse and complex data through the learned models. However, as the functionality and accuracy required for neural network inference services continue to increase, the parameter size, computational complexity, and memory bandwidth demands of neural network models are growing exponentially. Consequently, the performance requirements for processors and memory capable of supporting these inference tasks are also increasing. To address these demands, neural network inference services are increasingly being deployed on cloud-based servers that provide efficient processing capabilities for large-scale data processing.

[0005] Generative AI models are increasingly being developed to operate in a variety of environments. These models include various neural network models, including language models for conversational systems, stable diffusion models that generate images from text input, and transformer models that predict and determine future actions based on a series of events in specific situations. The computational cost of responding to input queries with these generative AI models can be very expensive. For example, commercialization of Large Language Models (LLMs) among generative AI models is beginning. Currently, only high-performance general-purpose graphics processing units (GPGPUs) can sufficiently provide the computational power required for LLM processing. However, the total power consumption of GPGPUs operating worldwide to process LLMs has reached a level comparable to that of entire countries, and this excessive energy demand poses a serious challenge to the commercialization of LLMs. Building GPGPU-powered data centers to deliver LLM services requires careful assessment of power plant capacity to determine whether the required energy can be supplied, and the difficulty in meeting the energy demands of these data centers in many urban areas presents a significant barrier to widespread deployment of large-scale AI models.

[0006] Neural networks are classified into 'single-layer neural networks' and 'multi-layer neural networks' depending on the number of layers. A typical multi-layer neural network consists of an input layer, a hidden layer, and an output layer. (1) The input layer is a layer that receives input values. The number of input layers is equal to the number of input variables. (2) The hidden layer is located between the input layer and the output layer. It receives signals from the input layer, extracts features, and transmits them to the output layer. (3) The output layer is a layer that receives signals from the hidden layer and outputs them to the outside.

[0007] To achieve higher-level artificial intelligence, deep neural networks (DNNs) with increased numbers of hidden layers have been developed. Among DNNs, the transformer neural network is a DNN based on attention technology. The transformer utilizes numerous matrix multiplication operations. The transformer can obtain attention (Q, K, V), which is an operation value, using parameters such as input values ​​and queries (Q), keys (K), and values ​​(V). The transformer can process various inference operations based on the operation values ​​(e.g., attention (Q, K, V)). The transformer has demonstrated excellent performance in the field of generative artificial intelligence.

[0008] Transformer-based neural network models are used in language models. For example, a language model receives a text-based query and generates a response. The response generated by the language model can be generated by re-inputting tokens generated as part of the response into the language model. The cost of generating a response using such a language model is proportional to the number of parameters in the language model. Therefore, the computational cost of generating a response using a language model with a large number of parameters is high. Conversely, reducing the number of parameters can lower computational costs and improve response speed, but this can also lead to a deterioration in the quality of the generated response.

[0009] However, the tasks of the present disclosure are not limited to the tasks mentioned above, and other tasks not mentioned will be clearly understood by those skilled in the art from the description below.

[0010] According to examples of the present disclosure, a device may be provided. The device may include a first memory having a first capacity configured to store a first generative neural network model comprising first parameters; and a first neural processing unit configured to generate a response corresponding to an input query by utilizing the first generative neural network model stored in the first memory. The first neural processing unit may be configured to store a first executable code of the first generative neural network model compiled to process speculative decoding.

[0011] According to some examples of the present disclosure, the first memory of the first capacity may be configured to store a second generative neural network model comprising second parameters. The number of the second parameters may be greater than the number of the first parameters.

[0012] According to some examples of the present disclosure, the first memory of the first capacity may have a capacity capable of storing both the first generative neural network model and the second generative neural network model.

[0013] According to some examples of the present disclosure, the first neural processing unit may be configured to store a second executable code of the second generative neural network model compiled to process the speculative decoding.

[0014] According to some examples of the present disclosure, the first neural processing unit may further include a first NPU memory configured to communicate with the first memory and a first NPU control unit configured to control the first neural processing unit.

[0015] According to some examples of the present disclosure, the first executable code may be configured to be stored in the first NPU memory or the first NPU control unit.

[0016] According to some examples of the present disclosure, the second memory may further include a second capacity configured to store a second generative neural network model comprising second parameters.

[0017] According to some examples of the present disclosure, a second memory having a second capacity configured to store a second generative neural network model comprising second parameters; and a second neural processing unit configured to generate a response corresponding to an input query by utilizing the second generative neural network model stored in the second memory. The second neural processing unit may be configured to store a second executable code of the second generative neural network model compiled to process the speculative decoding.

[0018] According to some examples of the present disclosure, the computational performance of the second neural processing unit may be higher than the computational performance of the first neural processing unit.

[0019] According to some examples of the present disclosure, at least one of the first neural processing unit and the second neural processing unit may be configured to operate in a low power mode when waiting for an operation result of another neural processing unit during the speculative decoding.

[0020] According to examples of the present disclosure, a system may be provided. The system may include an on-premises server, including a device comprising: a first memory having a first capacity configured to store a first generative neural network model comprising first parameters; and a first neural processing unit configured to generate a response corresponding to an input query by utilizing the first generative neural network model stored in the first memory; and a second memory having a second capacity configured to store a second generative neural network model comprising second parameters; and a second neural processing unit configured to generate a response corresponding to the input query by utilizing the second generative neural network model stored in the second memory. The first neural processing unit and the second neural processing unit may be configured to cooperate with each other to process speculative decoding.

[0021] According to some examples of the present disclosure, the first memory of the first capacity may be capable of storing the first generative neural network model, and the second memory of the second capacity may be capable of storing the second generative neural network model.

[0022] According to some examples of the present disclosure, the first neural processing unit may be configured to store a first executable code of the first generative neural network model compiled to process the speculative decoding, and the second neural processing unit may be configured to store a second executable code of the second generative neural network model compiled to process the speculative decoding.

[0023] According to some examples of the present disclosure, the computational performance of the second neural processing unit of the on-premise server may be higher than the computational performance of the first neural processing unit of the device.

[0024] According to some examples of the present disclosure, the devices may be multiple, and each device may be configured to process the speculative decoding via the on-premises server.

[0025] According to some examples of the present disclosure, at least one of the plurality of devices may be configured to operate in at least one of a sleep mode, a deep sleep mode, or a retention mode.

[0026] According to some examples of the present disclosure, the first generative neural network model may be a lower-level model, and the second generative neural network model may be a higher-level model.

[0027] According to some examples of the present disclosure, the upper model can be configured to process the speculative decoding while being optionally disabled.

[0028] According to some examples of the present disclosure, the speculative decoding may include a prefill operation, a generate operation, and an accept or reject operation.

[0029] According to examples of the present disclosure, a device may be provided. The device may include a first memory configured to store a first generative neural network model; and a first neural processing unit configured to generate a response corresponding to an input query by utilizing the first generative neural network model stored in the first memory. The first neural processing unit may include a processing core configured to receive an integer parameter input, a vector core and a scalar core configured to receive a floating-point parameter input, and a number system conversion unit configured to convert the integer parameter and the floating-point parameter to process an operation of the first generative neural network model.

[0030] Specific details of other examples are included in the detailed description and drawings.

[0031] According to the examples presented in this disclosure, utilizing a federated driving system of heterogeneous language models can significantly reduce energy consumption, carbon emissions, and energy costs compared to relying solely on data centers. Therefore, various generative AI services can be readily provided without significantly increasing current power plant capacity.

[0032] According to the examples of the present disclosure, although the inference performance is relatively lower than that of a server, the production cost of the on-device can be reduced because the generative artificial intelligence service can be processed with a neural processing unit that can be driven with low power.

[0033] According to the examples of the present disclosure, a generative artificial intelligence service can be provided in real time in an on-device environment.

[0034] According to the examples of the present disclosure, since AI services can be provided on-device, the user's personal information and privacy can be protected.

[0035] According to examples of the present disclosure, power consumption of the on-device can be reduced by implementing speculative decoding while providing at least one low-power mode in an on-device environment.

[0036] According to the examples of the present disclosure, speculative decoding can be implemented with heterogeneous language models in an on-device environment, thereby maintaining the inference accuracy level of the upper model while utilizing the token generation speed of the lower model.

[0037] According to the examples of the present disclosure, by selectively changing the on-device environment and the server environment, it is possible to implement guess decoding with heterogeneous language models in a fast token generation speed or high response accuracy, thereby maintaining the inference accuracy level of the upper model while utilizing the token generation speed of the lower model.

[0038] According to examples of the present disclosure, by providing at least one neural processing unit and at least one memory capable of implementing speculative decoding with heterogeneous language models in an on-device environment, the speculative decoding can be processed sequentially or in parallel to improve token generation speed on-device.

[0039] The effects according to the present disclosure are not limited to the contents exemplified above, and more diverse effects are included in the present disclosure.

[0040] Figure 1 illustrates tree data utilized for speculative decoding according to examples of the present disclosure.

[0041] Figure 2 schematically illustrates a speculative decoding operation according to a first example of the present disclosure.

[0042] FIG. 3 illustrates a device according to a first embodiment of the first example of the present disclosure.

[0043] FIG. 4 illustrates a device according to a second embodiment of the first example of the present disclosure.

[0044] FIG. 5 illustrates a device according to a third embodiment of the first example of the present disclosure.

[0045] FIG. 6 illustrates a system according to a first embodiment of the second example of the present disclosure.

[0046] FIG. 7 illustrates a system according to a second embodiment of the second example of the present disclosure.

[0047] Figure 8 schematically illustrates a speculative decoding operation according to a second example of the present disclosure.

[0048] FIG. 9 illustrates a system according to a third example of the present disclosure.

[0049] FIG. 10 illustrates a neural processing unit that can be applied to examples of the present disclosure.

[0050] FIG. 11 illustrates processing elements that may be applied to examples of the present disclosure.

[0051] FIG. 12 is a table schematically illustrating the energy consumption per unit operation of a neural processing unit according to examples of the present disclosure.

[0052] Specific structural or step-by-step descriptions of examples according to the concepts of the present disclosure disclosed in this disclosure are provided solely for the purpose of explaining examples according to the concepts of the present disclosure.

[0053] Embodiments according to the concepts of the present disclosure may be implemented in various forms. The present disclosure should not be construed as being limited to the embodiments described herein.

[0054] Embodiments based on the concepts of this disclosure can undergo various modifications. The present disclosure can take various forms. Therefore, specific examples are illustrated in the drawings and described in detail herein. However, this is not intended to limit the examples based on the concepts of this disclosure to a specific form. Therefore, it should be understood that all modifications, equivalents, or alternatives falling within the spirit and technical scope of this disclosure are encompassed by this disclosure.

[0055] The terms first and / or second may be used to describe various components. However, the present disclosure should not be limited by these terms.

[0056] The above terms are used solely to distinguish one component from another. For example, without departing from the scope of the invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component.

[0057] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.

[0058] Other expressions that describe the relationship between components, such as "between" and "directly between" or "adjacent to" and "directly adjacent to", should be interpreted similarly.

[0059] In this document, the expressions "A or B," "at least one of A and / or B," or "one or more of A or / and B" can include all possible combinations of the listed items. For example, "A or B," "at least one of A and B," or "at least one of A or B" can all refer to cases where (1) at least one A is included, (2) at least one B is included, or (3) at least one A and at least one B are included.

[0060] As used herein, the terms "first," "second," "first," or "second," etc., may describe various components, regardless of order and / or importance. These terms are only used to distinguish one component from another and do not limit the components. For example, a first user device and a second user device may represent different user devices, regardless of order or importance. For example, without departing from the scope of the rights set forth in this document, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component.

[0061] The terms used in this disclosure are used only to describe specific embodiments and may not be intended to limit the scope of other examples.

[0062] Singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art described in this document.

[0063] Terms used in this disclosure, if defined in a general dictionary, may be interpreted as having the same or similar meaning as they have in the context of the relevant technology. Unless explicitly defined herein, terms used in this disclosure shall not be interpreted in an idealized or overly formal sense. In some cases, even if a term is defined herein, it cannot be interpreted to exclude embodiments of this document.

[0064] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of this disclosure.

[0065] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as "comprises" or "has" are intended to indicate the presence of a described feature, number, step, operation, component, part, or combination thereof. Therefore, it should be understood that they do not preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0066] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art to which this disclosure pertains. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology. Unless explicitly defined herein, terms shall not be construed in an idealized or overly formal sense.

[0067] The individual features of the various examples of this disclosure can be partially or fully combined or combined with each other. As those skilled in the art will readily appreciate, the various examples of this disclosure are technically capable of various interconnections and operations. The examples of this disclosure may be implemented independently of each other or may be implemented together in a related manner.

[0068] In explaining each example, descriptions of technical details that are well-known in the technical field to which the present disclosure pertains and are not directly related to the present disclosure will be omitted. This is to avoid obscuring the gist of the present disclosure by omitting unnecessary explanations and to convey the gist more clearly.

[0069] According to examples of the present disclosure, a device, method, system, and computer-readable medium that enable a generative artificial intelligence service to operate on-device can be provided.

[0070] Typically, a Transformer-based generative AI model receives a query as input and generates a response. For example, a language model receives a text-based query and generates tokens. The query and the generated tokens are then fed back into the language model to generate the next token. This token generation operation can be repeated multiple times until the query response is complete. Tokens may correspond to words or parts of words. The response generated by the language model may include a series of tokens generated by passing the tokens generated as part of the response back through the language model.

[0071] The cost of generating responses from these language models is proportional to the number of parameters in the model. Therefore, the computational cost of generating responses using a language model with a large number of parameters is high. However, while reducing the number of parameters can lower computational costs and improve response speed, it can also lead to decreased response accuracy. In other words, there is a trade-off between the number of parameters in the language model and the time it takes the processor to process the language model's token generation operations.

[0072] In particular, generating responses of a certain quality requires a significant number of language model parameters. Meanwhile, processing these language models can present challenges in providing services at a level suitable for commercialization due to limitations in on-device hardware resources (e.g., energy consumption, battery capacity, memory capacity, AI processing capabilities, etc.).

[0073] Accordingly, the present disclosure presents a federated driving system for language models utilizing speculative decoding (or speculation decoding) to drive heterogeneous language models by utilizing limited hardware resources of an on-device through various examples.

[0074] According to examples of the present disclosure, a federated driving system for heterogeneous language models may include at least one neural processing unit and at least one memory, and may be configured to drive heterogeneous language models with at least one neural processing unit.

[0075] On-device, to accelerate token generation speed of a language model, speculative decoding is configured to utilize a heterogeneous language model. The heterogeneous language model according to examples of the present disclosure may include at least a first language model and a second language model. In some examples, the heterogeneous language model may include the first to third language models. In some examples, the heterogeneous language model may include the first to fourth language models.

[0076] Here, the number of parameters of the first language model is smaller than that of the second language model. The number of parameters of the second language model is smaller than that of the third language model. The number of parameters of the third language model is smaller than that of the fourth language model. In other words, the numbers of parameters of different language models are different. A language model with a relatively smaller number of parameters can be defined as a lower-order model, and a language model with a relatively larger number of parameters can be defined as a higher-order model. In other words, the first language model can be defined as a lower-order model of the second language model, and the second language model can be defined as a higher-order model of the first language model. The second language model can be defined as a lower-order model of the third language model, and the third language model can be defined as a higher-order model of the second language model. The first language model can be defined as a lower-order model of the second language model, and the second language model can be defined as a higher-order model of the first language model.

[0077] The lower-level model can utilize a language model whose number of parameters is determined by considering the hardware resource limitations of the on-device being processed. The upper-level model can utilize a language model whose number of parameters is determined by considering the hardware resource limitations of the processing device. For example, the number of parameters in the upper-level model can be 7 billion, and the number of parameters in the lower-level model can be 1 billion. For example, the number of parameters in the upper-level model can be 32 billion, the number of parameters in the middle-level model can be 7 billion, and the number of parameters in the lower-level model can be 1 billion.

[0078] The following describes the speculative decoding utilized in the examples of the present disclosure.

[0079] Systems according to the examples of the present disclosure are configured such that lower-level models and higher-level models collaborate to process guess decoding. Here, guess decoding utilizes at least one neural processing unit to generate a response to an input query through the collaboration of heterogeneous language models.

[0080] Using the neural processing unit of the on-device, the sub-model can generate probabilities used to sample the additional tokens based on the additional tokens generated speculatively and the currently accepted token set. Using the neural processing unit of the on-device, the super-model generates tokens based on the tokens generated by the sub-model. The super-model can sample the additional tokens speculatively generated by the sub-model and accept or reject at least one token. Here, the acceptance or rejection of the super-model can be determined by comparing the probability generated by the super-model corresponding to each token with the probability generated by the sub-model corresponding to each token.

[0081] According to the speculative decoding examples of the present disclosure, a lower-level model can generate one or more sets of token candidates to generate a response corresponding to a query input to the lower-level model. Accordingly, the upper-level model can sample from each set of token candidates and accept or reject at least one set of token candidates. When the system samples from each set of token candidates, the system can generate more tokens more quickly than when sampling from each token. Furthermore, the probability distributions of the lower-level model and the upper-level model can be maintained similarly. The token generation speed here may be referred to as TPS (tokens per second).

[0082] According to the speculative decoding of the system according to the examples of the present disclosure, the system can process the lower-level model and the upper-level model in parallel. When speculative decoding is processed in parallel, the lower-level model generates one or more tokens, and the upper-level model samples tokens previously generated by the lower-level model and accepts or rejects them.

[0083] The speculative decoding according to the examples of the present disclosure can operate in an automatic regression token generation manner. The automatic regression token generation manner generates tokens by inputting a series of tokens generated by a lower-level model back into a lower-level model. The lower-level model generates one token for each automatic regression. Therefore, when the lower-level model performs inference N times, N series of tokens are generated. The lower-level model generates a conditional probability distribution related to the N series of tokens. Subsequently, the upper-level model processes the N series of tokens generated by the lower-level model and the conditional probability distribution related to the N series of tokens to generate a probability distribution for the upper-level model. Therefore, the upper-level model can compare the probability distribution generated by the upper-level model with the conditional probability distribution generated by the lower-level model to accept or reject the token generated by the lower-level model. The upper-level model's acceptance of a token can be determined by setting a threshold. The threshold may be a value that accepts a token when the degree of similarity between the probability distribution value of the upper-level model and the conditional probability distribution value of the lower-level model is equal to or greater than the threshold, and rejects the token when the degree of similarity is less than the threshold.

[0084] The upper model can accept or reject a series of tokens generated by the lower model based on a threshold. If a particular token in the series is rejected, the tokens prior to rejection become the final tokens.

[0085] In some examples, the submodel of the speculative decoding system can generate tokens in sets. In such cases, the supermodel can accept or reject multiple token sets generated by the submodel. When the submodel generates tokens in sets, the supermodel's acceptance or rejection decisions can also be made on a per-token-set basis, which can improve the token generation speed of the system's speculative decoding system. Furthermore, as the number of token sets generated by the submodel increases, the probability of the token set being accepted by the supermodel increases.

[0086] The submodel generates one or more token sets in response to an input query. Each token set includes a series of tokens, and the submodel selects tokens with high probabilities within a probability distribution as the token set. Each token set can be selected in various ways. For example, each token set can be generated by selecting tokens with the highest probability within a probability distribution. However, the present disclosure is not limited thereto, and token sets can be generated in various ways. For the supermodel to accept a token set, the probabilities of the tokens in each token set can be summed. Therefore, the supermodel can accept or reject the token set based on the summed probability of the token set, rather than the probabilities of each token. To elaborate, utilizing the individual probabilities of tokens increases the probability of rejection by the supermodel, as the matching of the probabilities of each token generated by the submodel and the supermodel can be difficult. Utilizing the summed probability of the token set increases the probability of acceptance by the supermodel, as the matching of the summed probabilities of the token sets generated by the submodel and the supermodel can be more similar.

[0087] The submodel can generate one or more token sets in response to an input query. These token sets can be generated statically or dynamically. For example, the sequence length of each token set can be preset. The submodel can generate one or more token sets and combine them to generate tree data. The tree data contains nodes at each branch point.

[0088] Figure 1 illustrates tree data utilized for speculative decoding according to examples of the present disclosure.

[0089] Referring to FIG. 1, an example of tree data (TD) including one or more token sets generated from a sub-model is illustrated. Specifically, a first token set (S1), a second token set (S2), a third token set (S3), a fourth token set (S4), and a fifth token set (S5) are illustrated.

[0090] A first token set (S1) includes four tokens (A, B, C, D). A second token set (S2) includes four tokens (A, B, C, E). A third token set (S3) includes four tokens (A, F, G, H). A fourth token set (S4) includes four tokens (A, F, G, I). A fifth token set (S5) includes four tokens (A, F, J, K). Each token set is a candidate that can be selected or rejected by the upper model.

[0091] Each token set may have a set sentence length. For example, the sentence length of each token set may be set to 4. However, the present disclosure is not limited thereto, and the length of the token set may also be dynamically changed.

[0092] Each token can have an order level. The order level can correspond to the sentence length. Specifically, the first order level (L1) corresponds to token (A). The second order level (L2) corresponds to tokens (B, F). The third order level (L3) corresponds to tokens (C, G, J). The fourth order level (L4) corresponds to tokens (D, E, H, I, K). In the example where the sentence length is set to 4, the order level is also set to 4. The order of tokens in each token set can be arranged according to the corresponding order level. Specifically, the first token (A) among the tokens (A, B, C, D) of the first token set (S1) corresponds to the first order level (L1). The second token (B) among the tokens (A, B, C, D) of the first token set (S1) corresponds to the second order level (L2). Among the tokens (A, B, C, D) of the first token set (S1), the third token (C) corresponds to the third order level (L3). Among the tokens (A, B, C, D) of the first token set (S1), the fourth token (D) corresponds to the fourth order level (L4).

[0093] Each token is defined as a node in the tree data (TD). For example, a token (G) can be a branch between the third token set (S3) and the fourth token set (S4) at the third order level (L3). Each node contains branch information.

[0094] That is, the tree data (TD) generated by the lower model may include all or at least a part of one or more token sets (S1 to S5), order levels (L1 to L4), node information (A, B, C, D, E, F, G, H, I, J, K) of each token, and selection probability of each token set (e.g., the summed probability of the token sets). The upper model may accept or reject at least one token of the tree data (TD).

[0095] Speculative decoding can also be implemented using hybrid speculative decoding. In hybrid speculative decoding, previously generated tokens are verified by a higher-order model, and the probability distributions of the lower-order and higher-order models are compared to set a threshold α to determine whether the token is accepted or rejected.

[0096] Speculative decoding can also be implemented using group speculative decoding. Group speculative decoding generates multiple tokens in groups from a lower-order model and verifies them against a higher-order model, providing a high level of efficiency.

[0097] Figure 2 schematically illustrates a speculative decoding operation according to examples of the present disclosure.

[0098] Referring to FIG. 2, a lower model (LM) and an upper model (UM) are illustrated. The device according to the examples of the present disclosure includes one or more neural processing units. The one or more neural processing units may be configured to sequentially or in parallel process the lower model (LM) and the upper model (UM) for speculative decoding.

[0099] First, a typical transformer-based neural network model includes computational steps (e.g., a prefill step and a generation step) to generate a response to an input query. A transformer neural network is a neural network based on attention technology. A transformer utilizes a number of matrix multiplication operations. A transformer can obtain an output value, attention (Q, K, V), using input values ​​and parameters such as a query (Q), a key (K), and a value (V). A transformer can process various inference operations based on the output values ​​(i.e., attention (Q, K, V)). Transformers are actively used in language generation models. A tensor refers to a multidimensional matrix parameter processed in a neural network model. A tensor can refer to various parameters of a neural network model, such as weights and attention parameters.

[0100] The prefill stage is the initial step in processing input (e.g., a prompt). The prefill stage includes tokenization, context embedding, layer-by-layer processing, and preparation for decoding. The prefill stage can be processed by one or more neural processing units. In the tokenization stage, the input text is converted into tokens (subwords, words, or characters), which are numerical representations used by the neural network model. In the context embedding stage, the tokens are mapped to dense vector embeddings that encode semantic and syntactic information. In the layer-by-layer processing stage, the embeddings are processed through multiple layers of the neural network model (e.g., transformer blocks) to create contextual representations of the input tokens. In the preparation for decoding stage, the neural network model calculates the probability of the next token based on the input context, but does not yet generate the next token. Instead, it sets the context for the generation stage.

[0101] In the generation phase, a neural network model sequentially generates one or more tokens based on the context provided in the prefill phase and previously generated tokens. The generation phase includes an initial prediction phase, an autoregressive processing phase, and a stopping condition. The generation phase may be processed by one or more neural processing units. In the initial prediction phase, the first token is generated by sampling from the probability distribution calculated in the prefill phase. In the autoregressive processing phase, each subsequent token is generated by integrating the previous token (including the generated token) into the context of the neural network model. The generation process continues until a stopping condition is met, such as reaching a maximum token limit, encountering a special sequence-ending token (e.g., end of sequence (EOS)), or achieving a specific semantic goal. The series of tokens generated in the generation phase may correspond to the tree data (TD) of FIG. 1. The tree data may include at least one token set.

[0102] Referring again to Figure 2, an example of processing speculative decoding is illustrated. Speculative decoding according to the examples of the present disclosure is configured to be processed sequentially or in parallel by one or more neural processing units. Speculative decoding involves a superordinate model (UM) and a subordinate model (LM) collaborating to generate a response to an input query.

[0103] Specifically, in speculative decoding, the upper model (UM) first performs a prefill operation. Then, the lower model (LM) performs a prefill operation. Then, the lower model (LM) sequentially performs generation steps, and each generation step generates a token. Next, the upper model (UM) accepts or rejects the tokens generated by the lower model (LM). The upper model (UM) can accept or reject some or all of the tokens generated by the lower model (LM). The lower model (LM) then uses the accepted tokens as input queries. Next, the lower model (LM) sequentially performs generation steps, and each token corresponding to each generation step is generated. Next, the upper model (UM) accepts or rejects the tokens generated by the lower model (LM). The above steps are repeated until the stopping conditions are met.

[0104] Referring to Figure 2, the number of parameters of the upper model (UM) is greater than that of the lower model (LM). In other words, the number of parameters of the heterogeneous language models is different. Therefore, the processing time of the prefill stage of the upper model (UM) is longer than that of the prefill stage of the lower model (LM). The processing time of the accept or reject stage of the upper model (UM) is longer than that of the generate stage of the lower model (LM). Therefore, the more tokens the lower model generates, the faster the token generation speed of the system can be.

[0105] The lower-level model can utilize a language model whose parameter count is determined by considering the hardware resource limitations of the on-device being processed. The upper-level model can utilize a language model whose parameter count is determined by considering the hardware resource limitations of the processing device.

[0106] In some examples, the submodel may be a model with pruned parameters from the parent model. For example, the submodel may be a model with pruned parameters from the parent model. In such cases, the parameters of the pruned neural network model can be made lighter, and the memory bandwidth required may be reduced compared to the parent model.

[0107] One or more neural processing units according to the examples of the present disclosure are configured to select a superordinate model (UM) to process a prefill operation for speculative decoding. Furthermore, one or more neural processing units according to the examples of the present disclosure are configured to select a subordinate model (LM) to process a prefill operation and a generation operation for speculative decoding.

[0108] FIG. 3 illustrates a device according to a first embodiment of the first example of the present disclosure.

[0109] Referring to FIG. 3, a device (100) according to a first embodiment of the first example of the present disclosure includes a neural processing unit (NPU) and a memory (MEM). The device (100) may further include at least one of a communication bus (not shown), a CPU (not shown), a GPU (not shown), and a storage device (not shown).

[0110] A device (100) according to a first embodiment of the first example of the present disclosure is configured to provide a service of the level of ultra-large-scale AI on-device with low power by utilizing speculative decoding.

[0111] According to the device (100) according to the first embodiment of the first example of the present disclosure, the user's personal information and privacy can be protected because the AI ​​service can be provided only on-device.

[0112] A device (100) according to a first embodiment of the first example of the present disclosure is configured to receive an input query through an interface (IF). The interface (IF) may be an input device configured to transmit an input query such as a prompt, gesture, or voice. Input devices capable of transmitting an input query may include, for example, a mouse, a keyboard, a touchpad, a touch screen, a microphone, a wireless communication device, a wired communication device, and the like.

[0113] A communication bus (B) is provided between the neural processing unit (NPU) and the memory (MEM). For example, the communication bus (B) may include a high-speed data bus such as the Advanced eXtensible Interface (AXI). The communication bus (B) may be a dedicated communication bus for the neural processing unit (NPU) and the memory (MEM), and may be configured to provide stable memory bandwidth for speculative decoding. The memory bandwidth may be determined according to the processing performance of the neural processing unit (NPU) (e.g., Tera Operations Per Second (TOPS) of the neural processing unit). For example, a memory (MEM) bandwidth of 2 GB / s to 4 GB / s may be required per 1 TOPS. For example, an NPU of 10 TOPS may be configured to provide a memory (MEM) bandwidth of 20 GB / s to 40 GB / s. The memory bandwidth may be determined according to the operating frequency of the memory (MEM) and the number of communication channels. For example, to configure a memory bandwidth of 40 GB / s with LPDDR5, the following formula can be used: Memory bandwidth (GB / s) = (Data rate (Gbps per pin) × Bus width (bits) × Number of channels ÷ 8). To elaborate, LPDDR5 typically supports a data rate of up to 6400 Mbps per pin. To elaborate, the bus width of an LPDDR channel is typically 16 bits (2 bytes). However, the number of channels cannot be a decimal point but an integer. However, the examples of the present disclosure are not limited to the bandwidth of the memory bus.

[0114] The neural processing unit (NPU) of the device (100) according to the first embodiment of the first example of the present disclosure is an Application Specific Integrated Circuit (ASIC) implemented as a neural network inference dedicated accelerator. The neural processing unit (NPU) may be designed to have, for example, a computational performance of 5 TOPS, 10 TOPS, 25 TOPS, 50 TOPS, or the like. The neural processing unit (NPU) may be designed to have, for example, a computational performance of 5 TOPS to 50 TOPS. The computational performance of the neural processing unit (NPU) may vary depending on the number of processing elements included in the processing core. The neural processing unit (NPU) may be referred to as a tensor processing unit (TPU), a neural signal processor (NSP), a neural network processor (NNP), or the like.

[0115] The neural processing unit (NPU) of the device (100) according to the first embodiment of the first example of the present disclosure may be configured as a system-on-chip (SoC). The system-on-chip may refer to a semiconductor package that integrates at least one neural processing unit (NPU) and components of various electronic systems. The system-on-chip may integrate digital circuits, analog circuits, mixed-signal and radio frequency processing circuits into a single package. For example, a processor that may be further included in the system-on-chip may be at least one of a central processing unit (CPU), a digital signal processor (DSP), an image signal processor (ISP), and a graphics processing unit (GPU). The system-on-chip may include at least one memory. The system-on-chip may include a high-speed data bus for efficient communication between various circuits included in the system-on-chip. The system-on-chip may include at least one interface, such as PCIe, USB, I2C, SPI, UART, and GPIO, for connection with external devices and sensors. The system-on-chip may include an on-chip power management device that regulates voltage and power distribution of the semiconductor package. A system-on-chip may include communication interfaces that integrate wired and wireless communication protocols such as Ethernet, Wi-Fi, Bluetooth, and cellular connectivity for data transfer.

[0116] The neural processing unit (NPU) of the device (100) according to the first embodiment of the first example of the present disclosure is configured to infer an upper model (UM) and a lower model (LM). The upper model (UM) and the lower model (LM) are configured to cooperate with each other to process guess decoding. For a description of guess decoding, refer to FIGS. 1 and 2 .

[0117] Here, the neural processing unit (NPU) is set to process the upper model (UM) and the lower model (LM) alternately. That is, the neural processing unit (NPU) operates to process the upper model (UM) and the lower model (LM) sequentially. In other words, the neural processing unit (NPU) operates to process the upper model (UM) and the lower model (LM) in a time-sharing manner. Therefore, the upper model (UM) and the lower model (LM) can be processed sequentially while increasing the utilization rate of the computational circuits of the neural processing unit (NPU). In addition, since the operation method of the speculative decoding is such that the upper model (UM) and the lower model (LM) can be processed sequentially in the neural processing unit (NPU) of the device (100), and the parameters of the upper model (UM) and the lower model (LM) reside in the memory (MEM), the bottleneck phenomenon caused by the speculative decoding operation in the neural processing unit (NPU) of the device (100) can be reduced.

[0118] The memory (MEM) of the device (100) according to the first embodiment of the first example of the present disclosure is configured to store both the weight parameters of the upper model (UM) and the lower model (LM) when the device (100) operates. When the weight parameters of the upper model (UM) and the lower model (LM) are stored in the memory (MEM), the device (100) can efficiently process guess decoding.

[0119] The memory (MEM) of the device (100) according to the first embodiment of the first example of the present disclosure is configured to provide parameters of the upper model (UM) to the neural processing unit (NPU) when the neural processing unit (NPU) infers the upper model (UM), and is configured to provide parameters of the lower model (LM) to the neural processing unit (NPU) when the neural processing unit (NPU) infers the lower model (LM).

[0120] For example, when performing a speculative decoding operation, the device (100) can set an address map of the memory (MEM) to allocate a memory bank in which the weight parameters of the upper model (UM) and the lower model (LM) are to be stored. Accordingly, when performing a speculative decoding operation, the weight parameters of both the upper model (UM) and the lower model (LM) can reside in the memory (MEM).

[0121] The memory (MEM) of the device (100) according to the first embodiment of the first example of the present disclosure may include at least one memory among memories such as SRAM, LPDDR RAM, Resistive RAM, Magneto-resistive RAM, Phase-change RAM, Ferroelectric RAM, Flash Memory, HBM, etc. The memory (MEM) may be composed of at least one memory unit (e.g., bank, etc.). The memory (MEM) may be composed of a single (homogeneous) memory or a heterogeneous memory.

[0122] Accordingly, the capacity of the memory (MEM) of the device (100) according to the first embodiment of the first example of the present disclosure is configured to be larger than the sum of the weight parameters of the upper model (UM) and the lower model (LM). Accordingly, the weight parameters of the upper model (UM) and the lower model (LM) may reside in the memory (MEM). If the capacity of the upper model (UM) and the lower model (LM) of the memory (MEM) of the device (100) is insufficient, it may be difficult to quickly process the speculative decoding in real time. For example, the capacity of the memory (MEM) of the device (100) may be one of 4 GByte, 8 GByte, 16 GByte, and 32 GByte. For example, the capacity of the memory (MEM) of the device (100) may be 4 GByte to 32 GByte, and may be determined in consideration of the sizes of the parameters of the upper model (UM) and the lower model (LM) to be driven in the device (100).

[0123] If the parameters of both the upper model (UM) and the lower model (LM) are not stored in the memory (MEM), the conventional device must read the weight parameters of each model from an external storage device for speculative decoding. Therefore, the conventional device commands unnecessary memory read and memory write operations for speculative decoding. Therefore, the token generation speed of the conventional device becomes slower than the token generation speed of the device (100) according to the first embodiment of the first example of the present disclosure.

[0124] The power gating feature of a neural processing unit (NPU) completely cuts off power to unused circuit blocks, eliminating leakage power. Circuit blocks subject to power gating are equipped with power switches. These power switches can be used to isolate specific circuit blocks within the neural processing unit (NPU) from power. Power gating eliminates both dynamic power and leakage power, making it useful when the on-device is in low-power mode.

[0125] Clock gating in the neural processing unit (NPU) reduces dynamic power consumption by blocking clock signals from unused circuit blocks. Circuit blocks subject to clock gating can selectively block clock signals from specific circuit blocks using clock control logic. When clock gating is applied, power is continuously supplied to the NPU, maintaining the circuit's state. Because clock gating does not turn off power, operation can be resumed immediately when the clock is re-enabled, preventing unnecessary switching activity and reducing dynamic power consumption.

[0126] Register retention in the Neural Processing Unit (NPU) is a technology designed to allow specific registers in the NPU to maintain their state even when the NPU is powered off. Registers with register retention can maintain their stored data while using only low power during sleep mode. Consequently, even when the register enters sleep mode, important register values ​​are retained, eliminating the need for initialization upon recovery. Register retention can provide low-power operation and retain data using extremely low power consumption even when the NPU is powered off.

[0127] A device (100) according to a first embodiment of the first example of the present disclosure may be configured to provide low power capabilities for use on-device.

[0128] The low power mode of the device (100) according to the first embodiment of the first example of the present disclosure may be provided as, for example, a sleep mode, a deep sleep mode, a retention mode, etc. In some examples, the low power mode may be referred to as a standby mode, a hibernation mode, an idle mode, etc.

[0129] Sleep mode can be implemented, for example, by applying clock gating to a processing core (not shown) of a neural processing unit (NPU). Deep sleep mode can be implemented, for example, by applying power gating to a processing core (not shown) or memory (MEM) of a neural processing unit (NPU). Retention mode can be implemented by supplying a minimum maintenance voltage to a register (not shown) or memory (MEM) of a neural processing unit (NPU).

[0130] For example, the neural processing unit (NPU) of the device (100) according to the first embodiment of the first example of the present disclosure may be configured to provide at least one of power gating, clock gating, and register retention functions. In further detail, when the device (100) enters a sleep mode, the neural processing unit (NPU) may activate the clock gating function. In further detail, when the device (100) enters a deep sleep mode, the neural processing unit (NPU) may activate the power gating function. In further detail, when the device (100) enters a retention mode, the supply voltage input to the neural processing unit (NPU) may be reduced or the operating frequency may be reduced.

[0131] For example, the memory (MEM) of the device (100) according to the first embodiment of the first example of the present disclosure may be configured to provide at least one of deep sleep, retention, and maintain parameters functions. In further detail, when the device (100) enters a deep sleep mode, the memory (MEM) may be power gated, and in the case of a volatile memory device, data may be lost. In further detail, when the device (100) enters a retention mode, the supply voltage input to the memory (MEM) may be reduced or the operating frequency may be reduced, and in the case of a volatile memory device, data may be maintained. In further detail, when the device (100) enters a parameter maintenance mode, the supply voltage input to the memory (MEM) may be reduced or the operating frequency may be reduced, and in the case of a DRAM, the refresh period of a memory cell may be increased, and in the case of a volatile memory device, data may be maintained.

[0132] The low-power mode may be activated when no input query for speculative decoding is input to the device (100) for a certain period of time. The certain period of time may be provided based on, for example, a timer, a counter value, etc. The low-power mode may be activated based on a specific threshold time, and may be deactivated when an input query to be processed by speculative decoding is input to the device (100). The deactivation of the low-power mode may be referred to as a wake-up state.

[0133] The device (100) according to the first embodiment of the first example of the present disclosure may be a device capable of operating in a battery environment. Accordingly, the neural processing unit (NPU) may be designed to operate on battery voltage.

[0134] A device (100) according to a first embodiment of the first example of the present disclosure may be configured to provide at least one low-power mode while simultaneously providing speculative decoding in an on-device environment. The device (100) may enter the low-power mode based on preset conditions, thereby reducing the power consumption of the device (100).

[0135] FIG. 4 illustrates a device according to a second embodiment of the first example of the present disclosure.

[0136] In explaining Fig. 4, any content that overlaps with the explanations of Figs. 1 to 3 may be omitted for convenience of explanation.

[0137] Referring to FIG. 4, a device (200) according to a second embodiment of the first example of the present disclosure includes a neural processing unit (NPU), a first memory (MEM1), and a second memory (MEM2). The device (200) may further include at least one of a communication bus (not shown), a CPU (not shown), a GPU (not shown), and a storage device (not shown).

[0138] A device (200) according to the second embodiment of the first example of the present disclosure is configured to provide a service of the level of ultra-large AI on-device with low power by utilizing speculative decoding.

[0139] According to the device (200) according to the second embodiment of the first example of the present disclosure, the user's personal information and privacy can be protected because the AI ​​service can be provided only on-device.

[0140] A device (200) according to the second embodiment of the first example of the present disclosure is configured to receive an input query through an interface (IF).

[0141] A first communication bus (B1) is provided between the neural processing unit (NPU) and the first memory (MEM1). The first communication bus (B1) may be a dedicated communication bus for the neural processing unit (NPU) and the first memory (MEM1), and may be configured to provide stable memory bandwidth for speculative decoding.

[0142] A second communication bus (B2) is provided between the neural processing unit (NPU) and the second memory (MEM2). The second communication bus (B2) may be a dedicated communication bus for the neural processing unit (NPU) and the second memory (MEM2), and may be configured to provide stable memory bandwidth for speculative decoding.

[0143] The neural processing unit (NPU) of the device (200) according to the second embodiment of the first example of the present disclosure is an Application Specific Integrated Circuit (ASIC) implemented as an accelerator dedicated to neural network inference.

[0144] The neural processing unit (NPU) of the device (200) according to the second embodiment of the first example of the present disclosure may be configured as a system on a chip (SoC).

[0145] The neural processing unit (NPU) of the device (200) according to the second embodiment of the first example of the present disclosure is configured to infer an upper model (UM) and a lower model (LM). The upper model (UM) and the lower model (LM) are configured to cooperate with each other to process guess decoding. For a description of guess decoding, refer to FIGS. 1 and 2 .

[0146] Here, the neural processing unit (NPU) is configured to process the upper model (UM) and the lower model (LM) alternately. In other words, the neural processing unit (NPU) operates to sequentially process the upper model (UM) and the lower model (LM). In other words, the neural processing unit (NPU) operates to process the upper model (UM) and the lower model (LM) in a time-sharing manner. Therefore, the upper model (UM) and the lower model (LM) can be sequentially processed while increasing the utilization rate of the computational circuits of the neural processing unit (NPU). In addition, the operation method of the speculative decoding is such that the upper model (UM) and the lower model (LM) can be processed sequentially in the neural processing unit (NPU) of the device (200), and since the parameters of the upper model (UM) and the lower model (LM) reside in the first memory (MEM1) and the second memory (MEM2), respectively, a bottleneck phenomenon due to the speculative decoding operation may not occur in the neural processing unit (NPU) of the device (200).

[0147] The first memory (MEM1) of the device (200) according to the second embodiment of the first example of the present disclosure is configured to store weight parameters and tokens of the upper model (UM) when the device (200) operates.

[0148] The second memory (MEM2) of the device (200) according to the second embodiment of the first example of the present disclosure is configured to store the weight parameters and tokens of the lower model (LM) when the device (200) operates. When the weight parameters of both the upper model (UM) and the lower model (LM) are stored in the first memory (MEM1) and the second memory (MEM2), the device (200) can efficiently process guess decoding. To elaborate, the size of the weight parameters of the upper model (UM) and the lower model (LM) may be considerable, and it may be difficult to store them all in a single memory chip. In this case, the device (200) may configure a plurality of memories to increase the memory capacity.

[0149] The first memory (MEM1) of the device (200) according to the second embodiment of the first example of the present disclosure is configured to provide parameters of an upper model (UM) to the neural processing unit (NPU) when the neural processing unit (NPU) infers the upper model (UM), and the second memory (MEM2) is configured to provide parameters of a lower model (LM) to the neural processing unit (NPU) when the lower model (LM) is inferred.

[0150] For example, when the device (200) performs a speculative decoding operation, it can set an address map of each of the first memory (MEM1) and the second memory (MEM2) to allocate memory banks in which the weight parameters of the upper model (UM) and the lower model (LM) are to be stored. Accordingly, when the speculative decoding operation is performed, the weight parameters of the upper model (UM) and the lower model (LM) can both reside in the first memory (MEM1) and the second memory (MEM2).

[0151] The first memory (MEM1) of the device (200) according to the second embodiment of the first example of the present disclosure may be configured with at least one memory unit (e.g., bank, etc.). The first memory (MEM1) may be configured as a homogeneous memory or a heterogeneous memory.

[0152] The second memory (MEM2) of the device (200) according to the second embodiment of the first example of the present disclosure may be configured with at least one memory unit (e.g., a bank, etc.). The second memory (MEM2) may be configured as a homogeneous memory or a heterogeneous memory.

[0153] Accordingly, the capacity of the plurality of memories (MEM1, MEM2) of the device (200) according to the second embodiment of the first example of the present disclosure is configured to be larger than the sum of the weight parameters of the upper model (UM) and the lower model (LM). Accordingly, the weight parameters of the upper model (UM) and the lower model (LM) may reside in the plurality of memories (MEM1, MEM2). If the capacity of the upper model (UM) and the lower model (LM) of the plurality of memories (MEM1, MME2) of the device (200) is insufficient, it may be difficult to efficiently process speculative decoding.

[0154] If the parameters of both the upper model (UM) and the lower model (LM) are not stored in the multiple memories (MEM1, MEM2), the conventional device must read the weight parameters of each model from an external storage device for speculative decoding. Therefore, the conventional device commands unnecessary memory read and memory write operations for speculative decoding. Therefore, the token generation speed of the conventional device becomes slower than the token generation speed of the device (200) according to the second embodiment of the first example of the present disclosure.

[0155] A device (200) according to the second embodiment of the first example of the present disclosure may be configured to provide low power capabilities for use on-device.

[0156] The low power mode of the device (200) according to the second embodiment of the first example of the present disclosure may be provided as, for example, a sleep mode, a deep sleep mode, a retention mode, etc.

[0157] For example, the neural processing unit (NPU) of the device (200) according to the second embodiment of the first example of the present disclosure may be configured to provide at least one of power gating, clock gating, and register retention functions.

[0158] For example, at least one of the first memory (MEM1) and the second memory (ME2) of the device (200) according to the second embodiment of the first example of the present disclosure may be configured to provide at least one of deep sleep, retention, and maintain parameters functions.

[0159] The low power mode may be activated when no input query for speculative decoding is input to the device (200) for a certain period of time. The low power mode may be activated based on a specific threshold time, and may be deactivated when an input query to be processed by speculative decoding is input to the device (100).

[0160] A device (200) according to a second embodiment of the first example of the present disclosure may be configured to provide at least one low-power mode while simultaneously providing speculative decoding in an on-device environment. The device (200) may enter the low-power mode based on preset conditions, thereby reducing the power consumption of the device (200).

[0161] FIG. 5 illustrates a device according to a third embodiment of the first example of the present disclosure.

[0162] In explaining Fig. 5, any content that overlaps with the explanations of Figs. 1 to 4 may be omitted for convenience of explanation.

[0163] Referring to FIG. 5, a device (300) according to a third embodiment of the first example of the present disclosure includes a plurality of neural processing units (NPU1, NPU2) and a plurality of memories (MEM1, MEM2). The device (300) may further include at least one of a communication bus (not shown), a CPU (not shown), a GPU (not shown), and a storage device (not shown).

[0164] A device (300) according to the third embodiment of the first example of the present disclosure is configured to provide a service of the level of ultra-large AI on-device with low power by utilizing speculative decoding.

[0165] According to the device (300) according to the third embodiment of the first example of the present disclosure, the user's personal information and privacy can be protected because the AI ​​service can be provided only on-device.

[0166] A device (300) according to the third embodiment of the first example of the present disclosure is configured to receive an input query through an interface (IF).

[0167] A first communication bus (B1) is provided between the first neural processing unit (NPU1) and the first memory (MEM1). The first communication bus (B1) may be a dedicated communication bus for the first neural processing unit (NPU1) and the first memory (MEM1), and may be configured to provide a stable memory bandwidth for speculative decoding.

[0168] A second communication bus (B2) is provided between the second neural processing unit (NPU2) and the second memory (MEM2). The second communication bus (B2) may be a dedicated communication bus for the second neural processing unit (NPU2) and the second memory (MEM2), and may be configured to provide stable memory bandwidth for speculative decoding.

[0169] A communication channel capable of communicating with each other may be provided between the first neural processing unit (NPU1) and the second neural processing unit (NPU2). A communication channel capable of communicating with each other may be provided between the first memory (MEM1) and the second memory (MEM2). Accordingly, prompts, prefill tokens, accepted tokens, rejected tokens, or generated tokens generated through speculative decoding may be transmitted.

[0170] A plurality of neural processing units (NPU1, NPU2) of a device (300) according to the third embodiment of the first example of the present disclosure are Application Specific Integrated Circuits (ASICs) implemented as accelerators dedicated to neural network inference.

[0171] A plurality of neural processing units (NPU1, NPU2) of a device (300) according to the third embodiment of the first example of the present disclosure may be configured as a system on a chip (SoC).

[0172] According to the third embodiment of the first example of the present disclosure, the first neural processing unit (NPU1) of the device (300) is configured to infer an upper model (UM), and the second neural processing unit (NPU2) is configured to infer a lower model (LM). The upper model (UM) and the lower model (LM) are configured to cooperate with each other to process speculative decoding. For a description of speculative decoding, refer to FIGS. 1 and 2 .

[0173] Here, the first neural processing unit (NPU1) is set to process the upper model (UM), and the second neural processing unit (NPU2) is set to process the lower model (LM). That is, the plurality of neural processing units (NPU1, NPU2) operate to process the upper model (UM) and the lower model (LM) in parallel. In other words, the plurality of neural processing units (NPU1, NPU2) operate to independently process the upper model (UM) and the lower model (LM). Therefore, the plurality of neural processing units (NPU1, NPU2) can cooperate with each other to efficiently process the upper model (UM) and the lower model (LM). In addition, the operation method of the speculative decoding is such that the upper model (UM) and the lower model (LM) can be sequentially processed in the multiple neural processing units (NPU1, NPU2) of the device (200), and the parameters of the upper model (UM) and the lower model (LM) reside in the first memory (MEM1) and the second memory (MEM2), respectively, so that a bottleneck phenomenon due to the speculative decoding operation may not occur in the multiple neural processing units (NPU1, NPU2) of the device (200).

[0174] The first memory (MEM1) of the device (300) according to the third embodiment of the first example of the present disclosure is configured to store weight parameters and tokens of the upper model (UM) when the device (300) operates.

[0175] The second memory (MEM2) of the device (300) according to the third embodiment of the first example of the present disclosure is configured to store the weight parameters and tokens of the lower model (LM) when the device (300) operates. When the weight parameters of both the upper model (UM) and the lower model (LM) are stored in the first memory (MEM1) and the second memory (MEM2), the device (300) can efficiently process guess decoding. To elaborate, the size of the weight parameters of the upper model (UM) and the lower model (LM) may be considerable, and it may be difficult to store them all in a single memory chip. In this case, the device (300) may configure multiple memories to increase the memory capacity.

[0176] The first memory (MEM1) of the device (300) according to the third embodiment of the first example of the present disclosure is configured to provide parameters of the upper model (UM) to the first neural processing unit (NPU1) when the first neural processing unit (NPU1) infers the upper model (UM), and the second memory (MEM2) is configured to provide parameters of the lower model (LM) to the second neural processing unit (NPU2) when the second neural processing unit (NPU1) infers the lower model (LM).

[0177] For example, when the device (300) performs a speculative decoding operation, it can set an address map of each of the first memory (MEM1) and the second memory (MEM2) to allocate memory banks in which the weight parameters of the upper model (UM) and the lower model (LM) are to be stored. Accordingly, when the speculative decoding operation is performed, the weight parameters of the upper model (UM) and the lower model (LM) can both reside in the first memory (MEM1) and the second memory (MEM2).

[0178] The first memory (MEM1) of the device (300) according to the third embodiment of the first example of the present disclosure may be configured with at least one memory unit (e.g., bank, etc.). The first memory (MEM1) may be configured as a homogeneous memory or a heterogeneous memory.

[0179] The second memory (MEM2) of the device (300) according to the third embodiment of the first example of the present disclosure may be configured with at least one memory unit (e.g., bank, etc.). The second memory (MEM2) may be configured as a homogeneous memory or a heterogeneous memory.

[0180] Accordingly, the capacity of the plurality of memories (MEM1, MEM2) of the device (300) according to the third embodiment of the first example of the present disclosure is configured to be larger than the sum of the weight parameters of the upper model (UM) and the lower model (LM). Accordingly, the weight parameters of the upper model (UM) and the lower model (LM) may reside in the plurality of memories (MEM1, MEM2). If the capacity of the upper model (UM) and the lower model (LM) of the plurality of memories (MEM1, MME2) of the device (300) is insufficient, it may be difficult to efficiently process speculative decoding.

[0181] If the parameters of both the upper model (UM) and the lower model (LM) are not stored in the multiple memories (MEM1, MEM2), the conventional device must read the weight parameters of each model from an external storage device for speculative decoding. Therefore, the conventional device commands unnecessary memory read and memory write operations for speculative decoding. Therefore, the token generation speed of the conventional device becomes slower than the token generation speed of the device (300) according to the third embodiment of the first example of the present disclosure.

[0182] A device (300) according to the third embodiment of the first example of the present disclosure may be configured to provide low power functions for use on-device.

[0183] The low power mode of the device (300) according to the third embodiment of the first example of the present disclosure may be provided as, for example, a sleep mode, a deep sleep mode, a retention mode, etc.

[0184] For example, the plurality of neural processing units (NPU1, NPU2) of the device (300) according to the third embodiment of the first example of the present disclosure may be configured to provide at least one of power gating, clock gating, and register retention functions.

[0185] For example, the plurality of memories (MEM1, MEM2) of the device (300) according to the third embodiment of the first example of the present disclosure may be configured to provide at least one of deep sleep, retention, and maintain parameters functions.

[0186] The low power mode may be activated when no input query for speculative decoding is input to the device (300) for a certain period of time. The low power mode may be activated based on a specific threshold time, and may be deactivated when an input query to be processed by speculative decoding is input to the device (100).

[0187] Referring back to FIG. 2, when the first neural processing unit (NPU1) performs a prefill operation or an accept or reject operation of the upper model (UM), the second neural processing unit (NPU2) and the second memory (MEM2) may be in a low-power mode. Conversely, when the second neural processing unit (NPU2) performs a prefill operation or a generate operation of the lower model (LM), the first neural processing unit (NPU1) and the first memory (MEM1) may be in a low-power mode. However, each memory must preserve data in order to perform speculative decoding.

[0188] A device (300) according to a third embodiment of the first example of the present disclosure may be configured to provide at least one low-power mode while simultaneously providing speculative decoding in an on-device environment. The device (300) may enter the low-power mode based on preset conditions, thereby reducing the power consumption of the device (300).

[0189] FIG. 6 illustrates a system according to a first embodiment of the second example of the present disclosure.

[0190] In explaining Fig. 6, any content that overlaps with the explanations of Figs. 1 to 5 may be omitted for convenience of explanation.

[0191] Referring to FIG. 6, a system (400S) according to the first embodiment of the second example of the present disclosure includes a device (100) and a server (400). The system (400S) including the device (100) and the server (400) is configured to process joint operation between neural network models by utilizing speculative decoding.

[0192] The device (100) may be one of the devices (100, 200, 300) of the first example of the present disclosure. The device (100) includes a first neural processing unit (NPU1) and a first memory (MEM1). The device (100) may further include at least one of a communication bus (not shown), a CPU (not shown), a GPU (not shown), and a storage device (not shown).

[0193] The server (400) is an on-premise server. The server (400) may be designed to have, for example, a computing performance of 100 TOPS, 250 TOPS, 5,000 TOPS, or 10,000 TOPS. For example, the neural processing unit (NPU) may be designed to have a computing performance of 100 TOPS to 10,000 TOPS. Unlike cloud-based servers, on-premise servers are located in a data center, office, or dedicated server room of an organization, company, or institution. This allows direct control over hardware, software, security settings, etc.

[0194] The server (400) may include at least one neural processing unit (NPU) and various electronic system components. The server (400) may include digital circuits, analog circuits, mixed-signal and radio frequency processing circuits. For example, a processor that may be further included in the server (400) may be at least one of a central processing unit (CPU), a digital signal processor (DSP), an image signal processor (ISP), and a graphics processing unit (GPU). The server (400) may include at least one memory. The server (400) may include a high-speed data bus for efficient communication between various circuit components. The server (400) may include at least one interface, such as PCIe, USB, I2C, SPI, UART, and GPIO, for connection to external devices and sensors. The server (400) may include an on-chip power management device that regulates voltage and power distribution of the server (400). The server (400) may include a communication interface that integrates wired and wireless communication protocols such as Ethernet, Wi-Fi, Bluetooth, and cellular connections for data transmission.

[0195] The server (400) includes a second neural processing unit (NPU2) and a second memory (MEM2). The server (400) may further include at least one of a communication bus (not shown), a CPU (not shown), a GPU (not shown), and a storage device (not shown).

[0196] A system (400S) according to the first embodiment of the second example of the present disclosure is configured to provide a service of the level of ultra-large AI on-device with low power by utilizing speculative decoding.

[0197] According to the system (400S) according to the first embodiment of the second example of the present disclosure, since the inference AI service can be provided only on the on-premise server, the personal information and privacy of users within the server can be protected.

[0198] A system (400S) according to the first embodiment of the second example of the present disclosure is configured to receive an input query through an interface (IF).

[0199] A first communication bus (B1) is provided between the first neural processing unit (NPU1) and the first memory (MEM1). The first communication bus (B1) may be a dedicated communication bus for the first neural processing unit (NPU1) and the first memory (MEM1), and may be configured to provide a stable memory bandwidth for speculative decoding.

[0200] A second communication bus (B2) is provided between the second neural processing unit (NPU2) and the second memory (MEM2). The second communication bus (B2) may be a dedicated communication bus for the second neural processing unit (NPU2) and the second memory (MEM2), and may be configured to provide stable memory bandwidth for speculative decoding.

[0201] A plurality of neural processing units (NPU1, NPU2) of a system (400S) according to the first embodiment of the second example of the present disclosure are Application Specific Integrated Circuits (ASICs) implemented as accelerators dedicated to neural network inference.

[0202] A system (400S) according to the first embodiment of the second example of the present disclosure is configured to receive an input query via an interface (IF). The input query may first be transmitted to the server (400) to initiate guess decoding. However, the first embodiment of the second example of the present disclosure is not limited thereto, and the device (100) may also process guess decoding only in the device (100). In this case, the accuracy of the answer generated in the device (100) may be reduced, but the answer can be generated quickly. Alternatively, even if communication with the server (400) is interrupted, the device (100) can generate the answer.

[0203] The device (100) of the system (400S) according to the first embodiment of the second example of the present disclosure may be configured as a system on chip (SoC) and the server (400) may be configured as an on-premise server.

[0204] A communication channel capable of communicating with each other may be provided between the device (100) and the server (400). Accordingly, prompts, prefill tokens, accepted tokens, rejected tokens, or generated tokens generated from speculative decoding may be transmitted.

[0205] According to the first embodiment of the second example of the present disclosure, the first neural processing unit (NPU1) of the system (400S) is configured to infer a lower-order model (LM), and the second neural processing unit (NPU2) is configured to infer a higher-order model (UM). The upper-order model (UM) and the lower-order model (LM) are configured to cooperate with each other to process speculative decoding. For a description of speculative decoding, refer to FIGS. 1 and 2.

[0206] Here, the first neural processing unit (NPU1) is set to process the lower model (LM), and the second neural processing unit (NPU2) is set to process the upper model (UM), respectively. That is, the plurality of neural processing units (NPU1, NPU2) operate to process the upper model (UM) and the lower model (LM) in parallel. In other words, the plurality of neural processing units (NPU1, NPU2) operate to independently process the upper model (UM) and the lower model (LM). Therefore, the plurality of neural processing units (NPU1, NPU2) can cooperate with each other to efficiently process the upper model (UM) and the lower model (LM). In addition, the operation method of the speculative decoding is such that the upper model (UM) and the lower model (LM) can be sequentially processed in multiple neural processing units (NPU1, NPU2) of the system (400S), and the parameters of the lower model (LM) and the upper model (UM) reside in the first memory (MEM1) and the second memory (MEM2), respectively, so that a bottleneck phenomenon due to the speculative decoding operation may not occur in the multiple neural processing units (NPU1, NPU2) of the system (400S).

[0207] The first memory (MEM1) of the device (100) according to the first embodiment of the second example of the present disclosure is configured to store weight parameters and tokens of the lower model (LM) when the device (100) operates.

[0208] The second memory (MEM2) of the server (400) according to the first embodiment of the second example of the present disclosure is configured to store the weight parameters and tokens of the upper model (UM) when the server (400) operates. When the weight parameters of both the lower model (LM) and the upper model (UM) are stored in the first memory (MEM1) and the second memory (MEM2), the system (400S) can efficiently process guess decoding. To elaborate, the size of the weight parameters of the upper model (UM) and the lower model (LM) may be considerable, and it may be difficult to store them all in a single memory chip. In this case, the system (400S) may configure multiple memories to increase the memory capacity.

[0209] The first memory (MEM1) of the system (400S) according to the first embodiment of the second example of the present disclosure is configured to provide parameters of a lower model (LM) to the first neural processing unit (NPU1) when the first neural processing unit (NPU1) infers a lower model (LM), and the second memory (MEM2) is configured to provide parameters of an upper model (UM) to the second neural processing unit (NPU2) when the second neural processing unit (NPU1) infers a higher model (UM).

[0210] For example, when the system (400S) performs a speculative decoding operation, it can set an address map of each of the first memory (MEM1) and the second memory (MEM2) to allocate memory banks in which the weight parameters of the upper model (UM) and the lower model (LM) are to be stored. Accordingly, when the speculative decoding operation is performed, the weight parameters of the lower model (LM) and the upper model (UM) can both reside in the first memory (MEM1) and the second memory (MEM2).

[0211] The first memory (MEM1) of the system (400S) according to the first embodiment of the second example of the present disclosure may be configured with at least one memory unit (e.g., bank, etc.). The first memory (MEM1) may be configured as a homogeneous memory or a heterogeneous memory.

[0212] The second memory (MEM2) of the system (400S) according to the first embodiment of the second example of the present disclosure may be configured with at least one memory unit (e.g., a bank, etc.). The second memory (MEM2) may be configured as a homogeneous memory or a heterogeneous memory.

[0213] Accordingly, the capacity of the plurality of memories (MEM1, MEM2) of the system (400S) according to the first embodiment of the second example of the present disclosure is configured to be larger than the sum of the weight parameters of the upper model (UM) and the lower model (LM). Accordingly, the weight parameters of the upper model (UM) and the lower model (LM) may reside in the plurality of memories (MEM1, MEM2). If the capacity of the upper model (UM) and the lower model (LM) of the plurality of memories (MEM1, MME2) of the system (400S) is insufficient, it may be difficult to efficiently process speculative decoding.

[0214] If the parameters of both the upper model (UM) and the lower model (LM) are not stored in multiple memories (MEM1, MEM2), the conventional system must read the weight parameters of each model from an external storage device for speculative decoding. Therefore, the conventional system commands unnecessary memory read and write operations for speculative decoding. Therefore, the token generation speed of the conventional system becomes slower than the token generation speed of the system (400S) according to the first embodiment of the second example of the present disclosure.

[0215] A first device (100) according to the first embodiment of the second example of the present disclosure may be configured to provide a low-power function for use on-device.

[0216] The low power mode of the device (100) according to the first embodiment of the second example of the present disclosure may be provided as, for example, a sleep mode, a deep sleep mode, a retention mode, etc.

[0217] For example, the plurality of neural processing units (NPU1, NPU2) of the system (400S) according to the first embodiment of the second example of the present disclosure may be configured to provide at least one of power gating, clock gating, and register retention functions.

[0218] For example, the plurality of memories (MEM1, MEM2) of the system (400S) according to the first embodiment of the second example of the present disclosure may be configured to provide at least one of deep sleep, retention, and maintain parameters functions.

[0219] The low power mode may be activated when no input query for speculative decoding is input to the device (100) for a certain period of time. The low power mode may be activated based on a specific threshold time, and may be deactivated when an input query to be processed by speculative decoding is input to the device (100).

[0220] Referring back to FIG. 2, when the second neural processing unit (NPU2) performs a prefill operation or an accept or reject operation of the upper model (UM), the first neural processing unit (NPU1) and the first memory (MEM1) may be in a low-power mode. Conversely, when the first neural processing unit (NPU1) performs a prefill operation or a generate operation of the lower model (LM), the second neural processing unit (NPU2) and the second memory (MEM2) may be in a low-power mode. However, each memory must preserve data in order to perform speculative decoding.

[0221] A system (400S) according to the first embodiment of the second example of the present disclosure may be configured to provide at least one low-power mode while simultaneously providing speculative decoding in an on-premises server environment. The system (400S) may enter the low-power mode based on preset conditions, thereby reducing the power consumption of the system (400S).

[0222] FIG. 7 illustrates a system according to a second embodiment of the second example of the present disclosure.

[0223] In explaining Fig. 7, any content that overlaps with the explanations of Figs. 1 to 6 may be omitted for convenience of explanation.

[0224] Referring to FIG. 7, a system (410S) according to a second embodiment of the second example of the present disclosure includes a plurality of devices (100-1, 100-2) and a server (400). The system (410S) including the plurality of devices (100-1, 100-2) and the server (400) is configured to process joint operation between neural network models by utilizing speculative decoding.

[0225] Each of the plurality of devices (100-1, 100-2) may be substantially identical to one of the devices described in the first or second example of the present disclosure.

[0226] The server (400) is an on-premise server. The server (400) includes at least one neural processing unit (not shown) and at least one memory (not shown). The number of neural processing units and memories included in the server (400) may vary depending on the number of devices connected to the server (400). The server (400) may further include at least one of a communication bus (not shown), a CPU (not shown), a GPU (not shown), and a storage device (not shown).

[0227] A system (410S) according to a second embodiment of the second example of the present disclosure is configured to provide a service at the level of ultra-large-scale AI from a plurality of on-devices at low power by utilizing speculative decoding.

[0228] According to the system (410S) according to the second embodiment of the second example of the present disclosure, since the inference AI service can be provided only on the on-premise server, the personal information and privacy of users within the server can be protected.

[0229] A system (410S) according to a second embodiment of the second example of the present disclosure is configured to receive at least one input query via multiple interfaces (IF1, IF2). The input query may first be transmitted to a server (400) to initiate guess decoding. However, the second embodiment of the second example of the present disclosure is not limited thereto, and at least one of the multiple devices (100-1, 100-2) may process guess decoding only on the device. In this case, the accuracy of the answer generated by the device may be reduced, but the answer can be generated quickly. Alternatively, the answer can be generated on the device even if communication with the server (400) is interrupted.

[0230] Each of the plurality of devices (100-1, 100-2) of the system (410S) according to the second embodiment of the second example of the present disclosure may be configured as a system on a chip (SoC), and the server (400) may be configured as an on-premise server.

[0231] According to a second embodiment of the second example of the present disclosure, each of a plurality of devices (100-1, 100-2) of a system (410S) is configured to infer a first lower model (LM1) and a second lower model (LM2), respectively, and the server (400) is configured to infer an upper model (UM). The upper model (UM) and the first lower model (LM1) and the second lower model (LM2) are configured to cooperate with each other to process speculative decoding. For a description of speculative decoding, refer to FIGS. 1, 2, and 8.

[0232] Here, the first device (100-1) is set to process the first lower model (LM1), the second device (100-2) is set to process the second lower model (LM2), and the server (400) is set to process the upper model (UM). That is, each device (100-1, 100-2) processes its respective lower model (LM1, LM2), and the server (400) operates to process the upper model (UM) corresponding to the plurality of lower models (LM1, LM2). Therefore, the server (400) can efficiently process the plurality of lower models (LM). This will be described in detail with reference to FIG. 9.

[0233] The memory (not shown) of the first device (100-1) according to the second embodiment of the second example of the present disclosure is configured to store the weight parameters and tokens of the first sub-model (LM1) when the first device (100-1) operates.

[0234] The memory (not shown) of the second device (100-2) according to the second embodiment of the second example of the present disclosure is configured to store the weight parameters and tokens of the second sub-model (LM2) when the second device (100-2) operates.

[0235] The memory (not shown) of the server (400) according to the second embodiment of the second example of the present disclosure is configured to store the weight parameters and tokens of the upper model (UM) when the server (400) operates.

[0236] A plurality of devices (100-1, 100-2) according to the second embodiment of the second example of the present disclosure may be configured to provide low-power functions for use in each on-device.

[0237] The low power mode of the plurality of devices (100-1, 100-2) according to the second embodiment of the second example of the present disclosure may be provided as, for example, a sleep mode, a deep sleep mode, a retention mode, etc.

[0238] For example, the neural processing unit (not shown) of the first device (100-1) or the second device (100-2) according to the second embodiment of the second example of the present disclosure may be configured to provide at least one of power gating, clock gating, and register retention functions.

[0239] For example, the memory (not shown) of the first device (100-1) or the second device (100-2) according to the second embodiment of the second example of the present disclosure may be configured to provide at least one of deep sleep, retention, and maintain parameters functions.

[0240] For example, at least one neural processing unit (not shown) of a server (400) according to the second embodiment of the second example of the present disclosure may be configured to provide at least one of power gating, clock gating, and register retention functions.

[0241] For example, at least one memory (not shown) of a server (400) according to the second embodiment of the second example of the present disclosure may be configured to provide at least one of deep sleep, retention, and maintain parameters functions.

[0242] The low-power mode may be activated when no input query for speculative decoding is input to the system (410S) for a certain period of time. The low-power mode may be activated based on a specific threshold time, and may be deactivated when an input query processed by speculative decoding is input to the system (410S).

[0243] Referring again to FIGS. 2 and 8, when the server (400) performs a prefill operation or an acceptance or rejection operation of the upper model (UM), the neural processing units (not shown) and memories (not shown) of the multiple devices (100-1, 100-2) may be in a low power mode. Conversely, when the first device (100-1) performs a prefill operation or a generation operation of the first lower model (LM1), the second device (100-2) may be in a low power mode.

[0244] A system (410S) according to a second embodiment of the second example of the present disclosure can be configured to provide speculative decoding in an on-premises server environment while simultaneously providing at least one low-power mode. The system (410S) according to the second embodiment of the second example of the present disclosure can efficiently process speculative decoding through a plurality of devices (100-1, 100-2) and a server (400), and each of the devices (100-1, 100-2) can be independently activated (e.g., wake-up mode) or deactivated (e.g., low-power mode).

[0245] Figure 8 schematically illustrates a speculative decoding operation according to a second example of the present disclosure.

[0246] In explaining Fig. 8, any content that overlaps with the explanation of Fig. 2 may be omitted for convenience of explanation.

[0247] Referring to FIGS. 7 and 8, a plurality of lower level models (LM1, LM2) and an upper level model (UM) are illustrated. A system (410S) according to a second example of the present disclosure includes a plurality of devices (100-1, 100-2) and a server (400). Each device (100-1, 100-2) is connected to a corresponding interface (IF1, IF2). The system (410S) may be configured to sequentially or in parallel process the plurality of lower level models (LM1, LM2) and the upper level model (UM) for speculative decoding.

[0248] The upper model (LM) of the server (400) sequentially receives each input query through multiple devices (100-1, 100-2). Next, the server (400) sequentially processes the first guess decoding and the second guess decoding. Referring to Fig. 8, the hatched boxes filled with dots represent operations corresponding to the first guess decoding, and the boxes without dots represent operations corresponding to the second guess decoding.

[0249] For convenience of explanation, the following description will be given with reference to time points (T1 to T8) on the time axis.

[0250] The server (400) processes a prefill operation corresponding to the first guess decoding using the upper model (UM) and transmits the prefill operation result corresponding to the first guess decoding to the first device (100-1) (T1). Next, the server (400) processes a prefill operation corresponding to the second guess decoding using the upper model (UM) and transmits the prefill operation result corresponding to the second guess decoding to the second device (100-2) (T2). At this time, the first device (100-1) receives the prefill operation result of the upper model (UM) corresponding to the first guess decoding (T1), and processes the prefill operation and a series of generation operations corresponding to the first guess decoding using the first lower model (LM1) to generate a series of token candidates and transmits them to the server (400) (T3). The server (400) receives candidate tokens corresponding to the first guess decoding generated by the first device (100-1) (T3), and accepts or rejects the token candidates corresponding to the first guess decoding by utilizing the upper model (UM) (T4). Meanwhile, the second device (100-2) receives the prefill operation result of the upper model (UM) corresponding to the second guess decoding (T2), and processes the prefill operation and a series of generation operations corresponding to the second guess decoding by utilizing the second lower model (LM2), generates a series of token candidates, and transmits them to the server (400) (T5).

[0251] Meanwhile, the first device (100-1) receives the result of the acceptance or rejection operation of the upper model (UM) corresponding to the first guess decoding (T4), and processes a series of generation operations corresponding to the first guess decoding using the first lower model (LM1) to additionally generate a series of token candidates and transmits them to the server (400) (T6). Here, the server (400) receives candidate tokens generated by the second device (100-2) corresponding to the second guess decoding (T5), and accepts or rejects token candidates corresponding to the second guess decoding using the upper model (UM) (T7). Here, a delay in the operation corresponding to the first guess decoding may occur in the server (400). Specifically, the acceptance or rejection operation corresponding to the first guess decoding may be delayed from the start time (T5) to the end time (T7) of the acceptance or rejection operation corresponding to the second guess decoding. That is, from the time point (T6) when the first device (100-1) transmits a series of token candidates generated by utilizing the first lower model (LM1) to the server (400) to the time point (T7), the operation of the corresponding upper model (UM) is delayed.

[0252] Next, the server (400) transmits the acceptance or rejection results of the token candidates corresponding to the second guess decoding to the second device (100-2) (T7). At this time, the second device (100-2) receives the acceptance or rejection results of the upper model (UM) corresponding to the second guess decoding (T7), and uses the second lower model (LM2) to process a series of generation operations corresponding to the second guess decoding to generate a series of token candidates, and transmits them to the server (400) (T9). Subsequently, the server (400) uses the upper model (UM) to process the acceptance or delay operations corresponding to the first guess decoding from a time point (T7) after the time point (T6) at which the first device (100-1) receives the series of token candidates generated by the first lower model (LM1), and transmits the acceptance or rejection results to the first device (100-1) (T8).

[0253] Next, the first device (100-1) receives the acceptance or rejection result of the upper model (UM) corresponding to the first guess decoding (T8), and uses the first lower model (LM1) to process a series of generation operations corresponding to the first guess decoding to generate a series of token candidates and transmits them to the server (400) (T10).

[0254] The server (400) receives candidate tokens generated by the second device (100-2) corresponding to the second guess decoding (T9) and, using the upper model (UM), accepts or rejects the token candidates corresponding to the second guess decoding (T10). Next, the server (400) receives candidate tokens generated by the first device (100-1) corresponding to the first guess decoding (T10) and, using the upper model (UM), accepts or rejects the token candidates corresponding to the first guess decoding (T11).

[0255] The above steps are repeated until the stopping conditions are met.

[0256] According to the second embodiment of the second example of the present disclosure, the system (410S) can efficiently process speculative decoding by utilizing a plurality of devices (100-1, 100-2) and a server (400). In addition, the server (400) can efficiently collaborate with a plurality of devices (100-1, 100-2) in speculative decoding in a batch manner.

[0257] Referring to Figure 8, the number of parameters of the upper model (UM) is greater than the number of parameters of the first lower model (LM1) or the second lower model (LM2). In other words, the number of parameters of the heterogeneous language models is different. Therefore, the processing time of the prefill stage of the upper model (UM) is longer than the processing time of the prefill stage of the lower model (LM). The processing time of the accept or reject stage of the upper model (UM) is longer than the processing time of the generation stage of the lower model (LM). Therefore, the more tokens the lower model generates, the faster the token generation speed of the system can be.

[0258] The lower-level model can utilize a language model whose parameter count is determined by considering the hardware resource limitations of the on-device being processed. The upper-level model can utilize a language model whose parameter count is determined by considering the hardware resource limitations of the processing device.

[0259] In some examples, the submodel may be a model with pruned parameters from the parent model. For example, the submodel may be a model with pruned parameters from the parent model. In such cases, the parameters of the pruned neural network model can be made lighter, and the memory bandwidth required may be reduced compared to the parent model.

[0260] FIG. 9 illustrates a system according to a third example of the present disclosure.

[0261] In explaining Fig. 9, any content that overlaps with the explanations of Figs. 1 to 8 may be omitted for convenience of explanation.

[0262] Referring to FIG. 9, a system (500S) according to a third example of the present disclosure includes a plurality of devices (100-1, 100-2, 100-3, 100-4), a plurality of on-premise servers (400-1, 400-2), and a cloud server (500). The system (500S) including a plurality of devices (100-1, 100-2, 100-3, 100-4), a plurality of on-premise servers (400-1, 400-2), and a cloud server (500) is configured to process joint operation between neural network models by utilizing speculative decoding.

[0263] Each of the plurality of devices (100-1, 100-2, 100-3, 100-4) may be substantially identical to one of the devices described in the first or second example of the present disclosure.

[0264] Each of the plurality of on-premises servers (400-1, 400-2) may be a device substantially identical to the server described in the second example of the present disclosure.

[0265] The cloud server (500) includes at least one neural processing unit (not shown) and at least one memory (not shown). The number of neural processing units and memories included in the cloud server (500) may vary depending on the number of on-premise servers connected to the cloud server (500). The cloud server (500) may further include at least one of a communication bus (not shown), a CPU (not shown), a GPU (not shown), and a storage device (not shown). The cloud server (500) may be configured to communicate with the on-premise servers (400-1, 400-2) via a wired or wireless communication network. The cloud server (500) may be designed to have, for example, a computational performance of 10,000 TOPS to 100,000 TOPS.

[0266] A system (500S) according to a third example of the present disclosure is configured to provide a service at the level of ultra-large AI from multiple on-devices at low power by utilizing speculative decoding.

[0267] A system (500S) according to a third example of the present disclosure is configured to receive at least one input query through a plurality of interfaces (IF1, IF2, IF3, IF4). The input query may first be transmitted to a corresponding on-premises server to initiate guess decoding. However, the third example of the present disclosure is not limited thereto, and at least one of the plurality of devices (100-1, 100-2, 100-3, 100-4) may process guess decoding only on the device. In this case, the accuracy of the answer generated by the device may be reduced, but the answer can be generated quickly. Alternatively, the answer can be generated on the device even if communication with the corresponding on-premises server is cut off. However, the third example of the present disclosure is not limited thereto, and at least one of the plurality of devices (100-1, 100-2, 100-3, 100-4) may process guess decoding only up to the on-premises server. That is, the cloud server (500) can be optionally excluded. In this case, the accuracy of the answers generated by the device may be reduced, but answers can be generated quickly. Alternatively, even if communication with the cloud server (500) is interrupted, answers can be generated from the on-premises server.

[0268] Each of the plurality of devices (100-1, 100-2, 100-3, 100-4) of the system (500S) according to the third example of the present disclosure may be configured as a system on a chip (SoC).

[0269] According to the third example of the present disclosure, each of the plurality of devices (100-1, 100-2, 100-3, 100-4) of the system (500S) is configured to infer a respective lower-level model, and the plurality of on-premise servers (400-1, 400-2) are configured to infer a middle-level model. The middle-level model may refer to a model that has more parameters than the lower-level model and fewer parameters than the upper-level model. The plurality of lower-level models, the plurality of middle-level models, and the upper-level models are configured to cooperate with each other to process guess decoding. In order to generate a higher-level answer, the system (500S) may be configured to accept or reject tokens of the lower-level model by utilizing the higher-level model.

[0270] The memory (not shown) of each device according to the third example of the present disclosure is configured to store the weight parameters and tokens of each sub-model when each device operates.

[0271] The memory (not shown) of each on-premises server according to the third example of the present disclosure is configured to store the weight parameters and tokens of each intermediate model when each on-premises server operates.

[0272] The memory (not shown) of the cloud server (500) according to the third example of the present disclosure is configured to store weight parameters and tokens of the upper model when the cloud server (500) operates.

[0273] A plurality of devices (100-1, 100-2, 100-3, 100-4) according to the third example of the present disclosure may be configured to provide low power functions for use in each on-device.

[0274] The low power modes of the plurality of devices (100-1, 100-2, 100-3, 100-4) according to the third example of the present disclosure may be provided, for example, as a sleep mode, a deep sleep mode, a retention mode, etc.

[0275] A system (500S) according to a third example of the present disclosure may be configured to provide speculative decoding in an on-premise server environment and / or a cloud server environment while simultaneously providing at least one low power mode.

[0276] FIG. 10 illustrates a neural processing unit that can be applied to examples of the present disclosure.

[0277] A neural processing unit (1000) applicable to examples of the present disclosure may include a DMA (1100), an NPU memory (1200), a processing element core (1300), a vector core (1400), a scalar core (1500), a number system conversion unit (1600), and an NPU control unit (1700).

[0278] The neural processing unit (1000) is a processor specialized for operations of a deep neural network (DNN), separate from a processor such as a CPU or GPU. In particular, the neural processing unit (1000) may include multiple processing elements specialized for matrix multiplication, which accounts for most of the computational load of the DNN.

[0279] The NPU control unit (1700) may be electrically connected to the DMA (1100), the NPU memory (1200), the processing element core (1300), the vector core (1400), the scalar core (1500), and the number system conversion unit (1600). The NPU control unit (1700) may be configured to control operations related to speculative decoding operations in the neural processing unit (1000). The NPU control unit (1700) may be configured to execute speculative decoding by executable code generated by a compiler (not shown). Here, the compiler (not shown) may be a semiconductor circuit or may be software running on a separate device. The compiler (not shown) may convert a specific neural network model (e.g., an upper model (UM) or a lower model (LM)) into executable code that can be executed in the neural processing unit (1000). That is, a compiler (not shown) can generate a neural processing unit-specific executable code that can be executed on a specific neural processing unit having specific performance. The executable code may also be referred to as machine code or binary code. The executable code may include a plurality of executable codes corresponding to the upper model (UM) (e.g., a first executable code and a second executable code corresponding to the lower model (LM)). Accordingly, the NPU control unit (1700) may be configured to control the DMA (1100), the NPU memory (1200), the processing element core (1300), the vector core (1400), the scalar core (1500), and the number system conversion unit (1600) based on various commands and operation schedules included in the executable code. The executable code may be programmed to control the neural processing unit (1000) according to the length of the token of the generated response.

[0280] For example, the compiler can generate executable code that determines the read / write order of data according to speculative decoding, the order of neural network operation processing, etc. based on the hardware characteristics of the first neural processing unit (NPU1) (e.g., the computational performance of the processing element core, the computational performance of the vector core, the computational performance of the scalar core, the operating point of the number system conversion unit, the capacity of the NPU memory, and the memory bandwidth of the corresponding bus).

[0281] The DMA (1100) is configured to allow the neural processing unit (1000) to directly access and read / write a memory (e.g., at least the memory illustrated in FIGS. 3 to 5) electrically connected to the neural processing unit (1000). The neural processing unit (1000) can transmit various data related to speculative decoding from a memory (not illustrated) via the DMA (1100). The DMA (1100) can be configured to perform tasks such as setting an address of the NPU memory (1200), generating and controlling read / write commands, etc.

[0282] The NPU memory (1200) is a memory placed in the on-chip area of ​​the neural processing unit (1000) and may include a memory for caching or storing data processed in the on-chip area or a register file for storing instructions for driving a compiled neural network model. Here, the register file may also be configured to be included in the NPU control unit (1700). The NPU memory (1200) may read and store data required for a speculative decoding operation from a memory (not shown) connected to a bus. The NPU memory (1200) may include one of the following memories: SRAM, Register file, DRAM, Resistive RAM, Magneto-resistive RAM, Phase-change RAM, Ferroelectric RAM, Flash Memory, etc. The NPU memory (1200) may be composed of at least one memory unit. The NPU memory (1200) may be composed of a homogeneous memory unit or a heterogeneous memory unit. The capacity of the NPU memory (1200) may be, for example, several megabytes to tens of megabytes, but is not limited thereto.

[0283] The processing element core (1300), the vector core (1400), and the scalar core (1500) are computational circuits configured to efficiently process the attention layer of the upper model (UM) or the lower model (LM). The attention layer can obtain attention (Q, K, V), which is an operation value, using parameters such as an input value and a query (Q), a key (K), and a value (V). The processing element core (1300), the vector core (1400), and the scalar core (1500) are designed to process operations such as Matmul, DynamicQuant, RMSNorm, and Softmax for attention operations. The data format of the above operations may be one of a multidimensional matrix format, a two-dimensional matrix format, a one-dimensional array format, or a scalar format, and one of the processing element core (1300), the vector core (1400), and the scalar core (1500) may be selected to process the operation depending on the data format. Accordingly, the neural processing unit (1000) can quickly accelerate the operation of the generative artificial intelligence model with low power.

[0284] The processing element core (1300) is composed of NxM operators. N and M are integers greater than 1. The processing element core (1300) is a two-dimensional array. The processing element core (1300) may be configured to process a matrix multiplication operation of input data and corresponding parameters (e.g., weights, kernels, queries (Q), keys (K), values ​​(V), etc.) by processing a plurality of processing elements (PEs) in parallel. The processing element (PE) may include a MAC (Multiply And Accumulate) operator. The processing element core (1300) may efficiently process matrix multiplication (e.g., Matmul or GEMM) or convolution operations (e.g., Conv). The processing element core (1300) may be configured to input integer parameters and output integer parameters. In particular, as the number of processing elements (PEs) included in the processing element core (1300) increases, the number of transistors in the ASIC increases, which may increase manufacturing costs. If the processing element core (1300) is designed to process integer parameters rather than floating point parameters, the number of transistors in the ASIC can be reduced, and the power consumption of the neural processing unit (1000) can be reduced. The number of processing elements (PEs) may range from hundreds to tens of thousands.

[0285] The vector core (1400) is composed of 1xN operators. N is an integer greater than 1. The vector core (1400) is configured to include a plurality of ALU (Arithmetic Logic Unit) operators. The vector core (1400) includes a multiplier and an adder. The vector core (1400) may include a vector register file for storing one-dimensional array data. The vector core (1400) may be configured to receive floating-point parameters and output floating-point parameters.

[0286] The scalar core (1500) is composed of 1x1 calculation units. The scalar core (1500) is a circuit designed to process only one data element. The scalar core (1500) is configured to include an ALU (Arithmetic Logic Unit) calculation unit. The scalar core (1500) includes a multiplier and an adder. The scalar core (1500) may be configured to input floating-point parameters and output floating-point parameters.

[0287] The number system conversion unit (1600) may be configured to include a quantization circuit and a dequantization circuit. For example, the input and output of the processing element core (1300) may be integers. The input and output of the vector core (1400) and the scalar core (1500) may be floating point. Accordingly, the number system conversion unit (1600) may convert data into a data format required by each core when transmitting data between the processing element core (1300), the vector core (1400), and the scalar core (1500). The quantization circuit of the number system conversion unit (1600) may convert integers into floating point numbers. The dequantization circuit of the number system conversion unit (1600) may convert floating point numbers into integers.

[0288] According to the examples of the present disclosure, floating point is not limited to IEEE standard 754, and Brain Floating Point can be applied to improve computational efficiency, reduce memory usage, reduce power consumption, etc. of the neural processing unit (1000). However, the examples of the present disclosure are no longer limited, and Dynamic Floating-Point, Variable Precision Floating-Point (VPFP), Custom Floating-Point (CFP), Flexible Floating-Point (FFP), etc. can be applied. VPFP is a floating-point format that can dynamically set or adjust the bit width of the exponent and mantissa. VPFP allows for flexible precision levels based on specific calculations, thereby optimizing power and performance. Dynamic floating-point format, which is used interchangeably with variable precision, allows for the size of the mantissa and exponent to be adjusted on the fly during runtime, allowing for real-time response to data characteristics. CFP represents an application-specific format in which the bit allocation for the mantissa and exponent is adjusted to suit the needs of a specific use case. This can be useful for hardware implementations that select the correct bit width to optimize hardware resources and data accuracy. FFP stands for Floating Point Format, which allows for adjustable bit allocation between the exponent and mantissa. FFP can be implemented in hardware accelerators to provide an optimal balance between range and precision. The bit width of floating-point numbers according to the examples of the present disclosure may be between 4 and 32 bits. The bit width of integers according to the examples of the present disclosure may be between 4 and 32 bits.

[0289] FIG. 11 is a schematic conceptual diagram illustrating processing elements that can be applied to examples of the present disclosure.

[0290] Referring to FIG. 11, a processing element (PE) may be configured to include a multiplier, an adder, and an accumulator.

[0291] The multiplier multiplies the input (N) bit data by (M) bit data. The operation value of the multiplier is output as (N+M) bit data, where N and M are integers greater than 0. The first input unit may be configured to receive (N) bit data. The second input unit may be configured to receive (M) bit data. Here, the first input unit may be configured to receive an activation parameter, and the second input unit may be configured to receive a weight parameter.

[0292] The NPU control unit (1700) knows that when a parameter having a value of 0 is input to one of the first input and second input of the multiplier, the result of the operation will be 0 even if the operation is not performed, so the processing element (PE) can limit the operation (e.g., zero-skipping operation) so that the multiplier does not perform the operation. For example, when 0 is input to one of the first input and second input of the multiplier of the processing element (PE), the multiplier can be configured to operate in a zero-skipping manner. In further explanation, the lower model (LM) may have fewer parameters than the upper model (UM) or may be a model obtained by pruning the upper model (UM). In this case, the power consumption of the neural processing unit (1000) that processes the lower model (LM) can be reduced and the efficiency increased by zero-skipping. For zero-skipping, each processing element (PE) included in the processing element core (1300) can be activated or deactivated. The NPU control unit (1700) may be configured to provide an activation or deactivation signal (CLOCK-GATING CONTROL) to each processing element (PE) in clock units. Here, when the processing element (PE) is deactivated, the multiplier may be configured to be deactivated. Accordingly, power consumed for the operation of the multiplier may be reduced. For example, information on the power consumption of the multiplier may be referred to FIG. 12. The processing element (PE) may be designed to receive a control signal (CLOCK-GATING CONTROL) for zero-skipping operation control (i.e., activation or deactivation) from the NPU control unit (1700). Specifically, the multipliers of the processing elements (PEs) may be designed to receive each control signal (CLOCK-GATING CONTROL) for zero-skipping operation control from the NPU control unit (1700).Alternatively, the adder of the processing element (PE) may be modified to receive a control signal (CLOCK-GATING CONTROL) for zero-skipping operation control from the NPU control unit (1700). Alternatively, each multiplier and adder of the processing element (PE) may be modified to receive a control signal (CLOCK-GATING CONTROL) for zero-skipping operation control simultaneously from the NPU control unit (1700).

[0293] The bit width of the parameters input to the first input unit and the second input unit of the processing element (PE) can be determined according to the quantization of the activation parameters and the quantization of the weight parameters of the upper model (UM) and the lower model (LM), respectively. For example, the activation parameters of the upper model (UM) can be quantized to 12 bits and the weight parameters can be quantized to 8 bits, and the activation parameters of the lower model (UM) can be quantized to 8 bits and the weight parameters can be quantized to 4 bits. The quantization level of each parameter can be determined by a compiler (not shown). That is, the bit widths of the parameters input to each input unit of the processing element (PE) can be different from each other. The quantization information of the data input to each input unit of the processing element (PE) can be included in the execution code. The execution code can include a first execution code corresponding to the upper model (UM) and a second execution code corresponding to the lower model (LM).

[0294] The accumulator accumulates the operation value of the multiplier and the operation value of the accumulator using an adder for the number of (L) loops. Therefore, the bit width of the data of the output and input portions of the accumulator can be output as (N+M+log2(L)) bits. Here, L is an integer greater than 0. When accumulation is finished, the accumulator can initialize the data stored inside the accumulator to 0 by inputting an initialization signal (INTIALIZATION RESET). The accumulator of the processing element (PE) is configured to maintain the accumulated value when zero skipping is activated. The (X) bit can be set to a bit width that does not cause overflow of the output data (X) bit based on the maximum value that can be accumulated in the accumulator. For example, the (X) bit can be 16 bits to 64 bits.

[0295] To elaborate, the number system conversion unit (1600) can convert an integer parameter output from the processing element core (1300) into a floating point and transmit it to one of the vector core (1400), the scalar core (1500), and the NPU memory (1200). The number system conversion unit (1600) can be controlled by the NPU control unit (1700).

[0296] FIG. 12 is a table schematically illustrating the energy consumption per unit operation of a neural processing unit according to examples of the present disclosure.

[0297] Referring to FIG. 12 below, the power consumption of the processing element core (1300), vector core (1400), and scalar core (1500) of the neural processing unit (1000) will be described.

[0298] Referring to FIG. 12, a table schematically illustrates the energy consumed per unit operation of various calculation units of the neural processing unit (1000). For example, energy consumption can be explained by dividing it into addition and multiplication operations. However, the energy consumed per unit operation may vary depending on the foundry process technology of the semiconductor chip (e.g., 2 nm technology, 5 nm technology, 7 nm technology, 14 nm technology, 28 nm technology, etc.).

[0299] The processing core (1300) will be described with reference to the processing core (1300). The processing core (1300) can be designed to operate on integer parameters. "8b Add" in FIG. 12 refers to an 8-bit integer addition operation of an adder. An 8-bit integer addition operation can consume 0.03 pj of energy. "16b Add" refers to a 16-bit integer addition operation of an adder. A 16-bit integer addition operation can consume 0.05 pj of energy. "32b Add" refers to a 32-bit integer addition operation of an adder. A 32-bit integer addition operation can consume 0.1 pj of energy. To elaborate, when the processing core (1300) includes thousands or more processing elements (PEs) designed as integer operators, the power consumption reduction of the neural processing unit (1000) can be significantly reduced compared to when the processing core includes the same number of processing elements designed as floating-point operators. In general, most operations in generative artificial intelligence models are processed in the processing core (1300). Therefore,

[0300] A description will be made with reference to a vector core (1400) and a scalar core (1500). The vector core (1400) and the scalar core (1500) can be designed to operate on floating-point parameters. "16b FP Add" refers to a 16-bit floating-point add operation of an adder. A 16-bit floating-point add operation can consume 0.4 pj of energy. "32b FP Add" refers to a 32-bit floating-point add operation of an adder. A 32-bit floating-point add operation can consume 0.9 pj of energy. "8b Mult" refers to an 8-bit integer multiply operation of a multiplier. An 8-bit integer multiply operation can consume 0.2 pj of energy. "32b Mult" refers to a 32-bit integer multiply operation of a multiplier. A 32-bit integer multiplication operation can consume 3.1 pj of energy. "16b FP Mult" refers to a 16-bit floating-point multiplication operation on a multiplier. A 16-bit floating-point multiplication operation can consume 1.1 pj of energy. "32b FP Mult" refers to a 32-bit floating-point multiplication operation on a multiplier. A 32-bit floating-point multiplication operation can consume 3.7 pj of energy.

[0301] According to examples of the present disclosure, the bit width of the integer parameter of the lower model (LM) may be equal to or smaller than the bit width of the integer parameter of the upper model (UM). A compiler (not shown) may be configured to determine the bit width of the integer parameter of the lower model (LM) and the bit width of the integer parameter of the upper model (UM). For example, the executable code of the upper model (UM) compiled to process a weight parameter in the form of an integer of the first bit width may be executed in the NPU control unit (1700) in the neural processing unit (1000). The executable code of the upper model (UM) compiled to process an activation parameter in the form of an integer of the second bit width may be executed in the NPU control unit (1700) in the neural processing unit (1000). The executable code of the lower model (LM) compiled to process a weight parameter in the form of an integer of the third bit width may be executed in the NPU control unit (1700) in the neural processing unit (1000). The execution code of the lower model (LM) compiled to process the activation parameter in the form of an integer of the fourth bit width can be executed in the NPU control unit (1700) in the neural processing unit (1000).

[0302] According to examples of the present disclosure, the bit width of the floating-point parameter of the lower model (LM) may be equal to or smaller than the bit width of the floating-point parameter of the upper model (UM). A compiler (not shown) may be configured to determine the bit width of the floating-point parameter of the lower model (LM) and the bit width of the floating-point parameter of the upper model (UM). For example, the executable code of the upper model (UM) compiled to process the weight parameter in the form of a floating-point of the fifth bit width may be executed in the NPU control unit (1700) of the neural processing unit (1000). The executable code of the upper model (UM) compiled to process the activation parameter in the form of a floating-point of the sixth bit width may be executed in the NPU control unit (1700) of the neural processing unit (1000). The execution code of the lower model (LM) compiled to process the weight parameter in the form of a floating point number with a 7-bit width can be executed in the NPU control unit (1700) of the neural processing unit (1000). The execution code of the lower model (LM) compiled to process the activation parameter in the form of a floating point number with an 8-bit width can be executed in the NPU control unit (1700) of the neural processing unit (1000).

[0303] The technical features of the examples of the present disclosure can be described as follows.

[0304] A device according to examples of the present disclosure may include a first memory having a first capacity configured to store a first generative neural network model comprising first parameters; and a first neural processing unit configured to generate a response corresponding to an input query by utilizing the first generative neural network model stored in the first memory. The first neural processing unit may be configured to store a first executable code of the first generative neural network model compiled to process speculative decoding. Accordingly, the device may process speculative decoding. Furthermore, a federated driving system of a plurality of language models utilizing speculative decoding (or speculation decoding) within an on-device may be provided.

[0305] According to some examples of the present disclosure, the first memory may be configured to further store a second generative neural network model comprising two parameters, wherein the number of the second parameters is greater than the number of the first parameters. Accordingly, speculative decoding can be implemented on a single device.

[0306] According to some examples of the present disclosure, the first memory of the first capacity may be configured to store both the first generative neural network model and the second generative neural network model. This can prevent a decrease in computational speed due to increased memory operations caused by separate external memory during speculative decoding.

[0307] According to some examples of the present disclosure, the first neural processing unit may be configured to store a second executable code of the second generative neural network model compiled to process speculative decoding. Thus, the device may process multiple generative neural network models simultaneously.

[0308] According to some examples of the present disclosure, the first neural processing unit may further be configured to include a first NPU memory configured to communicate with the first memory and a first NPU control unit configured to control the first neural processing unit.

[0309] According to some examples of the present disclosure, the first executable code may be configured to be stored in the first NPU memory or the first NPU control unit.

[0310] The device according to the examples of the present disclosure may further include a second memory having a second capacity configured to store a second generative neural network model comprising second parameters. Accordingly, the device can effectively implement guess decoding with one neural processing unit and multiple memories.

[0311] A device according to examples of the present disclosure may further include a second memory having a second capacity configured to store a second generative neural network model comprising second parameters; and a second neural processing unit configured to generate a response corresponding to an input query by utilizing the second generative neural network model stored in the second memory. The second neural processing unit may be configured to store a second executable code of the second generative neural network model compiled to process speculative decoding.

[0312] According to some examples of the present disclosure, the computational performance of the second neural processing unit may be higher than that of the first neural processing unit. For example, the computational performance of the first neural processing unit may be 25 TOPS, and the computational performance of the second neural processing unit may be 100 TOPS. Accordingly, the token generation speed of the second generative neural network model processed by the second neural processing unit may be reduced, thereby improving the guess decoding speed of the device.

[0313] According to some examples of the present disclosure, the device may be configured to operate in at least one of a sleep mode, a deep sleep mode, or a retention mode. At least one of the first neural processing unit and the second neural processing unit may be configured to operate in a low-power mode when waiting for an operation result of another neural processing unit during speculative decoding. Accordingly, when one neural processing unit processes a prefill, generate, or accept or reject operation, the power consumption of the other neural processing unit can be reduced.

[0314] A system according to examples of the present disclosure may include an on-premises server, including a device comprising a first memory having a first capacity configured to store a first generative neural network model comprising a first number of parameters; and a first neural processing unit configured to generate a response corresponding to an input query by utilizing the first generative neural network model stored in the first memory; and a second memory having a second capacity configured to store a second generative neural network model comprising a second number of parameters; and a second neural processing unit configured to generate a response corresponding to the input query by utilizing the second generative neural network model stored in the second memory. The first neural processing unit and the second neural processing unit may be configured to cooperate with each other to process speculative decoding. Accordingly, the system may process speculative decoding. Furthermore, a federated driving system of a plurality of language models utilizing speculative decoding (or speculation decoding) within an on-device may be provided.

[0315] According to some examples of the present disclosure, the first memory of the first capacity may be capable of storing the first generative neural network model, and the second memory of the second capacity may be capable of storing the second generative neural network model. Accordingly, the system can prevent a decrease in computational speed due to an increase in memory operations caused by a separate external memory during speculative decoding.

[0316] According to some examples of the present disclosure, the first neural processing unit may be configured to store a first executable code of the first generative neural network model compiled to process speculative decoding, and the second neural processing unit may be configured to store a second executable code of the second generative neural network model compiled to process speculative decoding. Accordingly, the system may implement speculative decoding by utilizing a plurality of generative neural network models.

[0317] According to some examples of the present disclosure, the computational performance of the second neural processing unit of the server may be higher than the computational performance of the first neural processing unit of the device. Accordingly, the processing speed of the system's speculative decoding may be improved.

[0318] According to some examples of the present disclosure, the system may include multiple devices, each of which may be configured to process speculative decoding via the server. Accordingly, the server can effectively process speculative decoding corresponding to multiple devices.

[0319] According to some examples of the present disclosure, at least one of the plurality of devices may be configured to operate in at least one of a sleep mode, a deep sleep mode, or a retention mode. Accordingly, due to the nature of the speculative decoding, the power consumption of the device in standby can be reduced.

[0320] According to some examples of the present disclosure, the first generative neural network model may be a lower-level model, and the second generative neural network model may be a higher-level model. The higher-level model may be configured to process speculative decoding while being selectively deactivated. Accordingly, even if the standby neural processing unit operates in low-power mode, the token generation speed may not be affected, and power consumption may be reduced.

[0321] According to some examples of the present disclosure, the speculative decoding may include a prefill operation, a generate operation, and an accept or reject operation.

[0322] A device according to examples of the present disclosure includes a first memory configured to store a first generative neural network model, and a first neural processing unit configured to generate a response corresponding to an input query by utilizing the first generative neural network model stored in the first memory. The first neural processing unit may include a processing core configured to receive an integer parameter input, a vector core and a scalar core configured to receive a floating-point parameter input, and a number system conversion unit configured to convert the integer parameter and the floating-point parameter to process an operation of the first generative neural network model. Accordingly, the device can quickly process speculative decoding with low power.

[0323] The examples of the present disclosure disclosed in this specification and drawings are intended solely to facilitate the technical content of the present disclosure and provide specific examples to aid understanding thereof, and are not intended to limit the scope of the present disclosure. It will be apparent to those skilled in the art to which the present disclosure pertains that other variations based on the technical concepts of the invention are possible, in addition to the examples disclosed herein.

[0324] [National Research and Development Project Supporting This Invention]

[0325] [Project ID] 1711195792

[0326] [Assignment Number] 00228938

[0327] [Ministry Name] Ministry of Science and ICT

[0328] [Name of Project Management (Specialist) Agency] Information and Communications Technology Planning and Evaluation Institute

[0329] [Research Project Name] Development of AI Semiconductor Software Integrated Platform Technology

[0330] [Research Project Name] Development of a Commercial Edge AI SoC Semiconductor SW Development Platform Technology

[0331] [Name of Project Performing Organization] DeepX

[0332] Research Period: April 1, 2023 - December 31, 2023

Claims

1. A first memory having a first capacity configured to store a first generative neural network model composed of first parameters; and A first neural processing unit configured to generate a response corresponding to an input query by utilizing the first generative neural network model stored in the first memory; A device wherein the first neural processing unit is configured to store a first executable code of the first generative neural network model compiled to process guess decoding.

2. In paragraph 1, The first memory of the first capacity is configured to store a second generative neural network model composed of second parameters, A device wherein the number of the second parameters is greater than the number of the first parameters.

3. In paragraph 2, A device wherein the first memory of the first capacity has a capacity capable of storing both the first generative neural network model and the second generative neural network model.

4. In paragraph 2, A device wherein the first neural processing unit is configured to store a second execution code of the second generative neural network model compiled to process the guess decoding.

5. In paragraph 1, A device wherein the first neural processing unit further comprises a first NPU memory configured to communicate with the first memory and a first NPU control unit configured to control the first neural processing unit.

6. In paragraph 5, A device configured such that the first execution code is stored in the first NPU memory or the first NPU control unit.

7. In paragraph 1, A device further comprising a second memory having a second capacity configured to store a second generative neural network model comprising second parameters.

8. In paragraph 1, A second memory having a second capacity configured to store a second generative neural network model comprising second parameters; and Further comprising a second neural processing unit configured to generate a response corresponding to an input query by utilizing the second generative neural network model stored in the second memory; A device wherein the second neural processing unit is configured to store a second execution code of the second generative neural network model compiled to process the guess decoding.

9. In paragraph 8, A device wherein the computational performance of the second neural processing unit is higher than the computational performance of the first neural processing unit.

10. In paragraph 8, A device, wherein at least one of the first neural processing unit and the second neural processing unit is configured to operate in a low power mode when waiting for an operation result of another neural processing unit during the speculative decoding.

11. A device comprising: a first memory having a first capacity configured to store a first generative neural network model composed of first parameters; and a first neural processing unit configured to generate a response corresponding to an input query by utilizing the first generative neural network model stored in the first memory; and An on-premise server comprising a second memory having a second capacity configured to store a second generative neural network model composed of second parameters; and a second neural processing unit configured to generate a response corresponding to an input query by utilizing the second generative neural network model stored in the second memory; A system wherein the first neural processing unit and the second neural processing unit are configured to cooperate with each other to process guess decoding.

12. In paragraph 11, A system wherein the first memory of the first capacity has a capacity capable of storing the first generative neural network model, and the second memory of the second capacity has a capacity capable of storing the second generative neural network model.

13. In paragraph 11, A system wherein the first neural processing unit is configured to store a first execution code of the first generative neural network model compiled to process the speculative decoding, and the second neural processing unit is configured to store a second execution code of the second generative neural network model compiled to process the speculative decoding.

14. In paragraph 11, A system wherein the computational performance of the second neural processing unit of the on-premise server is higher than the computational performance of the first neural processing unit of the device.

15. In paragraph 11, A system in which the above devices are multiple, and each device is configured to process the speculative decoding through the on-premises server.

16. In paragraph 15, A system wherein at least one of the plurality of devices is configured to operate in at least one of a sleep mode, a deep sleep mode, or a retention mode.

17. In paragraph 11, A system wherein the first generative neural network model is a lower-level model and the second generative neural network model is a higher-level model.

18. In paragraph 17, A system configured to process the speculative decoding while the above upper model is optionally disabled.

19. In paragraph 11, The above speculative decoding system includes a prefill operation, a generation operation, and an accept or reject operation.

20. A first memory configured to store a first generative neural network model; and A first neural processing unit configured to generate a response corresponding to an input query by utilizing the first generative neural network model stored in the first memory; The first neural processing unit: In order to process the operation of the first generative neural network model, A processing core configured to receive integer parameter input; A vector core and a scalar core configured to receive floating-point parameters; and A device comprising a number system conversion unit configured to convert the integer parameter and the floating point parameter.

Images