System for processing heterogeneous generative artificial intelligence models

Abstract

In accordance with the present disclosure, an apparatus is provided. The apparatus includes: a first memory having a first capacity configured to store a first generative neural network model including a first parameter; and a first neural processing unit configured to generate a response corresponding to the input query using the first generative neural network model stored in the first memory; and wherein the first neural processing unit may be configured to store first execution code of the first generative neural network model, the first execution code being compiled to process the speculative decoding.

Description

Technical Field

[0001] The present disclosure relates to a system for processing heterogeneous generative artificial intelligence models. More specifically, it relates to a federated operation system for processing heterogeneous language models. Background Art

[0002] Humans possess intelligence such as recognition, classification, inference, prediction, and control / decision-making. Artificial intelligence (AI) refers to the artificial imitation of human intelligence.

[0003] The human brain is composed of numerous nerve cells called neurons. Each neuron is connected to hundreds to thousands of other neurons via connections called synapses. To emulate human intelligence, neural network (NN) models simulate the operation and connections of biological neurons. Specifically, a NN model is a system of nodes connected in a layered structure that mimics the structure of neurons.

[0004] With advances in neural network technology, the applications of neural network inference services leveraging big data-based training are becoming increasingly diverse. These inference services train neural networks on massive datasets, enabling the trained models to reason about diverse and complex data. However, as the demand for the performance and accuracy of neural network inference services continues to increase, the parameter size, computational complexity, and memory bandwidth requirements of neural network models are also growing exponentially. Consequently, the performance requirements for processors and memory capable of supporting these inference tasks are also increasing. To meet these demands, neural network inference services are increasingly deployed on cloud computing-based servers, which provide efficient processing power for large-scale data operations.

[0005] Generative AI models are increasingly being developed for a variety of contexts. These models include various neural network models, such as language models for interactive systems, stable diffusion models for generating images from text input, and Transformer models for predicting and determining future actions based on a sequence of events in a given situation. Using these generative AI models to respond to input queries can be computationally expensive. For example, large language models (LLMs) are a particularly commercially viable type of generative AI model. Currently, only high-performance general-purpose graphics processing units (GPGPUs) can provide sufficient computational power for LLM processing. However, the total power consumption of GPGPUs operating worldwide to process these LLMs has reached levels equivalent to the energy consumption of an entire country. This excessive energy demand poses a significant challenge to the commercialization of LLMs. Building GPGPU-based data centers to provide LLM services requires careful evaluation of power plant capacity to determine whether they can meet the required energy requirements. In many urban areas, the difficulty of meeting the energy needs of these data centers is a significant barrier to the widespread deployment of large-scale AI models. Summary of the Invention Technical issues to be resolved

[0006] Based on the number of layers, neural networks are divided into "single-layer neural networks" and "multi-layer neural networks." A typical multi-layer neural network consists of an input layer, a hidden layer, and an output layer. (1) The input layer is the layer that receives input values. The number of input layers is the same as the number of input variables. (2) The hidden layer is located between the input layer and the output layer. It receives signals from the input layer, extracts features, and passes them to the output layer. (3) The output layer is the layer that receives signals from the hidden layer and outputs them to the outside.

[0007] To achieve higher levels of artificial intelligence, a deep neural network (DNN) with a larger number of hidden layers has been disclosed. Among DNNs, the Transformer neural network is a DNN based on the attention mechanism. The Transformer utilizes many matrix multiplication operations. The Transformer can accept input values ​​and parameters such as a query (Q), a key (K), and a value (V) to obtain attention parameters (Q, K, V), which are operation values. The Transformer can perform various inference operations based on the operation values ​​(e.g., attention (Q, K, V)). The Transformer has performed well in the field of generative artificial intelligence.

[0008] Transformer-based neural network models are finding application in language models. For example, a language model can take a text query as input and generate a response. The response generated by the language model can be generated by re-inputting the tokens generated as part of the language model's response into the language model. The cost of generating such a response from a language model is proportional to the number of parameters in the language model. Therefore, generating responses using a language model with a large number of parameters is computationally expensive. Conversely, reducing the number of parameters can reduce computational cost and increase response speed, but the quality of the generated response may be affected.

[0009] However, the tasks of the present disclosure are not limited to the above-mentioned tasks, and those of ordinary skill in the art will understand other tasks not mentioned from the following description. Technical Solution

[0010] According to an example of the present disclosure, a device may be provided. The device may include: a first memory having a first capacity, configured to store a first generative neural network model including first parameters; and a first neural processing unit configured to generate a response corresponding to an input query using the first generative neural network model stored in the first memory. The first neural processing unit may be configured to store first execution code of the first generative neural network model, the first execution code being compiled to process speculative decoding.

[0011] According to some examples of the present disclosure, a first memory having a first capacity may be configured to store a second generative neural network model including second parameters. The number of the second parameters may be greater than the number of the first parameters.

[0012] According to some examples of the present disclosure, a first memory having a first capacity may have a capacity capable of storing both a first generative neural network model and a second generative neural network model.

[0013] According to some examples of the present disclosure, the first neural processing unit may be configured to store second execution code of a second generative neural network model, the second execution code being compiled to process speculative decoding.

[0014] According to some examples of the present disclosure, the first neural processing unit may further include: a first neural processing unit memory configured to communicate with the first memory; and a first neural processing unit controller configured to control the first neural processing unit.

[0015] According to some examples of the present disclosure, the first execution code may be configured to be stored in a first neural processing unit memory or in a first neural processing unit controller.

[0016] According to some examples of the present disclosure, it may further include: a second memory having a second capacity, configured to store a second generative neural network model including second parameters.

[0017] According to some examples of the present disclosure, the system may further include: a second memory having a second capacity configured to store a second generative neural network model including second parameters; and a second neural processing unit configured to generate a response corresponding to the input query using the second generative neural network model stored in the second memory. The second neural processing unit may be configured to store second execution code of the second generative neural network model, the second execution code being compiled to process speculative decoding.

[0018] According to some examples of the present disclosure, the computing performance of the second neural processing unit may be higher than the computing performance of the first neural processing unit.

[0019] According to some examples of the present disclosure, during speculative decoding, at least one of a first neural processing unit and a second neural processing unit may be configured to operate in a low power mode while waiting for a computation result from another neural processing unit.

[0020] According to an example of the present disclosure, a system may be provided. The system may include a device comprising: a first memory having a first capacity, configured to store a first generative neural network model including first parameters; and a first neural processing unit configured to generate a response to an input query using the first generative neural network model stored in the first memory; and a local server comprising: a second memory having a second capacity, configured to store a second generative neural network model including second parameters; and a second neural processing unit configured to generate a response to the input query using the second generative neural network model stored in the second memory. The first neural processing unit and the second neural processing unit may be configured to cooperate with each other to process speculative decoding.

[0021] According to some examples of the present disclosure, a first memory having a first capacity may store a first generative neural network model, and a second memory having a second capacity may store a second generative neural network model.

[0022] According to some examples of the present disclosure, a first neural processing unit may be configured to store a first execution code of a first generative neural network model, the first execution code being compiled to process speculative decoding, and the second neural processing unit may be configured to store a second execution code of the second generative neural network model, the second generative neural network model being compiled to process speculative decoding.

[0023] According to some examples of the present disclosure, the computing performance of the second neural processing unit of the local server may be higher than the computing performance of the first neural processing unit of the device.

[0024] According to some examples of the present disclosure, the device may be plural, and each device may be configured to process speculative decoding through the local server.

[0025] According to some examples of the present disclosure, at least one of the plurality of devices may be configured to operate in at least one of a sleep mode, a deep sleep mode, or a retention mode.

[0026] According to some examples of the present disclosure, the first generative neural network model may be a lower model, and the second generative neural network model may be an upper model.

[0027] According to some examples of the present disclosure, the upper model may be configured to handle speculative decoding with selective deactivation.

[0028] According to some examples of the present disclosure, speculative decoding may include a pre-fill operation, a generate operation, and an accept or reject operation.

[0029] According to an example of the present disclosure, a device may be provided. The device may include: a first memory configured to store a first generative neural network model; and a first neural processing unit configured to generate a response corresponding to an input query using the first generative neural network model stored in the first memory. The first neural processing unit may include: a processing core configured to receive an input integer parameter; a vector core and a scalar core configured to receive an input floating-point parameter; and a digital format conversion unit configured to convert integer parameters and floating-point parameters for operation of the first generative neural network model.

[0030] Specific details of other examples are included in the detailed description and accompanying drawings. Effects of the Invention

[0031] According to examples of the present disclosure, a federated operating system utilizing heterogeneous language models can significantly reduce energy consumption, carbon emissions, and energy costs compared to relying solely on data centers. This can facilitate the implementation of various generative AI services without significantly increasing the capacity of existing power plants.

[0032] According to the examples of the present disclosure, since generative AI services can be processed by neural processing units that have lower reasoning performance than servers but can operate at low power consumption, the unit manufacturing cost on the device side can be reduced.

[0033] According to examples of the present disclosure, a generative artificial intelligence service can be provided in a device-side environment in real time.

[0034] According to examples of the present disclosure, an artificial intelligence service can be provided in a device-side to protect user privacy and personal information.

[0035] According to examples of the present disclosure, a device-side environment can be configured to implement speculative decoding while providing at least one low-power mode, thereby reducing power consumption of the device-side.

[0036] According to examples of the present disclosure, speculative decoding can be implemented in a device-side environment with a heterogeneous language model to take advantage of the token generation speed of the lower model while maintaining the inference accuracy level of the upper model.

[0037] According to examples of the present disclosure, a device-side environment and a server environment can be selectively modified to implement speculative decoding with a heterogeneous language model having a fast token generation rate or a high response accuracy, thereby taking advantage of the token generation rate of the lower model while maintaining the inference accuracy level of the upper model.

[0038] According to examples of the present disclosure, by providing at least one neural processing unit and at least one memory capable of implementing speculative decoding with a heterogeneous language model in a device-side environment, speculative decoding can be processed sequentially or in parallel, thereby increasing the token generation speed on the device.

[0039] Effects according to examples of the present disclosure are not limited to the above examples, and the present disclosure includes more effects. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 A tree data for speculative decoding according to examples of the present disclosure is shown.

[0041] Figure 2 A speculative decoding operation according to a first example of the present disclosure is schematically shown.

[0042] Figure 3 A device according to a first implementation of the first example of the present disclosure is shown.

[0043] Figure 4 A device according to a second implementation of the first example of the present disclosure is shown.

[0044] Figure 5 A device according to a third implementation of the first example of the present disclosure is shown.

[0045] Figure 6 A system according to a first implementation of a second example of the present disclosure is shown.

[0046] Figure 7A system is shown in accordance with a second embodiment of the second example of the present disclosure.

[0047] Figure 8 A speculative decoding operation is shown in accordance with the second example of the present disclosure is shown schematically.

[0048] Figure 9 A system is shown in accordance with a third embodiment of the present disclosure.

[0049] Figure 10 A neural processing unit applicable to examples of the present disclosure is shown.

[0050] Figure 11 A processing element applicable to examples of the present disclosure is shown.

[0051] Figure 12 A table showing energy per operation of a neural processing unit in accordance with examples of the present disclosure is shown schematically. DETAILED DESCRIPTION

[0052] Certain structural or procedural descriptions are provided to illustrate examples in accordance with the concepts of the present disclosure.

[0053] Examples in accordance with the concepts of the present disclosure can be practiced in a variety of forms. The present disclosure is not to be construed as being limited to only the embodiments described herein.

[0054] Various modifications can be made to examples in accordance with the concepts of the present disclosure. The concepts of the present disclosure can take many different forms. Accordingly, certain examples are shown in the drawings and are described in detail in the present disclosure. However, this is not meant to limit examples in accordance with the concepts of the present disclosure to any particular disclosed form and it is to be understood that all modifications, equivalents or alternatives falling within the scope of the present disclosure are included in the present disclosure.

[0055] Terms such as "first" and / or "second" can be used to describe various elements. However, the present disclosure is not limited by such terms.

[0056] The above terms are used only to distinguish one element from another. For example, a first element can be named a second element, and similarly, a second element can be named a first element, without departing from the scope of the appended claims in accordance with the concepts of the present disclosure.

[0057] When referring to an element being "connected" or "coupled" to another element, it should be understood as the element can be directly connected or coupled to the other element, but other elements can be present therebetween. On the other hand, when referring to an element being "directly connected" or "directly coupled" to another element, it should be understood that no other element is present therebetween.

[0058] Other expressions describing the relationship between elements, such as “between” and “directly between” or “adjacent” and “directly adjacent” should be interpreted similarly.

[0059] As used herein, expressions such as "A or B," "at least one of A and / or B," or "one or more of A and / or B" may include any possible combination of the above listed items. For example, "A or B," "at least one of A and B," or "at least one of A or B" may refer to all of the following: (1) including at least one A, (2) including at least one B, or (3) including at least one A and at least one B.

[0060] As used in this disclosure, expressions such as "first," "second," "first," or "second" may refer to various elements, regardless of their order and / or importance. These expressions are used to distinguish one element from another and do not limit these elements. For example, a first user device and a second user device may refer to different user devices, regardless of their order or importance. For example, a first element may be named a second element, and similarly, a second element may be interchangeably named with a first element without departing from the scope of this disclosure.

[0061] The terminology used in this disclosure is for describing particular embodiments only and is not intended to limit the scope of other examples.

[0062] Unless the context clearly indicates otherwise, singular expressions may include pluralities. Terms used herein, including technical or scientific terms, should have the same meanings as those generally understood by those skilled in the art to which this document relates.

[0063] The terms used in this disclosure with common dictionary definitions should be interpreted as having the same or similar meanings as those in the context of the relevant art. Unless otherwise expressly defined herein, the terms used in this disclosure should not be interpreted in an idealized or overly formal manner. Therefore, even if a term is defined herein, it should not be interpreted as excluding embodiments of the present disclosure.

[0064] The terminology used in the present disclosure is for describing particular embodiments only and is not intended to be limiting of the present disclosure.

[0065] Unless the context clearly indicates otherwise, singular expressions also include the plural. In this specification, terms such as "including" or "having" are intended to indicate the presence of the described features, numbers, steps, operations, elements, components, or combinations thereof, and should therefore be understood as not excluding the possibility of the presence or addition of one or more other features, numbers, steps, operations, elements, components, or combinations thereof.

[0066] Unless otherwise defined, all terms used herein, including technical or scientific terms, shall have the same meaning as commonly understood by those skilled in the art to which this disclosure belongs. Such terms, as defined in commonly used dictionaries, shall be understood to have the same meaning as they have in the context of the relevant art. Unless otherwise expressly defined in this disclosure, they should not be interpreted in an idealized or overly formal manner.

[0067] The features of the various examples of the present disclosure may be combined in part or in whole or in combination with one another. The various examples of the present disclosure may be associated with one another and operated in various technical ways that will be understood by those skilled in the art. The various examples of the present disclosure may be implemented independently of one another or in association with one another.

[0068] When describing each example, technical details that are familiar to those skilled in the art and not directly related to the present disclosure will be omitted. This is done to make the present disclosure clearer without obscuring the essence of the present disclosure by omitting unnecessary explanations.

[0069] According to examples of the present disclosure, devices, methods, systems, and computer-readable media capable of running generative artificial intelligence services on a device side may be provided.

[0070] Generally speaking, Transformer-based generative AI models take a query as input and generate a response. For example, a language model receives a query in text form and generates tokens. The query and generated tokens are then fed back into the language model to generate the next token. This token generation operation can be repeated multiple times until the response to the query is complete. A token can correspond to a word or part of a word. The response generated by the language model can contain a series of tokens, which are further generated by passing tokens generated as part of the response generated by the language model back to the language model.

[0071] The cost of generating responses for such language models is proportional to the number of parameters in the language model. Therefore, generating responses using language models with a large number of parameters is computationally expensive. However, reducing the number of parameters can reduce computational cost and increase response speed, but at the expense of response accuracy. In other words, there is a trade-off between the number of parameters in the language model and the time it takes the processor to process operations to generate tokens in the language model.

[0072] Specifically, in order to generate responses of a certain quality level, the number of parameters in the language model must be large enough. On the other hand, processing such a language model can be challenging due to the limitations of device-side hardware resources (e.g., energy consumption, battery capacity, memory capacity, AI processing power, etc.).

[0073] Accordingly, the present disclosure proposes, through various examples, a federated operating system of a language model that utilizes speculative decoding to drive heterogeneous language models with limited device-end hardware resources.

[0074] According to examples of the present disclosure, the federated operating system of the heterogeneous language model can include at least one neural processing unit, at least one memory, and can be configured to drive the heterogeneous language model using the at least one neural processing unit.

[0075] At the device end, in order to speed up the generation of word pieces of the language model, the speculative decoding is configured to utilize the heterogeneous language model. According to examples of the present disclosure, the heterogeneous language model can include at least a first language model and a second language model. In some examples, the heterogeneous language model can include a first to third language model. In some examples, the heterogeneous language model can include a first to fourth language model.

[0076] Among them, the parameter quantity of the first language model is less than that of the second language model. The parameter quantity of the second language model is less than that of the third language model. The parameter quantity of the third language model is less than that of the fourth language model, that is, the parameter quantity of the heterogeneous language model is different. The language model with a relatively small parameter quantity can be defined as a lower model, and the language model with a relatively large parameter quantity can be defined as an upper model. That is, the first language model can be defined as a lower model of the second language model, and the second language model can be defined as an upper model of the first language model. The second language model can be defined as a lower model of the third language model, and the third language model can be defined as an upper model of the second language model. The first language model can be defined as a lower model of the second language model, and the second language model can be defined as an upper model of the first language model.

[0077] The lower model can adopt a language model whose parameter quantity is determined by considering the hardware resource limitation of the device end being processed. The upper model can adopt a language model whose parameter quantity is determined by considering the hardware resource limitation of the device being processed. For example, the parameter quantity of the upper model can be 7 billion, and the parameter quantity of the lower model can be 1 billion. For example, the parameter quantity of the upper model can be 32 billion, the parameter quantity of the intermediate model can be 7 billion, and the parameter quantity of the lower model can be 1 billion.

[0078] The following describes the speculative decoding used in examples of the present disclosure.

[0079] Systems according to examples of the present disclosure are configured to cause lower and upper models to cooperate with each other to perform inference decoding. Here, inference decoding utilizes at least one neural processing unit to generate a response to an input query, where heterogeneous language models cooperate with each other to generate the response to the input query.

[0080] Using a neural processing unit at a device end, the lower model can generate probabilities for sampling additional word pieces based on the inference generated additional word pieces and a current allowed set of word pieces. Using a neural processing unit at a device end, the upper model can generate word pieces based on the word pieces generated by the lower model. The upper model can accept or reject at least one word piece based on sampling the additional word pieces generated by the lower model. Here, the acceptance or rejection of the upper model can be determined by comparing probabilities generated by the upper model corresponding to each word piece with probabilities generated by the lower model corresponding to each word piece.

[0081] Inference decoding according to examples of the present disclosure, the lower model can generate one or more word piece candidate sets to generate a response corresponding to a query input to the lower model. Accordingly, the upper model can accept or reject at least one word piece candidate set by sampling each word piece candidate set. Here, if the system samples each word piece candidate set, the system can generate more word pieces faster than sampling each word piece. Further, the probability distribution of the lower and upper models can remain similar. Here, the word piece generation rate can be referred to as tokens per second (TPS).

[0082] According to examples of the present disclosure, in inference decoding of the system, the system can process the lower and upper models in parallel. In parallel processing inference decoding, the lower model can generate one or more word pieces while the upper model can sample and accept or reject the word pieces previously generated by the lower model.

[0083] According to examples of the present disclosure, speculative decoding can operate in a way of autoregressive token generation. Autoregressive token generation methods generate tokens by feeding a set of tokens generated by a lower model back to the lower model. The lower model generates one token at a time autoregressively, so if the lower model makes N inferences, it will generate a series of N tokens. The lower model generates a conditional probability distribution related to this series of N tokens. Then, the upper model processes the series of N tokens generated by the lower model and the conditional probability distribution related to this series of N tokens, thereby generating a probability distribution of the upper model. The upper model can then accept or reject the tokens generated by the lower model by comparing the probability distribution generated by the upper model with the conditional probability distribution generated by the lower model. Whether the upper model accepts the tokens can be determined by setting a threshold. The threshold can be a value such that when the similarity between the probability distribution value of the upper model and the conditional probability distribution value of the lower model is equal to or greater than the threshold, the token is accepted; when the similarity is less than the threshold, the token is rejected.

[0084] The upper model can accept or reject the series of tokens generated by the lower model according to the threshold. If a token in the series is rejected, the tokens before the rejection will become the final tokens.

[0085] In some examples, the lower model of speculative decoding can generate tokens in a set form. In this case, the upper model can accept or reject multiple tokens of each set generated by the lower model. When the lower model generates tokens in a set form, the acceptance or rejection judgment in the upper model can also be in the form of token sets, which can improve the speed of token generation of the system speculative decoding. In addition, as the number of token sets generated by the lower model increases, the probability of the token sets being accepted by the upper model also increases.

[0086] The lower model generates one or more token sets in response to the input query. Each token set contains a series of tokens, and the lower model selects a series of high-probability tokens from the probability distribution as the token set. The selection of each token set is various. For example, each token set can be generated by selecting the token with the highest probability in the probability distribution, but the present disclosure is not limited thereto, and the generation of token sets is various. In order for the upper model to accept the token set, the probabilities of the tokens in each token set can be added. Therefore, the upper model can accept or reject the token set based on the sum of the probabilities of the token set, rather than the probability of each single token. It should be noted that using the probability of each token will increase the probability of rejection by the upper model, as it can be difficult to match the probability of each token generated by the lower model and the upper model. Using the sum of the probabilities of the token set will increase the probability of acceptance by the upper model, as the sum of the probabilities of the token set generated by the lower model and the upper model can have a higher matching similarity.

[0087] The lower model can generate one or more sets of tokens in response to an input query. The one or more sets of tokens can be generated statically or dynamically. For example, a sequence length of each set of tokens can be preset. The lower model can generate one or more sets of tokens and combine the sets of tokens to generate tree data. The tree data can include nodes at each branch.

[0088] Figure 1 Tree data for speculative decoding is shown according to examples of the present disclosure.

[0089] Reference Figure 1 An example of tree data (TD) including one or more sets of tokens generated by a lower model is shown. More specifically, a first set of tokens (SI), a second set of tokens (S2), a third set of tokens (S3), a fourth set of tokens (S4), and a fifth set of tokens (S5) are shown in the figure.

[0090] The first set of tokens (SI) includes four tokens (A, B, C, D). The second set of tokens (S2) includes four tokens (A, B, C, E). The third set of tokens (S3) includes four tokens (A, F, G, H). The fourth set of tokens (S4) includes four tokens (A, F, G, I). The fifth set of tokens (S5) includes four tokens (A, F, J, K). Each set of tokens is a candidate set that can be selected or rejected by an upper model.

[0091] Each set of tokens can have a determined sequence length. For example, the sequence length of each set of tokens can be set to 4. However, the present disclosure is not limited thereto, and the length of the set of tokens can vary dynamically.

[0092] Each token can have a sequence level. The sequence level can correspond to the sequence length. More specifically, a first sequence level (LI) corresponds to the token (A). A second sequence level (L2) corresponds to the tokens (B, F). A third sequence level (L3) corresponds to the tokens (C, G, J). A fourth sequence level (L4) corresponds to the tokens (D, E, H, I, K). In the example where the sequence length is set to 4, the sequence level is also set to 4. The order of tokens of each set of tokens can be arranged according to the corresponding sequence level. Specifically, the first token (A) of the tokens (A, B, C, D) of the first set of tokens (SI) corresponds to the first sequence level (LI). The second token (B) of the tokens (A, B, C, D) of the first set of tokens (SI) corresponds to the second sequence level (L2). The third token (C) of the tokens (A, B, C, D) of the first set of tokens (SI) corresponds to the third sequence level (L3). The fourth token (D) of the tokens (A, B, C, D) of the first set of tokens (SI) corresponds to the fourth sequence level (L4).

[0093] Each wordpiece is defined as a node in the tree data (TD). For example, a wordpiece (G) can be a branch point of a third wordpiece set (S3) and a fourth wordpiece set (S4) on a third sequence level (L3). Each node contains branch information.

[0094] That is, the tree data (TD) generated by the lower model can include all or at least some of one or more wordpiece sets (S1 to S5), sequence levels (L1 to L4), node (A, B, C, D, E, F, G, H, I, J, K) information of each wordpiece, and selection probability (e.g., total probability of the wordpiece set) of each wordpiece set. The upper model can accept or reject at least one wordpiece in the tree data (TD).

[0095] Speculative decoding can also be implemented as hybrid speculative decoding. In hybrid speculative decoding, a wordpiece previously generated by speculation is verified by the upper model, and a threshold a can be set to determine whether the wordpiece is accepted or rejected by comparing the probability distribution of the lower model and the upper model.

[0096] Speculative decoding can also be implemented as group speculative decoding. Group speculative decoding can provide high efficiency in which a plurality of wordpieces are generated as a group from the lower model and configured to be verified by the upper model.

[0097] Figure 2 A speculative decoding operation according to the first example of the disclosure is schematically shown.

[0098] Referring to Figure 2 , a lower model (LM) and an upper model (UM) are shown. The device according to the embodiment of the disclosure includes one or more neural processing units. The one or more neural processing units can be configured to sequentially or in parallel process the lower model (LM) and the upper model (UM) for speculative decoding.

[0099] First, a typical Transformer-based neural network model includes a calculation step (e.g., a pre-padding step and a generation step) to generate a response to an input query. The Transformer neural network is a neural network based on an attention mechanism. It utilizes many matrix multiplication operations. The Transformer can accept input values and parameters such as a query (Q), a key (K), and a value (V) to obtain output values, i.e., attention (Q, K, V). Based on the output values (i.e., attention (Q, K, V)), the Transformer can perform various inference operations. The Transformer is widely used in language generation models. A tensor is a multi-dimensional matrix parameter processed in a neural network model. The tensor can refer to various parameters of the neural network model, such as weights, attention parameters, etc.

[0100] The prefill stage is an initial stage of processing an input (e.g., a prompt). The prefill step can include a tokenization step, a context embedding step, a processing through layers step, and a decoding preparation step. The prefill step can be processed in one or more neural processing units. In the tokenization step, the input text is converted into tokens (subwords, words, or characters) that are numerical representations used by the neural network model. In the context embedding step, the tokens are mapped to dense vector embeddings that encode semantic and syntactic information. In the processing through layers stage, the embeddings are processed through multiple layers (e.g., Transformer modules) of the neural network model to create a contextual representation of the input tokens. In the decoding preparation step, the neural network model computes probabilities of next tokens from the input context, but has not yet generated the next tokens. Instead, it sets up the context for the generation stage.

[0101] In the generation step, the neural network model sequentially generates one or more tokens from the context provided by the prefill step and previously generated tokens. The generation step includes an initial prediction step, an autoregressive processing step, and a stopping condition. The generation step can be processed in one or more neural processing units. In the initial prediction step, the first token is generated by sampling from a probability distribution computed in the prefill step. In the autoregressive processing step, each subsequent token is generated by incorporating the previous tokens (including the generated tokens) into the context of the neural network model. The generation process continues until the stopping condition is met, such as reaching a maximum token limit, encountering a special sequence end token (e.g., end of sequence (EOS)), or reaching a particular semantic goal. The series of tokens generated in the generation step can correspond to a tree data (TD) in Figure 1 The tree data can include at least one set of tokens.

[0102] Referring again to Figure 2 , an example of processing a speculative decoding is shown. The speculative decoding according to examples of the present disclosure is configured to be processed sequentially or in parallel by one or more neural processing units. The speculative decoding includes an upper model (UM) and a lower model (LM) that cooperate with each other to generate a response to an input query.

[0103] In particular, in the speculative decoding, the upper model (UM) first performs a pre-padding operation. Then, the lower model (LM) performs a pre-padding operation. Then, the lower model (LM) performs successive generation steps, in which each token is generated in each generation step. Next, the upper model (UM) accepts or rejects the tokens generated by the lower model (LM). The upper model (UM) can accept or reject some or all of the tokens generated by the lower model (LM). Then, the lower model (LM) can feed the accepted tokens into the input query, and the lower model (LM) can perform generation steps in turn, thereby generating each token corresponding to each generation step. Next, the upper model (UM) accepts or rejects the tokens generated by the lower model (LM). The above steps are repeated until a stop condition is met.

[0104] Reference is made to Figure 2 The number of parameters of the upper model (UM) is greater than the number of parameters of the lower model (LM), i.e., the number of parameters of the heterogeneous language model is different. Therefore, the processing time of the pre-padding step of the upper model (UM) is longer than the processing time of the pre-padding step of the lower model (LM). The processing time of the accepting or rejecting step of the upper model (UM) is longer than the processing time of the generation step of the lower model (LM). Therefore, the number of times of token generation in the lower model can improve the token generation speed of the system.

[0105] The lower model can utilize a language model in which the number of parameters is determined by considering the hardware resource constraints of the device end being processed. The upper model can utilize a language model in which the number of parameters is determined by considering the hardware resource constraints of the device being processed.

[0106] In some examples, the lower model can be a model that prunes the parameters of the upper model. For example, the lower model can be a model that prunes the parameters of the upper model. In this case, the parameters of the pruned neural network model can be lightweight and can require less memory bandwidth than the upper model.

[0107] One or more neural processing units according to examples of the present disclosure are configured to process the pre-padding operation by selecting an upper model (UM) to process the speculative decoding. Further, one or more neural processing units according to examples of the present disclosure are configured to process the pre-padding operation and the generation operation by selecting a lower model (LM) to process the speculative decoding.

[0108] Figure 3 A device according to a first implementation of the first example of the present disclosure is shown.

[0109] Reference is made to Figure 3The device 100 according to the first embodiment of the first example of the disclosure includes a neural processing unit (NPU) and a memory (MEM). The device 100 can further include at least one of a communication bus (not shown), a CPU (not shown), a GPU (not shown), and a storage device (not shown).

[0110] According to the first embodiment of the first example of the disclosure, the device 100 is configured to provide a super-large-scale AI service at the device end with low power consumption using speculative decoding.

[0111] According to the device 100 of the first embodiment of the first example of the disclosure, the AI service can be provided only at the device end, thereby protecting the user's personal information and privacy.

[0112] The device 100 according to the first embodiment of the first example of the disclosure is configured to receive an input query through an interface (IF). The interface (IF) can be an input device configured to convey an input query (e.g., a prompt, a gesture, a voice, etc.). The input device capable of conveying the input query can include, for example, a mouse, a keyboard, a touchpad, a touch screen, a microphone, a wireless communication device, a wired communication device, etc.

[0113] A communication bus (B) is provided between the neural processing unit (NPU) and the memory (MEM). In one example, the communication bus (B) can include a high-speed data bus, such as an advanced extensible interface (AXI). The communication bus (B) can be a dedicated communication bus for the neural processing unit (NPU) and the memory (MEM), and can be configured to provide a stable memory bandwidth for speculative decoding. The memory bandwidth can be determined according to the processing power of the neural processing unit (NPU), such as tera-operations per second (TOPS) of the neural processing unit. For example, 1 TOPS can require 2 GB / s to 4 GB / s of memory (MEM) bandwidth. For example, an NPU of 10 TOPS can configure the memory (MEM) to provide a memory (MEM) bandwidth of 20 GB / s to 40 GB / s. The memory bandwidth can be determined according to the operating frequency of the memory (MEM) and the number of communication channels. For example, to configure a memory bandwidth of 40 GB / s using LPDDR5, the following formula can be used: memory bandwidth (GB / s) = (data rate (Gbps / pin) x bus width (bits) x number of channels ÷ 8). In addition, it should be noted that LPDDR5 generally supports a data rate of up to 6,400 Mbps / pin. Furthermore, the bus width of the LPDDR channel is generally 16 bits (2 bytes). However, the number of channels cannot be a fraction, but must be an integer. Examples of the disclosure are not limited to the bandwidth of the memory bus.

[0114] The neural processing unit (NPU) of the device 100 according to the first embodiment of the first example of the present disclosure is an application specific integrated circuit (ASIC) implemented as an accelerator dedicated to neural network inference. As an example of the neural processing unit (NPU), its computing performance can be designed to be 5 TOPS, 10 TOPS, 25 TOPS, 50 TOPS, etc. The computing performance of the exemplary neural processing unit (NPU) can be designed to be 5 TOPS to 50 TOPS. The computing performance of the neural processing unit (NPU) can vary depending on the number of processing units contained in the processing core. The neural processing unit (NPU) can be referred to as a tensor processing unit (TPU), a neural signal processor (NSP), a neural network processor (NNP), etc.

[0115] The neural processing unit (NPU) of the device 100 according to the first embodiment of the first example of the present disclosure can be configured as a system on chip (SoC). The system on chip refers to a semiconductor package that integrates at least one neural processing unit (NPU) and various electronic system elements. The system on chip can integrate digital circuits, analog circuits, mixed signal and radio frequency processing circuits into a single package. For example, the processor that can also be contained in the system on chip can be at least one of a central processing unit (CPU), a digital signal processor (DSP), an image signal processor (ISP), and a graphics processing unit (GPU). The system on chip can contain at least one memory. The system on chip can contain a high-speed data bus for efficient communication between the plurality of circuits contained in the system on chip. The system on chip can contain at least one interface, such as PCIe, USB, I2C, SPI, UART, GPIO, for connecting external devices and sensors. The system on chip can include a system on chip power management device to regulate voltage and power distribution of the semiconductor package. The system on chip can include a communication interface integrated with wired and wireless communication protocols (e.g., Ethernet, Wi-Fi, Bluetooth, and cellular connectivity) to enable data transfer.

[0116] The neural processing unit (NPU) of the device 100 according to the first embodiment of the first example of the present disclosure is configured to infer an upper model (UM) and a lower model (LM). The upper model (UM) and the lower model (LM) are configured to cooperate with each other to perform speculative decoding. For a description of speculative decoding, see Figures 1-2 .

[0117] Here, the neural processing unit (NPU) is configured to process the upper model (UM) and the lower model (LM) alternately, i.e., the neural processing unit (NPU) operates to process the upper model (UM) and the lower model (LM) sequentially. In other words, the neural processing unit (NPU) operates to process the upper model (UM) and the lower model (LM) in a time-division manner. In this way, the upper model (UM) and the lower model (LM) can be processed sequentially while improving the utilization of the neural processing unit (NPU) computing circuit. Furthermore, since the upper model (UM) and the lower model (LM) can be processed sequentially in the neural processing unit (NPU) of the device 100, and the parameters of the upper model (UM) and the lower model (LM) reside in the memory (MEM), the operation of speculative decoding can reduce the bottleneck caused by the speculative decoding operation in the neural processing unit (NPU) of the device 100.

[0118] The memory (MEM) of the device 100 according to the first embodiment of the first example of the present disclosure is configured to store the weight parameters of the upper model (UM) and the weight parameters of the lower model (LM) when the device 100 is in a running state. By storing the weight parameters of the upper model (UM) and the lower model (LM) in the memory (MEM), the device 100 can efficiently process speculative decoding.

[0119] The memory (MEM) of the device 100 according to the first embodiment of the first example of the present disclosure is configured to provide the neural processing unit (NPU) with the parameters of the upper model (UM) when the neural processing unit (NPU) infers the upper model (UM), and provide the neural processing unit (NPU) with the parameters of the lower model (LM) when the neural processing unit (NPU) infers the lower model (LM).

[0120] For example, the device 100 can set the address mapping of the memory (MEM) to allocate memory banks for storing the weight parameters of the upper model (UM) and the lower model (LM) during the speculative decoding operation. Therefore, the weight parameters of the upper model (UM) and the lower model (LM) can both reside in the memory (MEM) during the speculative decoding operation.

[0121] The memory (MEM) of the device 100 according to the first embodiment of the first example of the present disclosure can include at least one memory, such as SRAM, LPDDR RAM, resistive RAM, magnetoresistive RAM, phase change RAM, ferroelectric RAM, flash memory, HBM, etc. The memory (MEM) can include at least one memory unit (e.g., memory bank, etc.). The MEM can include homogeneous memory or heterogeneous memory.

[0122] Accordingly, the capacity of the memory (MEM) of the device 100 according to the first embodiment of the first example of the present disclosure is configured to be greater than the sum of the weight parameters of the upper model (UM) and the lower model (LM). Accordingly, the weight parameters of the upper model (UM) and the lower model (LM) can reside in the memory MEM. If the capacity of the upper model (UM) and the lower model (LM) in the memory (MEM) of the device 100 is insufficient, it can be difficult to process the speculative decoding quickly in real time. For example, the memory (MEM) capacity of the device 100 can be one of 4 GB, 8 GB, 16 GB, or 32 GB, for example, the capacity of the memory (MEM) of the device 100 can be 4 GB to 32 GB, and can be determined by considering the parameter size of the upper model (UM) and the lower model (LM) to be run on the device 100.

[0123] If the parameters of the upper model (UM) and the lower model (LM) are not stored in the memory (MEM), the prior art device must read the weight parameters of each model from an external storage device to perform speculative decoding. Accordingly, the prior art device will command unnecessary memory read and write operations to perform speculative decoding. Therefore, the token generation rate of the conventional device will be slower than that of the device 100 according to the first embodiment of the first example of the present disclosure.

[0124] Power gating in a neural processing unit (NPU) is a technique to eliminate leakage by completely powering off unused circuit blocks. Circuit blocks that apply power gating are equipped with power switches. Accordingly, the power switches can be used to disconnect certain circuit blocks of the neural processing unit (NPU) from the power supply. The power gating function can eliminate both dynamic power consumption and leakage power consumption at the same time, which is useful when the device is in a low power mode.

[0125] Clock gating in a neural processing unit (NPU) is a technique to reduce dynamic power consumption by blocking clock signals from unused circuit blocks. Circuit blocks that apply clock gating can use clock control logic to selectively block clock signals input to a particular circuit block. When clock gating is applied, the neural processing unit (NPU) is still powered on, so the circuit state can be maintained. Since clock gating does not power off, operations can be resumed immediately after re-enabling the clock, and dynamic power consumption can be reduced by avoiding unnecessary switching activity.

[0126] Register retention in the Neural Processing Unit (NPU) is a technology designed to allow certain registers in the Neural Processing Unit (NPU) to retain their state even when the power is turned off. Registers using register retention can maintain the data stored in the registers using only low power consumption during sleep mode. Therefore, even if the registers enter the sleep state, important register values ​​are retained, eliminating the need for initialization on resume. Register retention can provide low-power operation, using extremely low power consumption to retain data even when the Neural Processing Unit (NPU) is powered off.

[0127] The device 100 according to the first embodiment of the first example of the present disclosure may be configured to provide a low power consumption function for use on the device side.

[0128] The low power consumption mode of the device 100 according to the first embodiment of the first example of the present disclosure may be, for example, a sleep mode, a deep sleep mode, a retention mode, etc. In some examples, the low power consumption mode may be referred to as a standby mode, a hibernation mode, an idle mode, etc.

[0129] For example, the sleep mode may be implemented by applying clock gating to a processing core (not shown) of the neural processing unit (NPU). For example, the deep sleep mode may be implemented by applying power gating to a processing core (not shown) or memory (MEM) of the neural processing unit (NPU). The retention mode may be implemented by applying a minimum retention voltage to a register (not shown) or memory (MEM) of the neural processing unit (NPU).

[0130] For example, the neural processing unit (NPU) of the device 100 of the first embodiment of the first example of the present disclosure may be configured to provide at least one of the following functions: power gating, clock gating, and register retention. As further described, when the device 100 enters sleep mode, the neural processing unit (NPU) may enable the clock gating function. In addition, when the device 100 enters deep sleep mode, the neural processing unit (NPU) may activate the power gating function. In addition, when the device 100 enters retention mode, the power supply voltage input to the neural processing unit (NPU) may be reduced or the operating frequency may be reduced.

[0131] For example, the memory (MEM) of the device 100, in accordance with the first embodiment of the first example of the present disclosure, can be configured to provide at least one of the following functions: deep sleep, retention, and maintenance parameters. Further, when the device 100 enters a deep sleep mode, the memory (MEM) can be power gated and, if a volatile memory device, the data can be lost. Further, when the device 100 enters a retention mode, the supply voltage of the memory (MEM) can be reduced or the operating frequency can be reduced; if a volatile memory device, the data can be retained. As further described, when the device 100 enters a parameter retention mode, the supply voltage of the memory (MEM) can be reduced or the operating frequency can be reduced; if a DRAM, the refresh cycle of the memory cells can be increased in order to retain the data in a volatile memory device.

[0132] The low power mode can be activated based on a particular threshold time and can be deactivated upon entry of an input query for speculative decoding in the device 100. The deactivation of the low power mode can be referred to as a wake-up state.

[0133] The device 100, in accordance with the first embodiment of the first example of the present disclosure, can be a device capable of operating in a battery environment. Accordingly, the neural processing unit (NPU) can be designed to operate using a battery voltage.

[0134] The device 100, in accordance with the first embodiment of the first example of the present disclosure, can be configured to provide at least one low power mode while providing speculative decoding in a device end environment. The device 100 can enter the low power mode in accordance with a pre-set condition, thereby reducing the power consumption of the device 100.

[0135] Figure 4 A device, in accordance with the second embodiment of the first example of the present disclosure, is shown.

[0136] In describing Figure 4 the application, the Figures 1-3 repetitive description of well-known functions and components can be omitted.

[0137] Figure 4 Referring to , the device 200, in accordance with the second embodiment of the first example of the present disclosure, includes a neural processing unit (NPU), a first memory (MEM1), and a second memory (MEM2). The device 200 can further include at least one of a communication bus (not shown), a CPU (not shown), a GPU (not shown), and a storage device (not shown).

[0138] According to a second embodiment of the first example of the disclosure, the device 200 is configured to provide an ultra-large-scale AI service at the device end with low power consumption using speculative decoding.

[0139] According to the device 200 of the second embodiment of the first example of the disclosure, the AI service can be provided only at the device end, thereby protecting the personal information and privacy of the user.

[0140] The device 200 according to the second embodiment of the first example of the disclosure is configured to receive an input query through an interface (IF).

[0141] A first communication bus (B1) is provided between the neural processing unit (NPU) and the first memory (MEM1). The first communication bus (B1) can be a dedicated communication bus of the neural processing unit (NPU) and the first memory (MEM1), and can be configured to provide a stable memory bandwidth for speculative decoding.

[0142] A second communication bus (B2) is provided between the neural processing unit (NPU) and the second memory (MEM2). The second communication bus (B2) can be a dedicated communication bus of the neural processing unit (NPU) and the second memory (MEM2), and can be configured to provide a stable memory bandwidth for speculative decoding.

[0143] The neural processing unit (NPU) of the device 200 according to the second embodiment of the first example of the disclosure is an application specific integrated circuit (ASIC) implemented as an accelerator dedicated to neural network inference.

[0144] The neural processing unit (NPU) of the device 200 according to the second embodiment of the first example of the disclosure can be configured as a system on chip (SoC).

[0145] The neural processing unit (NPU) of the device 200 according to the second embodiment of the first example of the disclosure is configured to infer an upper model (UM) and a lower model (LM). The upper model (UM) and the lower model (LM) are configured to cooperate with each other to process speculative decoding. For a description of speculative decoding, see Figures 1-2 .

[0146] Here, the neural processing unit (NPU) is configured to process the upper model (UM) and the lower model (LM) alternately, i.e., the neural processing unit (NPU) operates to process the upper model (UM) and the lower model (LM) in sequence. In other words, the neural processing unit (NPU) operates to process the upper model (UM) and the lower model (LM) in a time-division manner. Thus, the upper model (UM) and the lower model (LM) can be processed in sequence while improving the utilization of the computing circuit of the neural processing unit (NPU). Moreover, the speculative decoding operation does not cause the neural processing unit (NPU) of the device 200 to bottleneck, as the upper model (UM) and the lower model (LM) can be processed in sequence in the neural processing unit (NPU) of the device 200, and the parameters of the upper model (UM) and the parameters of the lower model (LM) reside in the first memory (MEM1) and the second memory (MEM2), respectively.

[0147] According to a second implementation of the first example of the present disclosure, the first memory (MEM1) of the device 200 is configured to store the weight parameters and the tokens of the upper model (UM) while the device 200 operates.

[0148] According to a second implementation of the first example of the present disclosure, the second memory (MEM2) of the device 200 is configured to store the weight parameters and the tokens of the lower model (LM) while the device 200 operates. By storing the weight parameters of the upper model (UM) and the lower model (LM) in the first memory (MEM1) and the second memory (MEM2), respectively, the device 200 can efficiently process the speculative decoding. As will be further described, the weight parameters of the upper model (UM) and the lower model (LM) can be very large and difficult to store on a single memory chip. In this case, the device 200 can configure multiple memories to increase the memory capacity.

[0149] According to a second implementation of the first example of the present disclosure, the first memory (MEM1) of the device 200 is configured to provide the parameters of the upper model (UM) to the neural processing unit (NPU) when the neural processing unit (NPU) infers the upper model (UM), while the second memory (MEM2) is configured to provide the parameters of the lower model (LM) to the neural processing unit (NPU) when the neural processing unit (NPU) infers the lower model (LM).

[0150] For example, the device 200 can set the address mapping of each of the first memory (MEM1) and the second memory (MEM2) to allocate a memory bank in which the weight parameters of the upper model (UM) and the lower model (LM) are stored during the speculative decoding operation. Thus, in the speculative decoding operation, the weight parameters of the upper model (UM) and the lower model (LM) can all reside in the first memory (MEM1) and the second memory (MEM2).

[0151] The first memory (MEM1) of the device 200 according to the second embodiment of the first example of the present disclosure can include at least one memory unit (e.g., a memory bank, etc.). The first memory (MEM1) can include a homogeneous memory or a heterogeneous memory.

[0152] The second memory (MEM2) of the device 200 according to the second embodiment of the first example of the present disclosure can include at least one memory unit (e.g., a memory bank, etc.). The second memory (MEM2) can include a homogeneous memory or a heterogeneous memory.

[0153] Thus, the capacity of the plurality of memories (MEM1, MEM2) of the device 200 according to the second embodiment of the first example of the present disclosure is configured to be greater than the sum of the weight parameters of the upper model (UM) and the lower model (LM). Thus, the weight parameters of the upper model (UM) and the lower model (LM) can reside in the plurality of memories (MEM1, MEM2). If the capacity of the upper model (UM) and the lower model (LM) in the plurality of memories (MEM1, MEM2) of the device 200 is insufficient, it can be difficult to efficiently process the speculative decoding.

[0154] If the parameters of the upper model (UM) and the lower model (LM) are not stored in the plurality of memories (MEM1, MEM2), the prior art device must read the weight parameters of each model from an external storage device to perform the speculative decoding. Thus, the prior art device can command unnecessary memory read and write operations to perform the speculative decoding. Thus, the token generation rate of the conventional device can be slower than the token generation rate of the device 200 according to the second embodiment of the first example of the present disclosure.

[0155] The device 200 according to the second embodiment of the first example of the present disclosure can be configured to provide a low power consumption function for use at the device end.

[0156] The low power consumption mode of the device 200 according to the second embodiment of the first example of the present disclosure can be, for example, a sleep mode, a deep sleep mode, a reserved mode, etc.

[0157] For example, the neural processing unit (NPU) of the device 200 according to the second embodiment of the first example of the present disclosure may be configured to provide at least one of the following functions: power gating, clock gating, and register retention.

[0158] For example, at least one of the first memory (MEM1) and the second memory (MEM2) of the device 200 according to the second embodiment of the first example of the present disclosure may be configured to provide at least one of the following functions: deep sleep, retention, and maintenance parameters.

[0159] When the device 200 does not receive any input queries for speculative decoding for a period of time, the low power mode can be activated. The specific amount of time can be set based on, for example, a timer, a counter value, etc. The low power mode can be activated based on a specific threshold time and can be deactivated when an input query for speculative decoding is input into the device 100.

[0160] According to a second embodiment of the first example of the present disclosure, the device 200 may be configured to provide at least one low power mode in a device-side environment while providing speculative decoding. The device 200 may enter the low power mode based on a preset condition, thereby reducing the power consumption of the device 200.

[0161] Figure 5 An apparatus according to a third embodiment of the first example of the present disclosure is shown.

[0162] In the description Figure 5 For ease of explanation, the Figures 1-4 Duplicate description.

[0163] See also Figure 5 The device 300 according to the third embodiment of the first example of the present disclosure includes multiple neural processing units (NPU1, NPU2) and multiple memories (MEM1, MEM2). The device 300 may also include at least one of a communication bus (not shown), a CPU (not shown), a GPU (not shown), and a storage device (not shown).

[0164] According to a third implementation of the first example of the present disclosure, the device 300 is configured to provide ultra-large-scale AI services on the device side with low power consumption using speculative decoding.

[0165] According to the device 300 of the third embodiment of the first example of the present disclosure, AI services can be provided only on the device side, thereby protecting the user's personal information and privacy.

[0166] The device 300 according to the third embodiment of the first example of the present disclosure is configured to receive an input query through an interface (IF).

[0167] A first communication bus (B1) is provided between the first neural processing unit (NPU1) and the first memory (MEM1). The first communication bus (B1) can be a dedicated communication bus for the first neural processing unit (NPU1) and the first memory (MEM1), and can be configured to provide a stable memory bandwidth for speculative decoding.

[0168] A second communication bus (B2) is provided between the second neural processing unit (NPU2) and the second memory (MEM2). The second communication bus (B2) can be a dedicated communication bus for the second neural processing unit (NPU2) and the second memory (MEM2), and can be configured to provide a stable memory bandwidth for speculative decoding.

[0169] A communication channel capable of communicating with each other can be provided between the first neural processing unit (NPU1) and the second neural processing unit (NPU2). A communication channel capable of communicating with each other can be provided between the first memory (MEM1) and the second memory (MEM2). Therefore, hints, pre-filled tokens, accepted tokens, rejected tokens, or tokens generated in speculative decoding can be transmitted.

[0170] The plurality of neural processing units (NPU1, NPU2) in the device 300 according to the third implementation of the first example of the present disclosure are application specific integrated circuits (ASICs) implemented as accelerators dedicated to neural network inference.

[0171] The plurality of neural processing units (NPU1, NPU2) in the device 300 according to the third implementation of the first example of the present disclosure can be configured as system on chips (SoC).

[0172] The device 300 according to the third implementation of the first example of the present disclosure, the first neural processing unit (NPU1) is configured to infer an upper model (UM), and the second neural processing unit (NPU2) is configured to infer a lower model (LM). The upper model (UM) and the lower model (LM) are configured to cooperate with each other to perform speculative decoding. For a description of speculative decoding, see Figures 1-2 .

[0173] Here, the first neural processing unit (NPU1) is configured to process the upper model (UM), and the second neural processing unit (NPU2) is configured to process the lower model (LM). In other words, the plurality of neural processing units (NPU1, NPU2) run in parallel to process the upper model (UM) and the lower model (LM). In other words, the plurality of neural processing units (NPU1, NPU2) run independently to process the upper model (UM) and the lower model (LM). Thus, the plurality of neural processing units (NPU1, NPU2) can cooperate with each other to efficiently process the upper model (UM) and the lower model (LM). Moreover, the speculative decoding operation does not cause a bottleneck of the plurality of neural processing units (NPU1, NPU2) of the device 200, as the upper model (UM) and the lower model (LM) can be processed sequentially in the plurality of neural processing units (NPU1, NPU2) of the device 200, and the parameters of the upper model (UM) and the lower model (LM) reside in the first memory (MEM1) and the second memory (MEM2), respectively.

[0174] According to a third implementation of the first example of the present disclosure, the first memory (MEM1) of the device 300 is configured to store the weight parameters and the tokens of the upper model (UM) when the device 300 is running.

[0175] According to a third implementation of the first example of the present disclosure, the second memory (MEM2) of the device 300 is configured to store the weight parameters and the tokens of the lower model (LM) when the device 300 is running. By storing the weight parameters of the upper model (UM) and the lower model (LM) in the first memory (MEM1) and the second memory (MEM2), respectively, the device 300 can efficiently process the speculative decoding. As will be further described, the weight parameters of the upper model (UM) and the lower model (LM) can be large and can be difficult to store on a single memory chip. In this case, the device 300 can configure multiple memories to increase the capacity of the memories.

[0176] According to a third implementation of the first example of the present disclosure, the first memory (MEM1) of the device 300 is configured to provide the parameters of the upper model (UM) to the first neural processing unit (NPU1) when the first neural processing unit (NPU1) infers the upper model (UM), and the second memory (MEM2) is configured to provide the parameters of the lower model (LM) to the second neural processing unit (NPU2) when the second neural processing unit (NPU2) infers the lower model (LM).

[0177] For example, the device 300 can set the address mapping of each of the first memory (MEM1) and the second memory (MEM2) to allocate memory banks in which the weight parameters of the upper model (UM) and the lower model (LM) are stored during the speculative decoding operation. Thus, in the speculative decoding operation, the weight parameters of the upper model (UM) and the lower model (LM) can reside in the first memory (MEM1) and the second memory (MEM2).

[0178] The first memory (MEM1) of the device 300 according to the third implementation of the first example of the present disclosure can include at least one memory unit (e.g., a memory bank, etc.). The first memory (MEM1) can include a homogeneous memory or a heterogeneous memory.

[0179] The second memory (MEM2) of the device 300 according to the third implementation of the first example of the present disclosure can include at least one memory unit (e.g., a memory bank, etc.). The second memory (MEM2) can include a homogeneous memory or a heterogeneous memory.

[0180] Thus, the capacity of the plurality of memories (MEM1, MEM2) of the device 300 according to the third implementation of the first example of the present disclosure is configured to be greater than the sum of the weight parameters of the upper model (UM) and the lower model (LM). Thus, the weight parameters of the upper model (UM) and the lower model (LM) can reside in the plurality of memories (MEM1, MEM2). If the capacity of the upper model (UM) and the lower model (LM) in the plurality of memories (MEM1, MEM2) of the device 300 is insufficient, it can be difficult to efficiently process the speculative decoding.

[0181] If the parameters of the upper model (UM) and the lower model (LM) are not stored in the plurality of memories (MEM1, MEM2), the prior art device must read the weight parameters of each model from an external storage device to perform the speculative decoding. Thus, the prior art device can command unnecessary memory read and write operations to perform the speculative decoding. Thus, the token generation rate of the conventional device can be slower than that of the device 300 according to the third implementation of the first example of the present disclosure.

[0182] The device 300 according to the third implementation of the first example of the present disclosure can be configured to provide a low power consumption function for use at the device end.

[0183] The low power consumption mode of the device 300 according to the third implementation of the first example of the present disclosure can be, for example, a sleep mode, a deep sleep mode, a reserved mode, etc.

[0184] For example, the neural processing unit (NPU) of the device 300 according to the third implementation of the first example of the present disclosure can be configured to provide at least one of the following functions: power gating, clock gating, and register retention.

[0185] For example, at least one of the first memory (MEM1) and the second memory (MEM2) of the device 300 according to the third implementation of the first example of the present disclosure can be configured to provide at least one of the following functions: deep sleep, retention, and maintenance of parameters.

[0186] The low-power mode can be activated based on a certain threshold time and can be deactivated when an input query for speculative decoding is entered in the device 100.

[0187] Referring again to Figure 2 When the first neural processing unit (NPU1) performs the pre-population operation or the accept or reject operation of the upper model (UM), the second neural processing unit (NPU2) and the second memory (MEM2) can be in a low-power mode. Conversely, when the second neural processing unit (NPU2) performs the pre-population operation or the generation operation of the lower model (LM), the first neural processing unit (NPU1) and the first memory (MEM1) can be in a low-power mode. However, the respective memories can hold data to perform the speculative decoding.

[0188] According to the third implementation of the first example of the present disclosure, the device 300 can be configured to provide at least one low-power mode in a device-end environment while providing the speculative decoding. The device 300 can enter the low-power mode based on a pre-set condition, and thus the power consumption of the device 300 can be reduced.

[0189] Figure 6 A system according to the first implementation of the second example of the present disclosure is shown.

[0190] In describing Figure 6 the application, specific terminology will be used. However, the Figures 1-5 application should not be construed as limited to the specific terminology so as long as it is in the spirit of the present disclosure.

[0191] Referring to Figure 6 , the system 400S including the device 100 and the server 400 according to the first implementation of the second example of the present disclosure is configured to process a federated operation between neural network models with the speculative decoding.

[0192] Device 100 may be one of the devices (100, 200, 300) of the first example of the present disclosure. Device 100 includes a first neural processing unit (NPU1) and a first memory (MEM1). Device 100 may also include at least one of a communication bus (not shown), a CPU (not shown), a GPU (not shown), and a storage device (not shown).

[0193] Server 400 is a local server. The computing power of server 400 can be designed to reach, for example, 100 TOPS, 250 TOPS, 5,000 TOPS, or 10,000 TOPS. For example, the computing power of the neural processing unit (NPU) can be designed to reach 100 TOPS to 10,000 TOPS. Unlike cloud servers, local servers are located in the data center, office, or dedicated server room of an organization, enterprise, or institution. This allows direct control over hardware, software, security settings, and more.

[0194] The server 400 may include at least one neural processing unit (NPU) and elements of various electronic systems. The server 400 may include digital circuits, analog circuits, mixed signal and radio frequency processing circuits. For example, the processor that may also be included in the server 400 may be at least one of a central processing unit (CPU), a digital signal processor (DSP), an image signal processor (ISP) and a graphics processing unit (GPU). The server 400 may include at least one memory. The server 400 may include a high-speed data bus for efficient communication between various circuit components. The server 400 may include at least one interface for connecting to external devices and sensors, such as PCIe, USB, I2C, SPI, UART, GPIO. The server 400 may include an on-chip power management unit to adjust the voltage and power distribution of the server 400. The server 400 may include a communication interface that integrates wired and wireless communication protocols (such as Ethernet, Wi-Fi, Bluetooth and cellular connections) to achieve data transmission.

[0195] The server 400 includes a second neural processing unit (NPU2) and a second memory (MEM2). The server 400 may further include at least one of a communication bus (not shown), a CPU (not shown), a GPU (not shown), and a storage device (not shown).

[0196] According to a first implementation of the second example of the present disclosure, the system 400S is configured to provide ultra-large-scale AI services on the device side with low power consumption using speculative decoding.

[0197] According to the system 400S of the first embodiment of the second example of the present disclosure, the AI ​​service can be provided only on the local server, thereby protecting the user's personal information and privacy.

[0198] The system 400S according to the first implementation of the second example of the present disclosure is configured to receive an input query through an interface (IF).

[0199] A first communication bus (B1) is provided between the first neural processing unit (NPU1) and the first memory (MEM1). The first communication bus (B1) can be a dedicated communication bus for the first neural processing unit (NPU1) and the first memory (MEM1), and can be configured to provide a stable memory bandwidth for speculative decoding.

[0200] A second communication bus (B2) is provided between the second neural processing unit (NPU2) and the second memory (MEM2). The second communication bus (B2) can be a dedicated communication bus for the second neural processing unit (NPU2) and the second memory (MEM2), and can be configured to provide a stable memory bandwidth for speculative decoding.

[0201] The plurality of neural processing units (NPU1, NPU2) in the system 400S according to the first implementation of the second example of the present disclosure are application specific integrated circuits (ASICs) implemented as accelerators dedicated to neural network inference.

[0202] The system 400S according to the first implementation of the second example of the present disclosure is configured to receive an input query through an interface (IF). The input query can be first transmitted to the server 400 to initiate speculative decoding. However, the first implementation of the second example of the present disclosure is not limited thereto, and the device 100 can also perform speculative decoding only on the device 100. In this case, the accuracy of the answer generated by the device 100 can be reduced, but the answer can be generated quickly. Alternatively, even if the communication with the server 400 is interrupted, the device 100 can generate an answer.

[0203] According to the first implementation of the second example of the present disclosure, the device 100 of the system 400S can be configured as a system on chip (SoC), and the server 400 can be configured as a local server.

[0204] A communication channel can be provided between the device 100 and the server 400 to allow them to communicate with each other. Thus, the prompt, the pre-filled token, the accepted token, the rejected token, or the token generated by the speculative decoding can be transmitted.

[0205] According to the system 400S of the first implementation of the second example of the present disclosure, the first neural processing unit (NPU1) is configured to infer an upper model (UM), and the second neural processing unit (NPU2) is configured to infer a lower model (LM). The upper model (UM) and the lower model (LM) are configured to cooperate with each other to perform speculative decoding. For a description of speculative decoding, seeFigures 1-2 .

[0206] Here, the first neural processing unit (NPU1) is set to process the upper model (UM), and the second neural processing unit (NPU2) is set to process the lower model (LM). In other words, multiple neural processing units (NPU1, NPU2) run in parallel to process the upper model (UM) and the lower model (LM). In other words, multiple neural processing units (NPU1, NPU2) run independently to process the upper model (UM) and the lower model (LM). Therefore, multiple neural processing units (NPU1, NPU2) can cooperate with each other to efficiently process the upper model (UM) and the lower model (LM). In addition, the speculative decoding operation does not cause a bottleneck in the multiple neural processing units (NPU1, NPU2) of the system 400S, because the upper model (UM) and the lower model (LM) can be processed sequentially in the multiple neural processing units (NPU1, NPU2) of the system 400S, and the parameters of the upper model (UM) and the lower model (LM) reside in the first memory (MEM1) and the second memory (MEM2), respectively.

[0207] According to the first implementation of the second example of the present disclosure, the first memory (MEM1) of the device 100 is configured to store weight parameters and word-grams of the lower layer model (LM) when the device 100 is running.

[0208] According to a first embodiment of the second example of the present disclosure, the second memory (MEM2) of the service 400 is configured to store weight parameters and word units of the upper model (UM) when the server 400 is running. By storing the weight parameters of the upper model (UM) and the lower model (LM) in the first memory (MEM1) and the second memory (MEM2), respectively, the system 400S can efficiently process speculative decoding. As will be further described, the weight parameters of the upper model (UM) and the lower model (LM) may be large and may be difficult to store on a single memory chip. In this case, the system 400S can be configured with multiple memories to increase the capacity of the memory.

[0209] According to the first embodiment of the second example of the present disclosure, the first memory (MEM1) of the system 400S is configured to provide parameters of the lower-layer model (LM) to the first neural processing unit (NPU1) when the first neural processing unit (NPU1) infers the lower-layer model (LM), and the second memory (MEM2) is configured to provide parameters of the upper-layer model (UM) to the second neural processing unit (NPU2) when the second neural processing unit (NPU2) infers the upper-layer model (UM).

[0210] For example, the system 400S can set the address mapping of each of the first memory (MEM1) and the second memory (MEM2) to allocate memory banks in which the weight parameters of the upper model (UM) and the lower model (LM) are stored during the speculative decoding operation. Thus, in the speculative decoding operation, the weight parameters of the upper model (UM) and the lower model (LM) can reside in the first memory (MEM1) and the second memory (MEM2).

[0211] The first memory (MEM1) of the system 400S according to the first implementation of the second example of the present disclosure can include at least one memory unit (e.g., a memory bank, etc.). The first memory (MEM1) can include homogeneous memory or heterogeneous memory.

[0212] The second memory (MEM2) of the system 400S according to the first implementation of the second example of the present disclosure can include at least one memory unit (e.g., a memory bank, etc.). The second memory (MEM2) can include homogeneous memory or heterogeneous memory.

[0213] Thus, the capacity of the plurality of memories (MEM1, MEM2) of the system 400S according to the first implementation of the second example of the present disclosure is configured to be greater than the sum of the weight parameters of the upper model (UM) and the lower model (LM). Thus, the weight parameters of the upper model (UM) and the lower model (LM) can reside in the plurality of memories (MEM1, MEM2). If the capacity of the upper model (UM) and the lower model (LM) in the plurality of memories (MEM1, MEM2) of the system 400S is insufficient, it can be difficult to efficiently process the speculative decoding.

[0214] If the parameters of the upper model (UM) and the lower model (LM) are not stored in the plurality of memories (MEM1, MEM2), the prior art device must read the weight parameters of each model from an external storage device to perform the speculative decoding. Thus, the prior art device can command unnecessary memory read and write operations to perform the speculative decoding. Thus, the token generation rate of the conventional device can be slower than the token generation rate of the system 400S according to the first implementation of the second example of the present disclosure.

[0215] The first device 100 according to the first implementation of the second example of the present disclosure can be configured to provide a low-power consumption function for use at the device end.

[0216] The low-power consumption mode of the system 400S according to the first implementation of the second example of the present disclosure can be, for example, a sleep mode, a deep sleep mode, a hold mode, etc.

[0217] For example, the plurality of neural processing units (NPU1, NPU2) of the system 400S according to the first implementation of the second example of the present disclosure can be configured to provide at least one of the following functions: power gating, clock gating, and register retention.

[0218] For example, the plurality of memories (MEM1, MEM2) of the system 400S according to the first implementation of the second example of the present disclosure can be configured to provide at least one of the following functions: deep sleep, retention, and maintenance of parameters.

[0219] The low-power mode can be activated based on a certain threshold time and can be deactivated upon input of an input query for speculative decoding in the device 100.

[0220] Referring again to Figure 2 When the second neural processing unit (NPU2) performs a pre-population operation or an accept or reject operation of the upper model (UM), the first neural processing unit (NPU1) and the first memory (MEM1) can be in a low-power mode. Conversely, when the first neural processing unit (NPU1) performs a pre-population operation or a generation operation of the lower model (LM), the second neural processing unit (NPU2) and the second memory (MEM2) can be in a low-power mode. However, the respective memories can hold data to perform speculative decoding.

[0221] The system 400S according to the first implementation of the second example of the present disclosure can be configured to provide at least one low-power mode in a local server environment while providing speculative decoding. The system 400S can enter a low-power mode according to a pre-set condition, and thus power consumption of the system 400S can be reduced.

[0222] Figure 7 A system according to the second implementation of the second example of the present disclosure is illustrated.

[0223] In describing Figure 7 the second implementation of the second example of the present disclosure, repetitive descriptions will be omitted for ease of explanation. Figures 1-6

[0224] Referring to Figure 7 , the system 410S according to the second implementation of the second example of the present disclosure includes a plurality of devices 100-1, 100-2 and a server 400. The system 410S including the plurality of devices 100-1, 100-2 and the server 400 is configured to process federated operations between neural network models with speculative decoding.

[0225] ​Each of the plurality of devices 100 - 1 , 100 - 2 may be substantially the same as one of the devices described in the first example or the second example of the present disclosure.

[0226] Server 400 is a local server. Server 400 includes at least one neural processing unit (not shown) and at least one memory (not shown). The number of neural processing units and memories included in server 400 may vary depending on the number of multiple devices associated therewith. Server 400 may also include at least one of a communication bus (not shown), a CPU (not shown), a GPU (not shown), and a storage device (not shown).

[0227] According to a second embodiment of the second example of the present disclosure, the system 410S is configured to utilize speculative decoding technology to provide ultra-large-scale AI services with low power consumption by means of multiple device terminals.

[0228] According to the system 410S of the second embodiment of the second example of the present disclosure, the AI ​​service may be provided only on a local server, thereby protecting the user's personal information and privacy.

[0229] The system 410S according to the second embodiment of the second example of the present disclosure is configured to receive at least one input query through multiple interfaces (IF1, IF2). The input query can first be sent to the server 400 to start speculative decoding. However, the second embodiment of the second example of the present disclosure is not limited thereto, and speculative decoding can also be processed by at least one device in the multiple devices 100-1, 100-2. In this case, the accuracy of the answer generated by the device may be reduced, but the answer can be generated quickly. Alternatively, even if communication with the server 400 is lost, the answer can be generated on the device.

[0230] According to an embodiment of a second example of the present disclosure, each of the plurality of devices 100 - 1 , 100 - 2 of the system 410S may be configured as a system on a chip (SoC), and the server 400 may be configured as a local server.

[0231] According to a second exemplary embodiment of the present disclosure, each of the plurality of devices 100-1, 100-2 of the system 410S is configured to infer a first lower layer model (LM1) and a second lower layer model (LM2), respectively, and the server 400 is configured to infer an upper layer model (UM). The upper layer model (UM) and the first lower layer model (LM1) and the second lower layer model 1 (LM1, LM2) are configured to collaboratively process speculative decoding. For a description of speculative decoding, see Figure 1 、 Figure 2 and Figure 8 .

[0232] Here, the first device 100-1 is configured to process the first lower model (LM1), the second device 100-2 is configured to process the second lower model (LM2), and the server 400 is configured to process the upper model (UM). That is, the devices 100-1 and 100-2 each process a corresponding lower model (LM1, LM2), and the server 400 processes an upper model (UM) corresponding to the plurality of lower models (LM1, LM2). Accordingly, the server 400 can efficiently process the plurality of lower models (LM). This will be described in more detail with reference to FIGS. 4A and 4B. Figure 9 This will be described in more detail.

[0233] The memory (not shown) of the first device 100-1 according to the second embodiment of the second example of the present disclosure is configured to store weight parameters and tokens of the first lower model (LM1) while the first device 100-1 is running.

[0234] The memory (not shown) of the second device 100-2 according to the second embodiment of the second example of the present disclosure is configured to store weight parameters and tokens of the second lower model (LM2) while the second device 100-2 is running.

[0235] The memory (not shown) of the server 400 according to the second embodiment of the second example of the present disclosure is configured to store weight parameters and tokens of the upper model (UM) while the server 400 is running.

[0236] According to the second embodiment of the second example of the present disclosure, the plurality of devices 100-1, 100-2 can each be configured to provide a low-power consumption function for use at the device end.

[0237] The low-power consumption mode of the plurality of devices 100-1, 100-2 according to the embodiment of the second example of the present disclosure can provide, for example, a sleep mode, a deep sleep mode, a retention mode, etc.

[0238] For example, the neural processing unit (not shown) of the first device 100-1 or the second device 100-2 according to the second embodiment of the second example of the present disclosure can be configured to provide at least one of the following functions: power gating, clock gating, and register retention.

[0239] For example, the memory (not shown) of the first device 100-1 or the second device 100-2 according to the second embodiment of the second example of the present disclosure can be configured to provide at least one of the following functions: deep sleep, retention, and maintenance parameters.

[0240] For example, at least one neural processing unit (not shown) of the server 400 according to the second embodiment of the second example of the present disclosure can be configured to provide at least one of the following functions: power gating, clock gating, and register retention.

[0241] For example, at least one memory (not shown) of the server 400 according to the second embodiment of the second example of the present disclosure can be configured to provide at least one of the following functions: deep sleep, retention, and maintenance of parameters.

[0242] The low-power mode can be activated based on a certain threshold time and can be deactivated upon inputting an input query for speculative decoding into the system 410S.

[0243] Referring again to Figure 2 and Figure 8 When the server 400 performs a pre-population operation or an accept or reject operation of an upper model (UM), the neural processing units (not shown) and the memories (not shown) of the plurality of devices 100-1 and 100-2 can be in a low-power mode. In contrast, when the first device 100-1 performs a pre-population operation or a generation operation of a first lower model (LM1), the second device 100-2 can be in a low-power mode.

[0244] The system 410S according to the second embodiment of the second example of the present disclosure can be configured to provide at least one low-power mode in a local server environment while providing speculative decoding. The system 410S according to the second embodiment of the second example of the present disclosure can efficiently process speculative decoding through the plurality of devices 100-1, 100-2, and the server 400, and each of the devices 100-1, 100-2 can independently activate (e.g., in a wake-up mode) or deactivate (e.g., in a low-power mode).

[0245] Figure 8 A speculative decoding operation according to the second example of the present disclosure is schematically illustrated.

[0246] In describing Figure 8 the present disclosure, repetitive descriptions can be omitted for the sake of explanation. Figure 2

[0247] Referring to Figure 7 and Figure 8 ​, showing a plurality of lower models (LM1, LM2) and one upper model (UM). The system 410S according to the second embodiment of the disclosure comprises a plurality of devices (100-1, 100-2) and a server 400. Each of the devices (100-1, 100-2) is associated with a respective interface (IF1, IF2). The system 410S can be configured to sequentially or in parallel process a plurality of lower models (LM1, LM2) and an upper model (UM) for speculative decoding.

[0248] The upper model (LM) of the server 400 sequentially receives each input query through the plurality of devices (100-1, 100-2). Next, the server 400 sequentially processes a first speculative decoding and a second speculative decoding. Referring to Figure 8 , the hatched boxes correspond to the operations of the first speculative decoding, and the boxes without dots correspond to the operations of the second speculative decoding.

[0249] For the convenience of the following description, reference will be made to the points (T1 to T8) on the time axis.

[0250] The server 400 processes the pre-filling operations corresponding to the first speculative decoding with the upper model (UM) and transmits the results of the pre-filling operations corresponding to the first speculative decoding to the first device 100-1 (T1). Next, the server 400 processes the pre-filling operations corresponding to the second speculative decoding with the upper model (UM) and transmits the results of the pre-filling operations corresponding to the second speculative decoding to the second device 100-2 (T2). At this time, the first device 100-1 receives the results of the pre-filling operations of the upper model (UM) corresponding to the first speculative decoding (T1) and processes the pre-filling operations and a series of generation operations corresponding to the first speculative decoding with the first lower model (LM1) to generate a series of wordpiece candidates and transmits them to the server 400 (T3). The server 400 receives the candidate wordpieces generated by the first device 100-1 corresponding to the first speculative decoding (T3) and accepts or rejects the candidate wordpieces corresponding to the first speculative decoding with the upper model (UM) (T4). Meanwhile, the second device 100-2 receives the results of the pre-filling operations of the upper model (UM) corresponding to the second speculative decoding (T2) and processes the pre-filling operations and a series of generation operations corresponding to the second speculative decoding with the second lower model (LM2) to generate a series of candidate wordpieces and transmits them to the server 400 (T5).

[0251] Meanwhile, the first device 100-1 receives an acceptance or rejection operation result (T4) of the upper model (UM) corresponding to the first-time speculative decoding and processes a series of generation operations corresponding to the first-time speculative decoding using the first lower model (LM1) to generate a series of additional token candidates and transmits them to the server 400 (T6). At this time, the server 400 receives the candidate tokens generated by the second device 100-2 corresponding to the second-time speculative decoding (T5) and accepts or rejects the candidate tokens corresponding to the second-time speculative decoding using the upper model (UM) (T7). At this time, a delay in the operation corresponding to the first-time speculative decoding can occur in the server 400. Specifically, the acceptance or rejection operation corresponding to the second-time speculative decoding can be delayed from the start time (T5) to the end time (T7). In other words, when the first device 100-1 transmits a series of token candidates generated using the first lower model (LM1) to the server 400, the operation of the corresponding upper model (UM) can be delayed from time (T6) to time (T7).

[0252] Next, the server 400 transmits an acceptance or rejection result of the token candidates corresponding to the second-time speculative decoding to the second device 100-2 (T7). At this time, the second device 100-2 receives the acceptance or rejection result of the upper model (UM) corresponding to the second-time speculative decoding (T7) and processes a series of generation operations corresponding to the second-time speculative decoding using the second lower model (LM2) to generate a series of token candidates and transmits them to the server 400 (T9). Then, the server 400 processes the acceptance or delayed operation corresponding to the first-time speculative decoding using the upper model (UM) from a time point (T7) after a time point (T6) at which the series of token candidates generated by the first device 100-1 as the first lower model (LM1) is received and transmits an acceptance or rejection result to the first device 100-1 (T8).

[0253] Next, the first device 100-1 receives an acceptance or rejection result of the upper model (UM) corresponding to the first-time speculative decoding (T8) and processes a series of generation operations corresponding to the first-time speculative decoding using the first lower model (LM1) to generate a set of token candidates and transmits them to the server 400 (T10).

[0254] The server 400 receives the candidate tokens generated by the second device 100-2 corresponding to the second speculative decoding (T9) and accepts or rejects the candidate tokens corresponding to the second speculative decoding using the upper model (UM) (T10). Next, the server 400 receives the candidate tokens generated by the first device 100-1 corresponding to the first speculative decoding (T10) and accepts or rejects the candidate tokens corresponding to the first speculative decoding using the upper model (UM) (T11).

[0255] Repeat the above steps until the stopping condition is met.

[0256] According to the system 410S of the second embodiment of the second example of the present disclosure, the system 410S can efficiently process speculative decoding using multiple devices 100-1, 100-2 and the server 400. In addition, the server 400 can efficiently cooperate with the multiple devices 100-1, 100-2 in a batch manner to perform speculative decoding.

[0257] refer to Figure 8 , the number of parameters of the upper model (UM) is greater than the number of parameters of the first lower model (LM1) or the second lower model (LM2), that is, the number of parameters of the heterogeneous language models is different. Therefore, the processing time of the pre-filling step of the upper model (UM) is longer than that of the pre-filling step of the lower model (LM). The processing time of the acceptance or rejection step of the upper model (UM) is longer than that of the generation step of the lower model (LM). Therefore, more word unit generation times in the lower model can improve the word unit generation speed of the system.

[0258] The lower-layer model can use a language model with a certain number of parameters, which takes into account the hardware resource limitations of the device being processed. The upper-layer model can use a language model with a certain number of parameters, which takes into account the hardware resource limitations of the device being processed.

[0259] In some examples, the lower-level model can be a pruned version of the parameters of the upper-level model. For example, the lower-level model can be a pruned version of the parameters of the upper-level model. In this case, the parameters of the pruned neural network model may be lighter and require less memory bandwidth than the upper-level model.

[0260] Figure 9 A system according to a third embodiment of the present disclosure is shown.

[0261] In the description Figure 9 For ease of explanation, the Figures 1-8 Duplicate description.

[0262] refer to Figure 9 , a system 500S according to a third example of the present disclosure includes a plurality of devices 100-1, 100-2, 100-3, 100-4, a plurality of local servers 400-1, 400-2, and a cloud server 500. The system 500S including the plurality of devices 100-1, 100-2, 100-3, 100-4, the plurality of local servers 400-1, 400-2, and the cloud server 500 is configured to process federated operations between neural network models using speculative decoding.

[0263] Each of the plurality of devices 100-1, 100-2, 100-3, 100-4 can be substantially the same as one of the devices described in the first or second examples of the present disclosure.

[0264] Each of the plurality of local servers 400-1, 400-2 can be substantially the same as the server described in the second example of the present disclosure.

[0265] The cloud server 500 includes at least one neural processing unit (not shown) and at least one memory (not shown). The number of neural processing units and memories included in the cloud server 500 can vary depending on the number of local servers associated with the cloud server 500. The cloud server 500 can also include at least one of a communication bus (not shown), a CPU (not shown), a GPU (not shown), and a storage device (not shown). The cloud server 500 can be configured to communicate with the plurality of local servers 400-1, 400-2 through a wired or wireless communication network. The computing power of the cloud server 500 can be designed to be, for example, 10,000 TOPS to 100,000 TOPS.

[0266] According to the third example of the present disclosure, the system 500S is configured to provide a super-large-scale AI service with low power consumption at the plurality of device ends using speculative decoding.

[0267] The system 500S according to the third example of the present disclosure is configured to receive at least one input query through the plurality of interfaces (IF1, IF2, IF3, IF4). The input query can be first transmitted to the corresponding local server to initiate speculative decoding. However, the third example of the present disclosure is not limited thereto, and at least one of the plurality of devices 100-1, 100-2, 100-3, 100-4 can also process speculative decoding only at the device end. In this case, the accuracy of the answer generated by the device can be reduced, but the answer can be quickly generated. Alternatively, even if the communication of the device end with the corresponding local server is interrupted, the device end can generate an answer. However, the third example of the present disclosure is not limited thereto, and at least one of the plurality of devices 100-1, 100-2, 100-3, 100-4 can also process speculative decoding only up to the local server, that is, the cloud server 500 can be selectively excluded. In this case, the accuracy of the answer generated by the device can be reduced, but the answer can be quickly generated. Alternatively, even if the communication with the cloud server 500 is interrupted, the local server can generate an answer.

[0268] Each of the plurality of devices 100-1, 100-2, 100-3, 100-4 of the system 500S according to the third example of the present disclosure can be configured as a system on chip (SoC).

[0269] According to the third example of the present disclosure, each of the plurality of devices 100-1, 100-2, 100-3, 100-4 of the system 500S is configured to infer a respective lower model, and the plurality of local servers 400-1, 400-2 is configured to infer an intermediate model. The intermediate model can refer to a model having more parameters than the lower model and having fewer parameters than the upper model. The plurality of lower models, the plurality of intermediate models, and the upper model are configured to cooperate with each other to perform the speculative decoding. To generate the high-level answer, the system 500S can be configured to utilize the lower models to accept or reject the wordpiece from the lower models.

[0270] The memory (not shown) of each device according to the third example of the present disclosure is configured to store the weight parameters and the wordpiece of the respective lower model of the respective device while the respective device is running.

[0271] The memory (not shown) of each local server according to the third example of the present disclosure is configured to store the weight parameters and the wordpiece of the respective intermediate model while the respective local server is running.

[0272] The memory (not shown) of the cloud server 500 according to the third example of the present disclosure is configured to store the weight parameters and the wordpiece of the upper model while the cloud server 500 is running.

[0273] The plurality of devices 100-1, 100-2, 100-3, 100-4 according to the third example of the present disclosure can be configured to provide a low-power consumption function for use by their respective device-side application programs.

[0274] The low-power consumption mode of the plurality of devices 100-1, 100-2, 100-3, 100-4 according to the third example of the present disclosure can provide, for example, a sleep mode, a deep sleep mode, a reserved mode, and the like.

[0275] The system 500S according to the third example of the present disclosure can be configured to provide at least one low-power consumption mode while providing the speculative decoding in the local server environment and / or the cloud server environment.

[0276] Figure 10 A neural processing unit applicable to the examples of the present disclosure is shown.

[0277] The neural processing unit 1000 suitable for the examples of the present disclosure can include a DMA 1100, an NPU memory 1200, a processing element core 1300, a vector core 1400, a scalar core 1500, a digital format conversion unit 1600, and an NPU controller 1700.

[0278] The neural processing unit 1000 is a processor independent of processors such as a CPU or GPU, which is specifically designed for deep neural network (DNN) operations. Specifically, the neural processing unit 1000 may include multiple processing elements dedicated to matrix multiplication, which accounts for the majority of computation in DNNs.

[0279] The NPU controller 1700 may be electrically coupled to the DMA 1100, the NPU memory 1200, the processing element core 1300, the vector core 1400, the scalar core 1500, and the digital format conversion unit 1600. The NPU controller 1700 may be configured to control operations related to speculative decoding operations in the neural processing unit 1000. The NPU controller 1700 may be configured to perform speculative decoding using execution code generated by a compiler (not shown). Here, the compiler (not shown) may be a semiconductor circuit or software running on a separate device. The compiler (not shown) may convert a specific neural network model (e.g., an upper layer model (UM) or a lower layer model (LM)) into executable code that can be executed on the neural processing unit 1000. In other words, the compiler (not shown) may generate NPU-specific executable code that can be executed on a specific neural processing unit with specific capabilities. Execution code may also be referred to as machine code or binary code. The execution code may include multiple execution codes corresponding to the upper layer model (UM), such as a first execution code and a second execution code corresponding to the lower layer model (LM). Thus, the NPU controller 1700 can be configured to control the DMA 1100, NPU memory 1200, processing element core 1300, vector core 1400, scalar core 1500, and digital format conversion unit 1600, etc., based on various instructions and operation schedules contained in the execution code. The execution code can be programmed to control the neural processing unit 1000 according to the length of the word element of the response being generated.

[0280] For example, the compiler can generate execution code that determines the data read / write order according to speculative decoding, the processing order of neural network operations, etc. based on the hardware characteristics of the first neural processing unit (NPU1) (for example, the computing performance of the processing element core, the computing performance of the vector core, the computing performance of the scalar core, the running time of the digital format conversion unit, the capacity of the NPU memory, and the memory bandwidth of the corresponding bus).

[0281] DMA 1100 is configured to allow NPU 1000 to directly access and read / write memory electrically coupled thereto (e.g., at least Figures 3-5). The neural processing unit 1000 can transfer various data related to speculative decoding from a memory (not shown) via the DMA 1100. The DMA 1100 can be configured to perform tasks such as setting the address of the NPU memory 1200, generating and controlling read / write instructions, etc.

[0282] The NPU memory 1200 is a memory located in the on-chip area of ​​the neural processing unit 1000, and may include a memory for caching or storing data processed in the on-chip area, or a register file for storing instructions for driving the compiled neural network model. Here, the register file may be configured to be included in the NPU controller 1700. The NPU memory 1200 can read and store data required for speculative decoding operations from a memory (not shown) associated with the bus. The NPU memory 1200 may include, for example, one of SRAM, register files, DRAM, resistive RAM, magnetoresistive RAM, phase change RAM, ferroelectric RAM, flash memory, and the like. The NPU memory 1200 may include at least one storage unit. The NPU memory 1200 may include homogeneous storage units or heterogeneous storage units. The capacity of the NPU memory 1200 may be, for example, several megabytes to tens of megabytes, but is not limited thereto.

[0283] The processing element core 1300, the vector core 1400 and the scalar core 1500 are computing circuits configured to efficiently process the attention layer of the upper model (UM) or the lower model (LM). The attention layer can use input values ​​and parameters (such as query (Q), key (K) and value (V)) to obtain operation values, i.e., attention values ​​(Q, K, V). The processing element core 1300, the vector core 1400 and the scalar core 1500 are designed to process operations such as Matmul, DynamicQuant, RMSNorm and Softmax for calculating attention. The data form of the above operations can be one of a multi-dimensional matrix form, a two-dimensional matrix form, a one-dimensional array form or a scalar, and one of the processing element core 1300, the vector core 1400 and the scalar core 1500 can be selected according to the data form to process these operations. Therefore, the neural processing unit 1000 can accelerate the calculation of the generative artificial intelligence model quickly and with low power consumption.

[0284] The processing element core 1300 includes N x M multipliers. N and M are integers greater than 1. The processing element core 1300 in a two-dimensional array can be configured to process multiple processing elements (PEs) in parallel to perform matrix multiplication operations on input data and their corresponding parameters (e.g., weights, kernels, queries (Q), keys (K), values (V), etc.). The processing elements (PEs) can include multiply and accumulate (MAC) multipliers. The processing element core 1300 can efficiently process matrix multiplication (e.g., Matmul or GEMM) or convolution operations (e.g., Conv). The processing element core 1300 can be configured to receive integer parameters as input and output integer parameters as output. Specifically, as the number of processing elements (PEs) included in the processing element core 1300 increases, the number of transistors in the ASIC also increases, and the manufacturing cost also increases. If the processing element core 1300 is designed to process integer parameters instead of floating point parameters, the number of transistors in the ASIC can be reduced, and the power consumption of the neural processing unit 1000 can be reduced. The number of processing elements (PEs) can range from hundreds to tens of thousands.

[0285] The vector core 1400 includes 1 x N multipliers. N is an integer greater than 1. The vector core 1400 is configured to include multiple arithmetic logic unit (ALU) multipliers. The vector core 1400 includes multipliers and adders. The vector core 1400 can include a vector register file to store array data in a dimensioned form. The vector core 1400 can be configured to receive floating point parameters as input and output floating point parameters.

[0286] The scalar core 1500 includes 1 x 1 multipliers. The scalar core 1500 is a circuit designed to process only one data element. The scalar core 1500 is configured to include an arithmetic logic unit (ALU) multiplier. The scalar core 1500 includes multipliers and adders. The scalar core 1500 can be configured to receive floating point parameters as input and output floating point parameters as output.

[0287] The digital format conversion unit 1600 can be configured to include quantization circuitry and dequantization circuitry. For example, the inputs and outputs of the processing element core 1300 can be integers. The inputs and outputs of the vector core 1400 and the scalar core 1500 can be floating point numbers. Thus, when data is transferred between the processing element core 1300, the vector core 1400, and the scalar core 1500, the digital format conversion unit 1600 can convert the data to the form required by each core. The quantization circuitry of the digital format conversion unit 1600 can convert integers to floating point numbers. The dequantization circuitry of the digital format conversion unit 1600 can convert floating point numbers to integers.

[0288] According to examples of the present disclosure, floating point is not limited to IEEE Standard 754, and brain floating point can be applied to improve the computational efficiency of the neural processing unit 1000, reduce the memory usage of the neural processing unit 1000, reduce the power consumption of the neural processing unit 1000, etc. However, examples of the present disclosure are not limited thereto, and dynamic floating point, variable precision floating point (VPFP), custom floating point (CFP), flexible floating point (FFP), etc. can also be applied. VPFP is a floating point format that allows dynamic setting or adjustment of the bit width of the exponent and the mantissa. VPFP allows flexible adjustment of the precision level based on specific calculations to optimize power and performance. Dynamic floating point format is used interchangeably with variable precision format, so that the size of the mantissa and the exponent can be dynamically adjusted at runtime to respond to data characteristics in real time. CFP refers to an application-specific format in which the bit allocation of the mantissa and the exponent is tailored to the needs of a specific use case. This is very useful for hardware implementation that selects the correct bit width to optimize hardware resources and data precision. FFP refers to a floating point format that allows adjustable bit allocation between the exponent and the mantissa. FFP can be implemented in hardware accelerators to provide the best balance between range and precision. According to examples of the present disclosure, the bit width of the floating point can be between 4 bits and 32 bits. According to examples of the present disclosure, the bit width of the integer can be between 4 bits and 32 bits.

[0289] Figure 11 A processing element applicable to examples of the present disclosure is shown.

[0290] With reference to Figure 11 The processing element (PE) can be configured to include a multiplier, an adder, and an accumulator.

[0291] The multiplier multiplies the input (N) bit data with (M) bit data. The result of the multiplier operation is output as (N+M) bit data, where N and M are integers greater than zero. The first input can be configured to receive (N) bit data. The second input can be configured to receive (M) bit data. The first input can be configured to receive an activation parameter, and the second input can be configured to receive a weight parameter.

[0292] The NPU controller 1700 can limit the operation of the multiplier so that it does not perform the operation (e.g., a zero skip operation) because the processing element (PE) knows that when one value of zero is input to one of the first input and the second input of the multiplier, the operation result will be zero even if the operation is not performed. For example, when zero is input to one of the first input and the second input of the multiplier of the processing element (PE), the multiplier can be configured to operate in a zero skip manner. In addition, the parameters of the lower model (LM) can be less than those of the upper model (UM), or can be a pruned model of the upper model (UM). In this case, the power consumption of the neural processing unit 1000 processing the lower model (LM) can be reduced by zero skipping to improve the pruning efficiency. For zero skipping, each processing element (PE) included in the processing element core 1300 can be enabled or disabled, respectively. The NPU controller 1700 can be configured to provide an enable or disable signal (clock gating control) to each processing element (PE) on a clock-by-clock basis. Here, when the processing element (PE) is deactivated, the multiplier can also be configured to be deactivated. Accordingly, the power consumed by the operation of the multiplier can be reduced. For example, reference can be made to Figure 12 Information about the power consumption of the multiplier is provided. The processing element (PE) can be designed to receive a control signal (clock gating control) for controlling (i.e., enabling or disabling) the zero skip operation from the NPU controller 1700. Specifically, the multiplier of the processing element (PE) can be designed to receive a corresponding control signal (clock gating control) for controlling the zero skip operation from the NPU controller 1700. Alternatively, the adder of the processing element (PE) can be alternatively designed to receive a corresponding control signal (clock gating control) for controlling the zero skip operation from the NPU controller 1700. Alternatively, each of the multiplier and the adder of the processing element (PE) can be designed to simultaneously receive a corresponding control signal (clock gating control) for controlling the zero skip operation from the NPU controller 1700.

[0293] The bit width of the parameters input to the first input and the second input of the processing element (PE) can be determined according to the quantization of the activation parameters and the quantization of the weight parameters of the upper model (UM) and the lower model (LM), respectively. For example, for the upper model (UM), the activation parameters can be quantized to 12 bits and the weight parameters can be quantized to 8 bits; and for the lower model (LM), the activation parameters can be quantized to 8 bits and the weight parameters can be quantized to 4 bits. The quantization level of each parameter can be determined in a compiler (not shown), i.e., the bit width of the parameters input to each input of the processing element (PE) can be different. The quantization information of the data input to each input of the processing element (PE) can be included in the execution code. The execution code can include a first execution code corresponding to the upper model (UM) and a second execution code corresponding to the lower model (LM).

[0294] The accumulator uses the adder to perform (L) times of loop to accumulate the result of the multiplier and the result of the accumulator. Thus, the bit width of the data of the output end and the input end of the accumulator can be output as (N+M+log2(L)) bits, where L is an integer greater than zero. When the accumulator completes the accumulation, an initialization signal (initialization reset) can be input to initialize the data stored in the accumulator to zero. The accumulator of the processing element (PE) is configured to retain the accumulated value when the zero skip function is enabled. The (X) bits can be set to an appropriate bit width, such that based on the maximum value that can be accumulated in the accumulator, overflow of the (X) bit output data does not occur. For example, the (X) bits can be 16 bits to 64 bits.

[0295] As further described below, the digital format conversion unit 1600 can convert the integer parameters output from the processing element core 1300 to floating point numbers and transmit them to one of the vector core 1400, the scalar core 1500, and the NPU memory 1200. The digital format conversion unit 1600 can be controlled by the NPU controller 1700.

[0296] Figure 12 A schematic diagram of the energy consumption per unit operation of the neural processing unit according to examples of the present disclosure.

[0297] Referring to Figure 12 The power consumption of the processing element core 1300, the vector core 1400, and the scalar core 1500 of the neural processing unit 1000 will be described.

[0298] Referring to Figure 12 The table schematically describes the energy consumption of the unit operation of each operator in the neural processing unit 1000. For example, the energy consumption of the addition operation and the multiplication operation can be described respectively. However, the energy consumption of the unit operation can vary depending on the wafer foundry process technology of the semiconductor chip (for example, 2 nanometer technology, 5 nanometer technology, 7 nanometer technology, 14 nanometer technology, 28 nanometer technology, etc.).

[0299] This will be described with reference to the processing core 1300. The processing core 1300 can be designed for calculating integer parameters. In Figure 12In the table, “8b Add” refers to an 8-bit integer add operation by an adder. An 8-bit integer add operation can consume 0.03 pj of energy. “16b Add” refers to a 16-bit integer add operation by an adder. A 16-bit integer add operation can consume 0.05 pj of energy. “32b Add” indicates a 32-bit integer add operation by an adder. A 32-bit integer add operation can consume 0.1 pj of energy. It is further explained that when the processing core 1300 contains thousands or more processing elements (PEs) designed as integer operators, the level of power reduction of the neural processing unit 1000 can be very significant compared to containing the same number of processing elements designed as floating point operators. Typically, for generative artificial intelligence models, the majority of the computations are performed on the processing core 1300. Thus,

[0300] This will be described with reference to the vector core 1400 and the scalar core 1500. The vector core 1400 and the scalar core 1500 can be designed for computing floating point parameters. “16b FP Add” refers to a 16-bit floating point add operation by an adder. A 16-bit floating point add operation can consume 0.4 pj of energy. “32b FP Add” refers to a 32-bit floating point add operation by an adder. A 32-bit floating point add operation can consume 0.9 pj of energy. “8b Mult” refers to an 8-bit integer multiply operation by a multiplier. An 8-bit integer multiply operation can consume 0.2 pj of energy. “32b Mult” refers to a 32-bit integer multiply operation by a multiplier. A 32-bit integer multiply operation can consume 3.1 pj of energy. “16b FP Mult” refers to a 16-bit floating point multiply operation by a multiplier. A 16-bit floating point multiply operation can consume 1.1 pj of energy. “32b FP Mult” refers to a 32-bit floating point multiply operation by a multiplier. A 32-bit floating point multiply operation can consume 3.7 pj of energy.

[0301] According to examples of the present disclosure, a bit width of an integer parameter of a lower model (LM) can be equal to or less than a bit width of an integer parameter of an upper model (UM). A compiler (not shown) can be configured to determine the bit width of the integer parameter of the lower model (LM) and the bit width of the integer parameter of the upper model (UM). For example, execution code of the upper model (UM) compiled to process weight parameters in a first bit width integer form can be executed in the NPU controller 1700 of the neural processing unit 1000. Execution code of the upper model (UM) compiled to process activation parameters in a second bit width integer form can be executed in the NPU controller 1700 of the neural processing unit 1000. Execution code of the lower model (LM) compiled to process weight parameters in a third bit width integer form can be executed in the NPU controller 1700 of the neural processing unit 1000. Execution code of the lower model (LM) compiled to process activation parameters in a fourth bit width integer form can be executed in the NPU controller 1700 of the neural processing unit 1000.

[0302] According to examples of the present disclosure, a bit width of a floating point parameter of a lower model (LM) can be equal to or less than a bit width of a floating point parameter of an upper model (UM). A compiler (not shown) can be configured to determine the bit width of the floating point parameter of the lower model (LM) and the bit width of the floating point parameter of the upper model (UM). For example, execution code of the upper model (UM) compiled to process weight parameters in a fifth bit width floating point form can be executed in the NPU controller 1700 of the neural processing unit 1000. Execution code of the upper model (UM) compiled to process activation parameters in a sixth bit width floating point form can be executed in the NPU controller 1700 of the neural processing unit 1000. Execution code of the lower model (LM) compiled to process weight parameters in a seventh bit width floating point form can be executed in the NPU controller 1700 of the neural processing unit 1000. Execution code of the lower model (LM) compiled to process activation parameters in an eighth bit width floating point form can be executed in the NPU controller 1700 of the neural processing unit 1000.

[0303] Technical features of examples of the present disclosure can be described as follows.

[0304] According to examples of the present disclosure, a device can include a first memory having a first capacity configured to store a first generative neural network model containing first parameters, and a first neural processing unit configured to generate a response corresponding to an input query with the first generative neural network model stored in the first memory. The first neural processing unit can be configured to store first execution code of the first generative neural network model compiled for processing speculative decoding. Thus, the device can process speculative decoding. Furthermore, a federated operating system of multiple language models utilizing speculative decoding or speculative decoding within the device can be provided.

[0305] According to some examples of the present disclosure, the first memory with the first capacity can be configured to store the second generative neural network model containing the second parameters. The number of the second parameters can be greater than the number of the first parameters. Thus, the speculative decoding can be implemented at the device end, which is configured as a single device.

[0306] According to some examples of the present disclosure, the first memory with the first capacity can be the capacity capable of storing the first generative neural network model and the second generative neural network model simultaneously. The speculative decoding can avoid the decrease of the computing speed due to the increase of the memory operations by the separate external memory.

[0307] According to some examples of the present disclosure, the first neural processing unit can be configured to store a second execution code of the second generative neural network model, which is compiled to process the speculative decoding. Thus, multiple generative neural network models can be processed on the device simultaneously.

[0308] According to some examples of the present disclosure, the first neural processing unit can further include a first NPU memory configured to communicate with the first memory, and a first NPU controller configured to control the first neural processing unit.

[0309] According to some examples of the present disclosure, the first execution code can be configured to be stored in the first NPU memory or the first NPU controller.

[0310] According to some examples of the present disclosure, there can further include a second memory with a second capacity configured to store a second generative neural network model containing the second parameters. Thus, the device can efficiently implement the speculative decoding with one neural processing unit and multiple memories.

[0311] According to some examples of the present disclosure, there can further include a second memory with a second capacity configured to store a second generative neural network model containing the second parameters, and a second neural processing unit configured to generate a response corresponding to an input query with the second generative neural network model stored in the second memory. The second neural processing unit can be configured to store a second execution code of the second generative neural network model, which is compiled to process the speculative decoding.

[0312] According to some examples of the present disclosure, the computing performance of the second neural processing unit can be higher than the computing performance of the first neural processing unit. For example, the computing performance of the first neural processing unit can be 25 TOPS, and the computing performance of the second neural processing unit can be 100 TOPS. Thus, the token generation rate of the second generative neural network model processed by the second neural processing unit can be reduced, and thus the speculative decoding rate of the device can be improved.

[0313] According to some examples of the present disclosure, the device can be configured to operate in at least one of a sleep mode, a deep sleep mode, or a reserve mode. At least one of the first neural processing unit and the second neural processing unit can be configured to operate in a low power mode when waiting for a computation result from another neural processing unit during speculative decoding. Thus, when one neural processing unit is processing a pre-population operation, a generation operation, or an acceptance or rejection operation, the other neural processing unit can reduce power consumption.

[0314] According to examples of the present disclosure, a system can include a device including: a first memory having a first capacity configured to store a first generative neural network model containing first parameters; and a first neural processing unit configured to generate a response to an input query using the first generative neural network model stored in the first memory; and a local server including: a second memory having a second capacity configured to store a second generative neural network model containing second parameters; and a second neural processing unit configured to generate a response to an input query using the second generative neural network model stored in the second memory. The first neural processing unit and the second neural processing unit can be configured to cooperate with each other to process speculative decoding. Thus, the system can utilize speculative decoding. Furthermore, a federated operating system that utilizes speculative decoding or multiple language models of speculative decoding at a device end can also be provided.

[0315] According to some examples of the present disclosure, the first memory having the first capacity can be capable of storing the first generative neural network model, and the second memory having the second capacity can be capable of storing the second generative neural network model. Thus, the system can avoid a decrease in computation speed due to an increase in memory operations by a separate external memory during speculative decoding.

[0316] According to some examples of the present disclosure, the first neural processing unit can be configured to store a first execution code of the first generative neural network model compiled for processing speculative decoding, and the second neural processing unit can be configured to store a second execution code of the second generative neural network model compiled for processing speculative decoding. Thus, the system can utilize multiple generative neural network models to implement speculative decoding.

[0317] According to some examples of the present disclosure, the second neural processing unit of the local server can have a higher computation performance than the first neural processing unit of the device. Thus, the processing speed of the system speculative decoding can be improved.

[0318] According to some examples of the disclosure, the device can be plural, and each device can be configured to process speculative decoding through a server. Thus, the server can efficiently process the speculative decoding corresponding to the plural devices.

[0319] According to some examples of the disclosure, at least one of the plural devices can be configured to operate in at least one of a sleep mode, a deep sleep mode, or a reserve mode. Thus, due to the characteristics of the speculative decoding, the power consumption of the waiting device can be reduced.

[0320] According to some examples of the disclosure, the first generative neural network model can be a lower layer model, and the second generative neural network model can be an upper layer model. The upper layer model can be configured to process the speculative decoding while being selectively deactivated. Thus, even if the waiting neural processing unit is operated in a low power consumption mode, the token generation rate is not affected, and the power consumption can be reduced.

[0321] According to some examples of the disclosure, the speculative decoding can include a pre-filling operation, a generation operation, and an acceptance or rejection operation.

[0322] According to examples of the disclosure, a device can include a first memory configured to store a first generative neural network model, and a first neural processing unit configured to generate a response corresponding to an input query using the first generative neural network model stored in the first memory. The first neural processing unit can include a processing core configured to receive an input integer parameter, a vector core and a scalar core configured to receive an input floating point parameter, and a digital format conversion unit configured to convert the integer parameter and the floating point parameter for processing operations of the first generative neural network model. Thus, the device is capable of fast and low power consumption speculative decoding.

[0323] The examples of the disclosure disclosed herein and in the accompanying drawings are given only by way of illustration to facilitate description and understanding of the disclosure, and are not intended to limit the scope of the disclosure. It will be obvious to those of ordinary skill in the art to which the disclosure pertains that other variations based on the inventive concept disclosed herein can be practiced in addition to the examples disclosed herein.

[0324] [National R&D Project in Support of the Invention]

[0325] [Project Identification Number] 1711195792

[0326] [Project Number] 00228938

[0327] [Ministry Name] Ministry of Internal Affairs and Communications

[0328] [Project management (professional) agency name] Institute of Information and Communication Technology Planning and Evaluation

[0329] [Research project name] Development of artificial intelligence semiconductor software integration platform technology

[0330] [Research project name] Development of commercial artificial intelligence SoC semiconductor software development platform technology

[0331] [Project execution agency name] DeepX Co., Ltd.

[0332] [Research period] April 1, 2023 ~ December 31, 2023

Claims

1. A device comprising: a first memory having a first capacity configured to store a first generative neural network model including first parameters; as well as a first neural processing unit configured to generate a response corresponding to an input query using the first generative neural network model stored in the first memory, Wherein, the first neural processing unit is configured to store a first execution code of the first generative neural network model, and the first execution code is compiled to process speculative decoding.

2. The device according to claim 1, in, The first memory having the first capacity is configured to store a second generative neural network model including second parameters, The number of the second parameters is greater than the number of the first parameters.

3. The device according to claim 2, in, The first memory having the first capacity has a capacity capable of storing both the first generative neural network model and the second generative neural network model.

4. The device according to claim 2, in, The first neural processing unit is configured to store the second execution code of the second generative neural network model, the second execution code being compiled to process the speculative decoding.

5. The device according to claim 1, in, The first neural processing unit further includes: a first neural processing unit memory configured to communicate with the first memory; and a first neural processing unit controller configured to control the first neural processing unit.

6. The device according to claim 5, in, The first execution code is configured to be stored in the first neural processing unit memory or the first neural processing unit controller.

7. The device according to claim 1, It also includes: A second memory having a second capacity is configured to store a second generative neural network model including second parameters.

8. The device according to claim 1, It also includes: a second memory having a second capacity configured to store a second generative neural network model including second parameters; and a second neural processing unit configured to generate a response corresponding to the input query using the second generative neural network model stored in the second memory, Wherein, the second neural processing unit is configured to store a second execution code of the second generative neural network model, and the second execution code is compiled to process the speculative decoding.

9. The device according to claim 8, in, The computing performance of the second neural processing unit is higher than the computing performance of the first neural processing unit.

10. The device according to claim 8, in, At least one of the first neural processing unit and the second neural processing unit is configured to operate in a low power mode while waiting for computation results from the other neural processing unit during the speculative decoding.

11. A system comprising: A device comprising: a first memory having a first capacity, configured to store a first generative neural network model including first parameters; and a first neural processing unit configured to generate a response to an input query using the first generative neural network model stored in the first memory; and a local server comprising: a second memory having a second capacity configured to store a second generative neural network model including second parameters; and a second neural processing unit configured to generate a response to an input query using the second generative neural network model stored in the second memory; The first neural processing unit and the second neural processing unit are configured to cooperate with each other to process speculative decoding.

12. The system according to claim 11, in, The first memory having the first capacity is capable of storing the first generative neural network model, and the second memory having the second capacity is capable of storing the second generative neural network model.

13. The system according to claim 11, in, The first neural processing unit is configured to store first execution code of the first generative neural network model, the first execution code being compiled to process the speculative decoding; And the second neural processing unit is configured to store second execution code of the second generative neural network model, the second execution code being compiled to process the speculative decoding.

14. The system according to claim 11, in, The computing performance of the second neural processing unit of the local server is higher than the computing performance of the first neural processing unit of the device.

15. The system according to claim 11, in, There are a plurality of devices, and each device is configured to process the speculative decoding through the local server.

16. The system according to claim 15, in, At least one of the plurality of devices is configured to operate in at least one of a sleep mode, a deep sleep mode, or a retention mode.

17. The system according to claim 11, in, The first generative neural network model is a lower-layer model, and the second generative neural network model is an upper-layer model.

18. The system according to claim 17, in, The upper model is configured to process the speculative decoding while selectively disabled.

19. The system according to claim 11, in, The speculative decoding includes a pre-fill operation, a generate operation, and an accept or reject operation.

20. An apparatus comprising: a first memory configured to store a first generative neural network model; and a first neural processing unit configured to generate a response corresponding to an input query using the first generative neural network model stored in the first memory, Among them, the first neural processing unit includes: a processing core, which is configured to receive input integer parameters; a vector core and a scalar core, which are configured to receive input floating-point parameters; and a digital format conversion unit, which is configured to convert the integer parameters and the floating-point parameters in order to process the operation of the first generative neural network model.

Images