Device for processing multiple batches in electronic device and operation method thereof

Abstract

In the present disclosure, an electronic device may comprise a memory including one or multiple storage media for storing instructions. The electronic device may comprise at least one processor including a processing circuit. When the instructions are individually or collectively executed by the at least one processor, the instructions may cause the electronic device to perform at least one operation. The at least one operation may include an operation of generating a token in multiple batches by using a plurality of high-speed accumulators. The at least one operation may include an operation of identifying that token generation is completed in at least one of the plurality of high-speed accumulators. The at least one operation may comprise an operation of controlling the power consumption of the identified high-speed accumulator.

Description

Multi-batch processing device in electronic device and method of operation thereof

[0001] The present disclosure relates to a device for processing multi-batch based on an artificial intelligence (AI) model in an electronic device and a method of operating the same.

[0002] An artificial neural network (ANN) refers to a computational architecture that models the biological brain. Technologies such as deep learning or machine learning can be implemented based on the ANN. As an example of the ANN, a deep neural network or deep learning may have a multi-layer structure comprising multiple layers.

[0003] AI models are being used in a variety of ways to analyze visual and audio data. To ensure effective operation of AI models on mobile devices, active research and development is underway on hardware technologies related to AI models. For example, research is being conducted on improving hardware architectures that take AI models into account, aiming to optimize the MAC (multiply-accumulate) operation performed in deep learning AI models.

[0004] The above information may be provided as background information to aid in understanding this document. None of the above is claimed to be prior art related to this document or can be used to determine prior art.

[0005] In various embodiments of the present disclosure, a device and an operating method thereof can be provided for controlling high-speed cumulative operation processing for batches in which token generation is completed in an electronic device that performs multi-batch operation based on an AI model.

[0006] According to one embodiment, an electronic device may include a memory comprising one or more storage media for storing instructions. The electronic device may include at least one processor comprising a processing circuit. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to perform at least one operation. The at least one operation may include generating tokens in multiple batches from a plurality of high-speed accumulators. The at least one operation may include identifying that token generation has been completed from at least one of the plurality of high-speed accumulators. The at least one operation may include controlling power consumption of the identified high-speed accumulator.

[0007] According to one embodiment, a storage medium storing computer-readable instructions may be provided. The instructions, when executed by at least a portion of at least one processor of an electronic device, may cause the electronic device to perform at least one operation. The at least one operation may include generating tokens in multiple batches from a plurality of high-speed accumulators. The at least one operation may include identifying that token generation has been completed in at least one of the plurality of high-speed accumulators. The at least one operation may include controlling power consumption of the identified high-speed accumulator.

[0008] According to one embodiment, a method of operating an electronic device may be provided. The method may include generating tokens in multiple batches from a plurality of high-speed accumulators (MACs) (560, 570, 580, 590). The method may include identifying that token generation has been completed from at least one of the plurality of high-speed accumulators. The method may include controlling power consumption of the identified high-speed accumulator.

[0009] In connection with the description of the drawings, the same or similar reference numerals may be used for the same or similar components.

[0010] FIG. 1 is a block diagram of an exemplary electronic device capable of performing the operations described herein.

[0011] FIG. 2 is a block diagram for enabling an AI model to operate in an electronic device according to one embodiment.

[0012] FIG. 3 is a conceptual diagram of a configuration for performing a multi-batch operation in an electronic device according to one embodiment.

[0013] FIG. 4 is a control flowchart for performing a multi-batch operation in an electronic device according to one embodiment.

[0014] FIG. 5 is a configuration diagram of an NPU in an electronic device according to one embodiment of the present disclosure.

[0015] FIG. 6 is a block diagram of a processing engine that is a target for controlling power consumption in response to an EOT occurrence in an electronic device according to one embodiment.

[0016] FIG. 7 is an operational diagram for processing predicted EOT in an electronic device according to one embodiment.

[0017] FIG. 8 is a block diagram of an electronic device within a network environment according to various embodiments.

[0018] FIG. 9 is a block diagram of an exemplary AI system capable of performing the operations described in this document.

[0019] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and conciseness.

[0020] FIG. 1 is a block diagram of an exemplary electronic device (100) capable of performing the operations described in this document.

[0021] Referring to FIG. 1, the electronic device (100) may be one of various forms of electronic devices, such as a notebook (190), smartphones (191) having various form factors (e.g., a bar-type smartphone (191-1), a foldable-type smartphone (191-2), or a sliderable (or rollable) type smartphone (191-3)), a tablet (192), a cellular phone (not shown), and other similar computing devices (not shown). The components, their relationships, and their functions illustrated in FIG. 1 are exemplary only and do not limit the implementations described or claimed in this document. The electronic device (100) may be referred to as a mobile device, a user device, a multi-function device, a portable device, or a server.

[0022] The electronic device (100) may include components including at least one processor (110) (hereinafter, referred to as 'processor (110)'), at least one memory (120) (hereinafter, referred to as 'memory (120)'), at least one display (140) (hereinafter, referred to as 'display (140)'), at least one image sensor (150) (hereinafter, referred to as 'image sensor (150)'), at least one communication circuit (160) (hereinafter, referred to as 'communication circuit (160)'), and / or at least one sensor (170) (hereinafter, referred to as 'sensor (170)'). The components are merely exemplary. For example, the electronic device (100) may include other components (e.g., power management integrated circuitry (PMIC), audio processing circuitry, an antenna, a rechargeable battery, or an input / output interface). For example, some components may be omitted from the electronic device (100). For example, several components may be integrated into a single component. For example, the electronic device (100) may further include at least some of the configurations and / or functions not shown. At least some of the respective components of the electronic device shown (or not shown) may be operatively, functionally, and / or electrically connected to each other.

[0023] The processor (110) may be implemented as one or more IC (integrated circuit (or circuitry)) chips and may perform various data processing. The processor (110) may include at least one electrical circuit and may individually or collectively perform distributed processing of instructions (or programs, data, etc.) stored in the memory (120). The processor (110) may include a processor assembly including one or more processing circuits. The processor (110) may include any processing circuit operative to control the performance and operations of one or more components (e.g., the memory (120), the display (140), the image sensor (150), the communication circuit (160), and / or the sensor (170)) of the electronic device (100). For example, the processor (110) (e.g., an application processor (AP)) may be implemented as a system on chip (SoC) (e.g., a single chip or chipset). For example, the processor (110) may be implemented with multiple cores (or at least one core circuit), multiple chips, or multiple chipsets. For example, the processor (110) may include one or more processing circuits. For example, the processor (110) may include one or more processing circuits configured to individually and / or collectively perform various functions of the present disclosure. As a non-limiting example, at least a portion of the processor (110) may be included in a first chip of the electronic device (100), and at least another portion of the processor (110) may be included in a second chip of the electronic device (100) that is different from the first chip of the electronic device (100).

[0024] For example, the processor (110) may include a central processing unit (CPU) (111), a graphics processing unit (GPU) (112), a neural processing unit (NPU) (113), an image signal processor (ISP) (114), a display controller (115), a memory controller (116), a storage controller (117), a communication processor (CP) (118), and / or a sensor interface (119). These components of the processor (110) are merely exemplary. For example, the processor (110) may further include other components. For example, some components of the processor (110) may be omitted from the processor (110). For example, some components of the processor (110) may be included as separate components of the electronic device (100) outside the processor (110). For example, some components of the processor (110) (e.g., memory controller (116)) may be included within other components (e.g., at least a portion of memory (120), an interface (e.g., available for connection to at least one component of the electronic device (100)), a display (140) and / or an image sensor (150)).

[0025] The processor (110) may cause other components of the electronic device (100) to perform various operations by executing instructions stored in the memory (120). The CPU (111) (or central processing circuit) may be configured to control components of the processor (110) based on the execution of instructions stored in the memory (120) (e.g., volatile memory (121) and / or non-volatile memory (122)). The GPU (112) (or graphics processing circuit) may be configured to execute parallel operations (e.g., rendering). The NPU (113) (or neural processing circuit, or artificial intelligence (AI) chip) may be configured to execute operations for an AI model (e.g., convolution computation). The ISP (114) (or image signal processing circuit) may be configured to process a raw image acquired through the image sensor (150) into a format suitable for a component within the electronic device (100) or a component of the processor (110). The display controller (115) (or display control circuit, or DPU (display processing unit)) may be configured to process an image acquired from the CPU (111), GPU (112), ISP (114), or memory (120) (e.g., volatile memory (121)) into a format suitable for the display (140). The memory controller (116) (or memory control circuit) may be configured to control reading data from the volatile memory (121) and writing data to the volatile memory (121). The above storage controller (117) (or storage control circuit) may be configured to control reading data from the non-volatile memory (122) and writing data to the non-volatile memory (122).The CP (118) (communication processing circuit) may be configured to process data obtained from a component of the processor (110) into a format suitable for transmitting to another electronic device via the communication circuit (160), or to process data obtained from another electronic device via the communication circuit (160) into a format suitable for processing by a component of the processor (110). For example, the communication circuit (160) may include one or more communication circuits. The sensor interface (119) (or sensing data processing circuit, sensor hub) may be configured to process data on the state of the electronic device (100) and / or the state of the surroundings of the electronic device (100), obtained via the sensor (170), into a format suitable for a component of the processor (110).

[0026] The memory (120) may include one or more storage media (or one or more storage devices). For example, the memory (120) may include a memory assembly including one or more storage media. For example, the one or more storage media may include permanent memory (e.g., non-volatile memory (122)) such as a hard drive, flash memory, read-only memory (ROM), semi-permanent memory (e.g., volatile memory (121)) such as random access memory (RAM), any other suitable type of storage (or storage assembly), or any combination thereof. The memory (120) may include a cache memory, which is one or more different types of memory used to temporarily store data for a function or feature of the electronic device (100). As a non-limiting example, the cache memory may be included within the processor (110). The memory (120) may be fixedly embedded within the electronic device (100) or incorporated into one or more suitable types of components (e.g., a subscriber identity module (SIM) card and / or a secure digital (SD) card) that may be repeatedly inserted into and removed from the electronic device (100).

[0027] For example, the memory (120) may store one or more software applications, such as an operating system (or system) software application, a firmware software application, a driver software application, a plug-in (e.g., add-in, add-on, and / or applet) software application, and / or any other suitable software applications. For example, the one or more software applications may include instructions executable by the processor (110). For example, the memory (120) may store instructions callable by an application programming interface (API). For example, the memory (120) may store instructions within a library.

[0028] According to an example, the electronic device (100) can execute at least one instance of an AI model. The instance may be an object corresponding to a program (or application), such as an AI model, for example. The instance may be named a replica, a pod, a container, or a virtual machine, and there is no limitation on the name thereof. The number of instances may correspond to the size of a resource (e.g., a GPU (112) or an NPU (113)), and accordingly, the number of instances may be used interchangeably with the size of the resource, or the instances may be used interchangeably with the resource.

[0029] As an example, a plurality of user requests may be input to the electronic device (100). The user requests may be associated with a service. The user request may be processed by a first instance of a first AI model, and a first processing result may be provided from the first instance of the first AI model. The first processing result may be processed by a first instance of a second AI model, and accordingly, a second processing result may be provided by the first instance of the second AI model. By serial processing of the processing results, the first instance of the M-th AI model may receive and process the N-1-th processing result. The first instance of the M-th AI model may provide the N-th processing result as a response. Accordingly, a response corresponding to the user request may be provided.

[0030] Based on the above-described process, responses corresponding to each of a plurality of user requests may be provided. Meanwhile, since processing must be performed by an instance, the time for providing responses corresponding to each of a plurality of user requests (hereinafter referred to as “response time”) may take a relatively long time. The response time may affect the latency of the instance. In order to reduce the response time, the electronic device (100) may increase the number of instances of at least one AI model, which may be referred to as scaling out. However, there may be a limit to increasing the number of instances due to hardware and / or software constraints of the electronic device (100) and / or parameter restrictions of the AI ​​model (e.g., large language model (LLM)).

[0031] FIG. 2 is a block diagram for enabling an AI model to operate in an electronic device (e.g., the electronic device (100) of FIG. 1) according to one embodiment.

[0032] Referring to FIG. 2, the electronic device (100) may include a processor (210) (e.g., the processor (110) of FIG. 1), a memory (230) (e.g., the memory (120) of FIG. 1), and / or an interface (IF) (220) (e.g., the display (140) of FIG. 1). The electronic device (100) may be a device for providing a service linked to at least one AI model.

[0033] At least one of the above AI models may be based on natural language processing (NLP) technology. The NLP technology is, for example, a technology that allows the electronic device (100) to understand or process a user's input, i.e., a natural language that can be expressed in voice and / or text (hereinafter referred to as a "prompt (250)"). The electronic device (100) can understand natural language through NLP, and based on this, can identify human intentions or convey information in a language that humans can understand. The NLP can learn the order of words or tokens to understand human language and predict the probability of the next word or token in a given text. The token is a basic unit for processing or understanding a prompt in an AI model. The main technologies of the NLP include tokenization, part-of-speech tagging, syntax analysis, named entity recognition, or sentiment analysis for prompts corresponding to the user's input.

[0034] At least one AI model may be based on language model (LM) technology. The LM technology can predict the probability of each word or token (hereinafter collectively referred to as "token") based on a sequence of words or tokens. For example, the LM technology may predict the token that may follow a preceding token. In other words, the LM may be an AI model trained to output the most statistically appropriate token based on a prompt. For example, the electronic device (100) may include an LLM (240). The LLM (240) included in the electronic device (100) may be a single generative LLM. The LLM (240) may be a large-scale deep learning model pre-trained based on a vast amount of data. The LLM (240) may provide the ability to predict a user's intent based on a relatively small number of prompts. The LLM may be used, for example, in generative AI that generates content based on prompts input in the form of human language or text.

[0035] In one embodiment, an LLM may be referred to as a language model comprised of an artificial neural network pre-trained on a large amount of data (e.g., text data). The LLM may include approximately ten times more parameters (e.g., approximately 100 billion or more parameters) than a typical language model. The LLM may utilize a transformer artificial neural network structure based on an attention mechanism. The attention mechanism enables the AI ​​model to focus on important parts within the input data. The attention mechanism may be used to predict output data by predicting the extent to which at least a portion of time-series input data (e.g., input data such as voice or video, or input data of some layers of the neural network) contributes to the intermediate or final output of the neural network. The recurrent neural network (RNN) structure, which sequentially processes each element of a sequence, has poor prediction performance when there is information dependency between long time series distances, but the attention mechanism can consider information dependency between long time series distances by controlling the degree of weight concentration within the overall (or partial) context of the input data.

[0036] For example, a transformer may include an encoder-decoder structure. The encoder may process input data to output compressed information (e.g., an attention mechanism), and the decoder may process the compressed information to output output data in token units. The encoder and decoder may each include an independent attention network. Additionally, the encoder and decoder may include a cross-attention network connecting them.

[0037] For example, LLM can be trained through two operations: pre-training and fine-tuning. Pre-training is the process of allowing LLM to process large amounts of text data and acquire general linguistic knowledge. For example, this may involve self-supervised learning to predict the next word using a sequence of previous words in a text sequence. Fine-tuning is the process of training a large-scale language model to be suitable for a specific domain (e.g., chatbot, translation, summarization, Q&A) or task. Based on the pre-trained model, additional supervised learning (or adaptive learning) can be performed using a dataset suited to the domain's purpose. LLM can perform tasks with text input containing natural language, called prompts (310). For example, LLM can include BERT (bidirectional encoder representations from transformer) and GPT (generative pre-trained transformer). The term "LLM" can refer to the neural network model itself, but can also refer to the model of an LLM-based application (e.g., chatbot, translation, summarization, text classification, sentence generation). For example, an LLM-based chatbot such as chatGPT can also be referred to as an LLM. "LLM" can also include an inference engine that utilizes the LLM neural network model. For example, "entering an input prompt into an LLM" can be referred to as "entering an input prompt into an LLM-based inference engine."

[0038] As previously described, NLP is an AI model for the electronic device (100) to understand or analyze human language, whereas LLM may be an AI model for predicting the next word or sentence (e.g., a subsequent token) based on given data (e.g., a preceding token). The NLP is utilized for search engines, machine translation, or sentiment analysis. The LLM is utilized for sentence generation, auto-completion, or voice recognition. The electronic device (100) may provide a generative AI service that uses NLP to understand a user's question, for example, and LLM to generate an appropriate answer thereto.

[0039] The processor (210) may execute software (e.g., a program) to control at least one other component (e.g., a hardware or software component) of the electronic device (100) that is electrically connected thereto. The processor (210) may perform various data processing or operations. As at least a part of the data processing or operations, the processor (210) may store commands or data received from other components (e.g., the I / F (220)) in a memory (230) (e.g., a volatile memory, but without limitation). As at least a part of the data processing or operations, the processor (210) may process commands or data stored in the memory (230) (e.g., a volatile memory, but without limitation). As at least a part of the data processing or operations, the processor (210) may store data resulting from processing commands or data in the memory (230) (e.g., a non-volatile memory, but without limitation). The above processor (210) may include, but is not limited to, a CPU (211) (e.g., CPU (111) of FIG. 1), an NPU (213) (e.g., NPU (113) of FIG. 1), and / or a GPU (215) (e.g., GPU (112) of FIG. 1), which includes a processing circuit.

[0040] The memory (230) may store various data used by at least one component (e.g., processor (210) and / or I / F (220)) of the electronic device (100). The data may include, for example, software (e.g., program) and input data or output data for commands related thereto. The memory (230) may include volatile memory and / or non-volatile memory. The memory (230) may include a hard disk, a ROM, a RAM (e.g., SRAM, PSRAM, or DRAM), a cache memory, and / or a register, and there is no limitation on its implementation. Some of the above-described entities (e.g., registers, but not limited thereto) may be implemented as a part of the processor (210), and there is no limitation on the form of their implementation. At least one AI model (e.g., LLM (240)) for instance execution may be stored in the memory (230).

[0041] The memory (230) can store at least one instruction. The processor (210) can execute at least one instruction stored in the memory (230). The at least one instruction, when executed by the processor (210), can cause the electronic device (100) to perform at least one operation. For example, as the at least one instruction is executed by the processor (210), at least one other component may be controlled, and / or various data processing or calculations may be performed. The performance of one operation by the processor (210) may mean, for example, that the operation is performed by (or under the control of) one entity included in the processor (210) (for example, the main processor, but without limitation). The performance of one operation may mean, for example, that a specific operation is performed by (or under the control of) multiple entities (for example, multiple processors). The performing of multiple operations may mean, for example, that the multiple operations are all performed by (or under the control of) a single entity (e.g., but not limited to, the main processor). The performing of multiple operations may mean, for example, that some of the multiple operations are performed by at least one entity, and some of the remaining operations are performed by at least one other entity. At least one instruction causing the performance of one or more operations may be stored, for example, in a single memory, or may be stored distributedly in each of a plurality of memories.

[0042] In the electronic device (100), the LLM (240) may share resources (e.g., data processing or computational power) corresponding to part or all of at least one processor included in the processor (210) and / or resources (e.g., data recording area) corresponding to part or all of the memory (230). For example, the LLM (240) may be operated by at least one of the CPU (211), the NPU (213), or the GPU (215). The LLM (240) may be executed solely by the CPU (211), for example, by being allocated at least a portion of the memory (230). The LLM (240) may be executed solely by the NPU (213), for example, by being allocated at least a portion of the memory (230). The LLM (240) may be executed solely by the GPU (215), for example, by being allocated at least a portion of the memory (230). The LLM (240) can be performed by the CPU (211) and the NPU (213) in cooperation, for example, by being allocated at least a portion of the memory (230). The LLM (240) can be performed by the CPU (211) and the GPU (215) in cooperation, for example, by being allocated at least a portion of the memory (230). The LLM (240) can be performed by the NPU (213) and the GPU (215) in cooperation, for example, by being allocated at least a portion of the memory (230). The LLM (240) can be performed by the CPU (211), the NPU (213), and the GPU (215) in cooperation, for example, by being allocated at least a portion of the memory (230). The various embodiments to be described later in the present disclosure are not limited to the combination of components for performing the LLM (240) and can be implemented and / or applied based on any combination.

[0043] The I / F (220) may receive a prompt (250) corresponding to an input from a user, and transmit the received prompt (250) to the processor (210). The prompt (250) may be a medium that guides a generative AI (e.g., LLM (240)) to perform a task or generate a result in a desired direction. The prompt (250) may be the only window through which the user can communicate with the LLM (240). The prompt (250) needs to be clear and specific in order to obtain an answer close to the desired result from the LLM (240). The I / F (220) may receive a result processed by the LLM (240) based on the prompt (250) from the processor (210), and output a response result (260) converted into a natural language that is a form that can be recognized by humans (e.g., voice or text). The above I / F (220) can input or output natural language in the form of voice and / or text, for example, through at least one component such as a keyboard, a touch panel, a display, and / or a speaker.

[0044] FIG. 3 is a conceptual diagram of a configuration for performing a multi-batch operation in an electronic device (e.g., the electronic device (100) of FIG. 1) according to one embodiment.

[0045] Referring to FIG. 3, an AI model (240) can be generated through machine learning. The LLM (240) illustrated in FIG. 2 is a representative example of an AI model (240), and for convenience, the same reference numeral 240 will be used herein below. The machine learning can be performed, for example, on-device in an electronic device (100) capable of operating AI. The machine learning can also be performed, for example, through a separate external server based on a network environment. In this case, the learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above.

[0046] An AI model (240) according to one embodiment may be an artificial neural network model written in a specified language and including a plurality of layers and / or operations (or calculations). The AI ​​model according to one embodiment may be one of a feedforward neural network (FNN), a deep neural network (DNN), a convolutional neural network (CNN), a region with convolution neural network (R-CNN), a region proposal network (RPN), a recurrent neural network (RNN), a stacking-based deep neural network (S-DNN), a state-space dynamic neural network (S-SDNN), a Deconvolution Network, a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, a Fully Convolutional Network, a long short-term memory (LSTM) Network, a Classification Network, or a combination of two or more of the above, but is not limited to the examples described above. An AI model according to one embodiment can be trained on specified data, acquire input data, perform operations based on the input data, and generate output data. In addition to the hardware structure, the AI ​​model (240) may additionally or alternatively include a software structure.

[0047] The AI ​​model (240) can be trained on specified data. The AI ​​model (240) can obtain input data and perform operations based on the obtained input data to generate output data. The AI ​​model (240) can be a generative AI model. The generative AI model can produce text, images, or other media in response to a user's input based on learned information. A large language model (LLM) (e.g., LLM (240) of FIG. 2) can be a representative example of a generative AI. For example, the LLM (240) can be operated by at least one or a combination of at least two of a CPU (e.g., CPU (211) of FIG. 2), an NPU (e.g., NPU (213) of FIG. 2), or a GPU (e.g., GPU (215) of FIG. 2).

[0048] The AI ​​model (240) may receive a prompt (310) (e.g., a question) corresponding to the user's input as input. The AI ​​model (240) may process the user input entered in the prompt (310) at once. Thereafter, the AI ​​model may iterate inference to generate one token at each iteration. The prompt (310) may be a single sentence or multiple consecutive sentences. The prompt (310) may be a question for the user to obtain desired information from the AI ​​model. The AI ​​model (240) may analyze the user's input in natural language, i.e., the prompt, and convert it into a form that the AI ​​model can recognize (e.g., machine language). This is called natural language processing (NLP). The AI ​​model (240) may perform a processing operation to understand the prompt (310). For example, the AI ​​model (240) may process the prompt (310) through a multi-batch operation. The AI ​​model may provide a response (320) to the user based on the processing of the prompt (310). The response (320) may be, for example, information such as an answer or summary, converted into a natural language such as text and delivered to the user.

[0049] For example, the AI ​​model (240) may support multi-batch operations by interworking with the NPU (213) and the CPU (211). The AI ​​model (240) may include a multi-batch processing module (330) to process the multi-batch operations. The multi-batch processing module (330) may additionally or alternatively include a software structure based on a hardware structure. For example, a representative LLM of the AI ​​model (140) may generate an output (320) for a prompt (310) corresponding to a user's input. The output (320) generated by the LLM may be, for example, an answer, a summary, or a translation result.

[0050] In one embodiment, a memory (e.g., memory (230) of FIG. 2) may include data related to an AI model, such as layers, weights, and operations of the AI ​​model. In one embodiment, the memory (230) may include a plurality of nodes indicated by the AI ​​model, and / or weights assigned to connections between the plurality of nodes. For example, the memory (230) may include weights for one or more convolution operations related to the AI ​​model. For example, the memory (230) may include weights for one or more convolution operations based on a convolution filter related to the AI ​​model. For example, the memory (230) may include weights for one or more depth-wise convolution operations related to the AI ​​model. For example, the memory (230) may include weights for mean pooling operations related to the AI ​​model. For example, the memory (230) may include weights for obtaining parameters (e.g., argmax) to maximize the computational results of layers associated with the AI ​​model. For example, the memory (230) may include weights for linearizing the computational results of layers associated with the AI ​​model (e.g., linear computation).

[0051] In one embodiment, the memory (e.g., memory (230) of FIG. 2) may include at least one of node weight information, quantization / de-quantization weight information, or parameter information. For example, the node weight information may refer to information necessary for operation with input data as parameter information for performing an operation. For example, the node quantization / de-quantization weight information may refer to information for converting the inference data type of the operation, such as FP32<->INT32 or FP32<->INT8. For example, the node parameter information may refer to operation shape information, an axis for performing the operation, a padding option, a bias, and a threshold. In addition, the parameter information may further include information such as a memory handle for memory reuse.

[0052] The above multi-batch processing module (330) can process a multi-batch operation including an operation operation for token generation for each batch included in the multi-batch. The batch is a form of data processing of a computer, which means organizing data to be processed for a certain period of time or a certain amount and processing them in batches. In other words, the batch or batch processing method that can be applied to a computer system may be a data processing method that collects the data to be processed in a certain unit (e.g., day or month) and processes them as one. It is the counterpart of immediate processing or real-time processing that processes data immediately after it is generated. In order to provide an efficient multi-batch operation, the AI ​​model (240) can provide an EOT (end of token) processing operation to prevent power consumption in a batch in which token generation has ended among the multiple batches included in the multi-batch. In this case, power consumption resulting from processing the multi-batch operation can be reduced.

[0053] The multi-batch processing module (330) is a processing engine for processing various AI models such as LLM or CNN in the NPU (213), and may include a plurality of high-speed multiply-accumulators (e.g., MAC #1 (560), MAC #2 (570), MAC #3 (580), ……, MAC #N (590) of FIG. 5, where N is a natural number greater than or equal to 2) therein. The high-speed multiply-accumulators may be composed of at least two control cores (e.g., control core (510) of FIG. 5). As an example, the multi-batch processing module (330) may be a module that generates an output (e.g., a token) for a prompt (310) by an AI model (240) in a mobile environment. The multi-batch processing module (330) may support a multi-batch operation. The multi-batch processing module (330) may process inputs from various users or provide various processing results to users through a multi-batch operation.

[0054] The multi-batch processing module (330) may control (e.g., block, limit, or refrain from) an operation operation for the high-speed accumulator after an EOT event occurs in a specific batch included in the multi-batch. If the operation operation is not controlled (e.g., block, limit, or refrain from), the high-speed accumulator may continue to perform unnecessary operation operations (e.g., MAC operation). For example, “0,” which is an input for the high-speed accumulator operation, may continue to be fed to the high-speed accumulator. In this case, the high-speed accumulator may continue to process an operation that multiplies the input “0” by a weight and an operation that adds a bias value to the result value. The result value of the operation processed by the high-speed accumulator may have a certain value other than 0 due to the bias value. The non-zero result value may be transmitted as an input to the next layer, which may cause additional power consumption in the next layer. In addition, power consumption may occur due to the non-zero result value (dummy) when performing vector operations by a processing engine (e.g., processing engine (600) of FIG. 6).

[0055] Therefore, in order to prevent power consumption that occurs unnecessarily, the multi-batch processing module (330) may block the operation operation of the high-speed accumulator that was generating the token for the specific batch when an EOT event occurs in a specific batch included in the multi-batch. For example, the multi-batch processing module (330) may operate to limit (e.g., block, restrict, or refrain from) the power consumption of the high-speed accumulator. For example, the multi-batch processing module (330) may provide a designated command (e.g., an operation blocking command (done signal or skip signal)) to the high-speed accumulator so that the high-speed accumulator does not process a matrix operation. The operation blocking command may be input to a control pin provided to the high-speed accumulator, for example. For example, the multi-batch processing module (330) may operate to stop loading weights to the high-speed accumulator. When the weight loading is stopped, the high-speed accumulator cannot perform the calculation processing, so power consumption may not occur. For example, the multi-batch processing module (330) may operate to physically control (e.g., block, limit, or refrain) the power supplied to the high-speed accumulator. To this end, a switching element may be provided in front of the power terminal of the high-speed accumulator, and the switching element may be controlled to control (e.g., block, limit, or refrain) the power supply path to the high-speed accumulator. As described above, the multi-batch processing module (330) may operate to control (e.g., block, limit, or refrain) the matrix calculation operations in the plurality of high-speed accumulators in response to EOT.

[0056] FIG. 4 is a control flowchart for performing a multi-batch operation in an electronic device (e.g., the electronic device (100) of FIG. 1) according to one embodiment.

[0057] Referring to FIG. 4, the electronic device (100) can determine, in operation 411, whether an event requesting a multi-batch operation occurs. The multi-batch operation may be an operation in which multiple batches are divided and processed by multiple high-speed accumulators (e.g., MAC #1 (560), MAC #2 (570), MAC #3 (580), ......, MAC #N (590) of FIG. 5, where N is a natural number greater than or equal to 2). The multi-batch operation event may be generated, for example, by inputting a prompt (310) requesting the use of AI from the outside. The prompt (e.g., prompt (310) of FIG. 3) may be information (e.g., a question) input by a user. The multi-batch operation event may be generated, for example, when a task requiring the activation of an AI function must be performed. In addition, in most situations where calculations by multiple high-speed accumulators (MACs) (560, 570, 580, 590) are required, it can be determined that the multi-batch operation event has occurred. The electronic device (100) can, for example, continuously perform operation 411 until the multi-batch operation event occurs. The electronic device (100) can also, for example, perform other preset operations if the multi-batch operation event does not occur.

[0058] When a multi-batch operation event occurs, the electronic device (100) may initialize “n,” which is a parameter required for performing the multi-batch operation, in operation 413. The “n” may be a count value for counting tokens generated by the computational processing in the plurality of high-speed accumulators. The “n” is a natural number. Here, on the premise that the computational processing in the plurality of high-speed accumulators is synchronized, one count value “n” for counting the generated tokens is defined, but the present invention is not limited thereto. For example, multiple count values ​​may be defined, and the number of tokens to be generated in the plurality of high-speed accumulators may be independently counted or counted in the form of partial groups using the multiple count values. In the following, for the purpose of convenience of explanation, one count value “n” will be assumed for description. As an example, the “n” may be initialized to “1”.

[0059] The electronic device (100) may, when a multi-batch operation event occurs, initialize “k,” a parameter required for performing the multi-batch operation, in operation 413. The “k” is a count value for counting batches that generate an End of Token Generation (EOT) among the multi-batches. When an EOT is generated, the processor may process the EOT in the form of an event (input). The “k” may be a positive integer including 0. Upon initialization, k may have 0.

[0060] In the above electronic device (100), in addition to the two count values ​​defined above, additional count values ​​may be defined as needed.

[0061] The electronic device (100), in operation 415, may perform operation processing in the plurality of high-speed accumulators to generate an nth token corresponding to each of the K batches included in the multi-batch. For example, the operation processed in the plurality of high-speed accumulators may be a matrix operation. When the nth tokens are generated in the plurality of high-speed accumulators, the electronic device (100), in operation 417, may increase the count value “n” by “1” to indicate the next token to be generated.

[0062] The electronic device (100), in operation 419, may determine whether an EOT event has occurred among the K batches included in the multi-batch, for batches that are in an active state in which an EOT event has not occurred, i.e., an EOT token has not been generated. For example, if there is no batch in which an EOT event has occurred, the electronic device (100) may determine that all K batches are in an active state. The EOT event may occur when an EOT token has been generated and there is no need to generate any more tokens in the batch. The occurrence of the EOT event may be recognized, for example, by the high-speed accumulator. In this case, the high-speed accumulator may notify a component responsible for overall control related to the operation (e.g., the NPU (213) of FIG. 2 or the control core (510) of FIG. 5) that the EOT has been generated.

[0063] If the electronic device (100) determines that an EOT event has not occurred in any batch included in the multi-batch, in operation 415, it may perform an operation to generate an nth token for one or more batches in which an EOT has not occurred.

[0064] If the electronic device (100) determines that an EOT event has occurred in a specific batch included in the multi-batch, the electronic device (100) may control an operation of a corresponding high-speed accumulator that was processing an operation to generate a token for the specific batch in operation 421. For example, the electronic device (100) may operate to block power consumption of the corresponding high-speed accumulator. For example, the electronic device (100) may provide a designated control command (e.g., an operation blocking command (done signal or skip signal)) (hereinafter referred to as an 'operation blocking command') to the corresponding high-speed accumulator so as to prevent the corresponding high-speed accumulator from processing a matrix operation. The operation blocking command may be input to, for example, a control pin provided to the corresponding high-speed accumulator. For example, the electronic device (100) may operate to stop loading weights to the corresponding high-speed accumulator. When the above weight loading is stopped and a skip or done signal is generated at the control pin, the corresponding high-speed accumulator may not be able to perform calculation processing, so power consumption may not occur. For example, the electronic device (100) may operate to physically control (e.g., block, limit, or refrain) the power supplied to the corresponding high-speed accumulator. To this end, a switching element may be provided at a front end of a power terminal of the corresponding high-speed accumulator, and the switching element may be controlled to control (e.g., block, limit, or refrain) the power supply path to the corresponding high-speed accumulator. As described above, the electronic device (100) may operate to control (e.g., block, limit, or refrain) the matrix calculation operations in the plurality of high-speed accumulators in response to EOT.

[0065] The electronic device (100) may, in operation 423, increase the count value “k” for counting the occurrence of the EOT by “1”. In operation 425, the electronic device (100) may compare the count value “k” with “K”, which is the total number of batches included in the multi-batch, to determine whether an EOT has occurred in all batches. For example, if the count value “k” has not reached K, which is the total number of batches, the electronic device (100) may determine that there is still a high-speed accumulator that must perform an operation process for generating a token. In this case, the electronic device (100) may, in operation 415, perform an operation for generating a token for one or more batches in which an EOT has not occurred by using the updated count value “n.”

[0066] The electronic device (100) may determine that token generation for all batches included in the multi-batch is completed if the count value “k” reaches the total number of batches K. In this case, the electronic device (100) may, in operation 427, terminate the operation according to the multi-batch operation being performed by the plurality of high-speed accumulators. After terminating the operation according to the multi-batch operation, the electronic device (100) may repeatedly perform the above-described operation in response to the occurrence of a new multi-batch operation event.

[0067] FIG. 5 is a configuration diagram of an NPU (e.g., NPU (213) of FIG. 2) in an electronic device (e.g., electronic device (100) of FIG. 1) according to one embodiment.

[0068] Referring to FIG. 5, the NPU (213) may include a control core (510), an SRAM (520), or a matrix processor (530) to perform a multi-batch operation. The control core (510), the SRAM (520), or the matrix processor (530) may exchange data, signals, or information through an internal bus (540). The matrix processor (530) includes a plurality of high-speed accumulation units (MACs) to support the multi-batch operation. As an example, the matrix processor (530) may include MAC #1 (560), MAC #2 (570), MAC #3 (580), ......, MAC #N (590). The above MAC #1 (560), the MAC #2 (570), the MAC #3 (580), ......, the MAC #N (590) can generate tokens by performing computational processing on at least one batch among multiple batches (e.g., batch #1-1, batch #1-2, batch #2-1, batch #2-2, batch #3-1, batch #3-2, batch #N-1, batch #N-2 of FIG. 5).

[0069] The MAC #1 (560) can receive an input value ① for operation processing from the control core (510) through the bus (540). The MAC #1 (560) can receive a weight ⓐ for operation processing from the SRAM (520) through the bus (540). The MAC #1 (560) can include a control pin (561) to which the input value ① to be transmitted to the MAC controller is input, or input pins (563, 565) to which the weight ⓐ is input. The input pins (563, 565) can serve as an interface for batch #1-1 or batch #1-2 included in a multi-batch.

[0070] The MAC #2 (570) can receive an input value ② for operation processing from the control core (510) through the bus (540). The MAC #2 (570) can receive a weight ⓑ for operation processing from the SRAM (520) through the bus (540). The MAC #2 (570) can include a control pin (571) to which the input value ② to be transmitted to the MAC controller is input, or input pins (573, 575) to which the weight ⓑ is input. The input pins (573, 575) can serve as an interface of batch #2-1 or batch #2-2 included in a multi-batch.

[0071] The MAC #3 (580) can receive an input value ③ for operation processing from the control core (510) through the bus (540). The MAC #3 (580) can receive a weight ⓒ for operation processing from the SRAM (520) through the bus (540). The MAC #3 (580) can include a control pin (581) to which the input value ③ to be transmitted to the MAC controller is input, or input pins (583, 585) to which the weight ⓒ is input. The input pins (583, 585) can serve as an interface for batch #3-1 or batch #3-2 included in a multi-batch.

[0072] The MAC #N (590) can receive an input value ④ for operation processing from the control core (510) through the bus (540). The MAC #N (590) can receive a weight ⓓ for operation processing from the SRAM (520) through the bus (540). The MAC #N (590) can include a control pin (591) to which the input value ④ to be transmitted to the MAC controller is input, or input pins (593, 595) to which the weight ⓓ is input. The input pins (593, 595) can serve as an interface for batch #N-1 or batch #N-2 included in a multi-batch.

[0073] Each of the MAC #1 (560), the MAC #2 (570), the MAC #3 (580), ......, the MAC #N (590) can process a MAC operation using input values ​​and weights. For example, the MAC operation may be composed of a multiplication operation and an addition operation, for example, a multiply-accumulate operation. For example, the MAC operation in the MAC #1 (560) may be defined as “(input value ① × weight ⓐ) + bias)”. The same operation may be applied to each of the MAC #2 (570), the MAC #3 (580), ......, the MAC #N (590), with only different input values ​​and weights used.

[0074] For example, the control core (510) can monitor whether token generation is completed (EOT) for a batch being processed by the MAC #1 (560), the MAC #2 (570), the MAC #3 (580), ......, the MAC #N (590). The control core (510) can recognize the occurrence of EOT, which notifies that token generation of any MAC among the MAC #1 (560), the MAC #2 (570), the MAC #3 (580), ......, the MAC #N (590) is completed. The control core (510) can output an operation blocking command (done signal or skip signal) to any MAC in response to the occurrence of the EOT. The operation blocking command can be transmitted to a MAC controller included in any MAC through a control pin provided in the any MAC. The above MAC controller can control (e.g., block, limit, or refrain from) MAC operation processing that generates power consumption according to the above operation blocking command.

[0075] For example, the SRAM (520) can monitor whether token generation is completed (EOT) for a batch being processed by the MAC #1 (560), the MAC #2 (570), the MAC #3 (580), ......, the MAC #N (590). The SRAM (520) can be provided, for example, with the occurrence of an EOT event of the token generation from the control core (510). The SRAM (520) can recognize the occurrence of an EOT event according to the completion of token generation of any MAC among the MAC #1 (560), the MAC #2 (570), the MAC #3 (580), ......, the MAC #N (590). The SRAM (520) can stop providing weights to any MAC in response to the occurrence of the EOT event.

[0076] FIG. 6 is a block diagram of a processing engine (600) that is to control power consumption in response to an EOT occurrence in an electronic device (e.g., the electronic device (100) of FIG. 1) according to one embodiment.

[0077] Referring to FIG. 6, the processing engine (600) may include a matrix operator (610) (e.g., matrix processor (530)), a scalar operator (620), or a vector operator (630). The matrix operator (610) may include a plurality of high-speed accumulators (MACs). For example, the matrix operator (610) may include MAC #1 (611), MAC #2 (613), MAC #3 (615), and MAC #4 (617). The above MAC #1 (611), the above MAC #2 (613), the above MAC #3 (615), and the above MAC #4 (617) can generate tokens by performing computational processing on at least one batch among multiple batches (e.g., batch #1-1, batch #1-2, batch #2-1, batch #2-2, batch #3-1, batch #3-2, batch #N-1, batch #N-2 of FIG. 5).

[0078] Each of the MAC #1 (611), the MAC #2 (613), the MAC #3 (615), and the MAC #4 (617) may receive an operation blocking command C1, C2, C3, or C4 from the LLM (240), the NPU (213), or the control core (e.g., the control core (510) of FIG. 5) through a control pin in response to the completion of token generation for the assigned batch and the occurrence of EOT. Each of the MAC #1 (611), the MAC #2 (613), the MAC #3 (615), and the MAC #4 (617) may stop all operation processing so as not to generate power consumption when it receives an operation blocking command C1, C2, C3, or C4 through a control pin. When the above EOT occurs, the SRAM (e.g., SRAM (520) of FIG. 5) may also stop loading weights to the corresponding MAC. To this end, the LLM (240), NPU (213), or control core (510) may provide information about the MAC in which the EOT occurred to the SRAM (520).

[0079] below shows an example in which EOT occurs when generating tokens for each of the four batches included in the multi-batch by the matrix operator (610).

[0080] Token Index #1#2#3#4#5#6#7#8#9Batch #1IamgoingtoschooltomorrowEOT--Batch #2IamgoingtogototheschoolEOTBatch #3IwillbethereEOT----Batch #4IwillgotheschoolEOT---

[0081] Referring to the above , MAC #1 (611) can generate a token for batch #1, MAC #2 (613) can generate a token for batch #2, MAC #3 (615) can generate a token for batch #3, and MAC #4 (617) can generate a token for batch #4.

[0082] For the first token, MAC #1 (611), MAC #2 (613), MAC #3 (615), and MAC #4 (617) all generated “I”. For the second token, MAC #1 (611) and MAC #2 (613) generated “am”, and MAC #3 (615) and MAC #4 (617) generated “will”. For the third token, MAC #1 (611) and MAC #2 (613) generated “going”, MAC #3 (615) generated “be”, and MAC #4 (617) generated “go”. For the fourth token, MAC #1 (611) and MAC #2 (613) generated “to”, MAC #3 (615) generated “there”, and MAC #4 (617) generated “the”. For the fifth token, MAC #1 (611) generated “school”, MAC #2 (613) generated “go”, MAC #3 (615) generated “EOT”, and MAC #4 (617) generated “school”. In this case, the LLM (240), the NPU (213), or the control core (510) may determine that further computational processing for token generation in MAC #3 (615) is meaningless and may block power consumption of the MAC #3 (615). For example, the LLM (240), the NPU (213), or the control core (510) may input an operation blocking command (done signal or skip signal) to a control pin provided in the MAC #3 (615). In addition, the method of blocking power consumption of the MAC #3 (615) may be implemented in various ways as defined above.

[0083] Next, with the sixth token, MAC #4 (617) generated “EOT”, with the seventh token, MAC #1 (611) generated “EOT”, and with the ninth token, MAC #2 (613) generated “EOT”. Accordingly, the LLM (240) or NPU (213) can sequentially cut off power consumption for MAC #4 (617), MAC #1 (611), and MAC #2 (613).

[0084] below shows an example in which the LLM (240), NPU (213), or control core (510) blocks power consumption for four batches included in a multi-batch in response to an EOT generated as in above.

[0085] Blocking Action Command Classification#1#2#3#4#5#6#7#8#9MAC #1000000XXXMAC #200000000XMAC #30000XXXXXMAC #400000XXXX

[0086] In the above , it is shown that MAC #1 (611), MAC #2 (613), MAC #3 (615), and MAC #4 (617) input operation blocking commands to the corresponding MACs based on the time when EOT occurs. For example, the LLM (240), NPU (213), or control core (510) can input operation blocking command C3 to the control pin of MAC #3 (615) in response to EOT occurring with the fifth token, can input operation blocking command C4 to the control pin of MAC #4 (617) in response to EOT occurring with the sixth token, can input operation blocking command C1 to the control pin of MAC #1 (611) in response to EOT occurring with the seventh token, and can input operation blocking command C2 to the control pin of MAC #2 (613) in response to EOT occurring with the ninth token.

[0087] FIG. 7 is a block diagram illustrating an operation configuration for processing a predicted EOT in an electronic device (e.g., the electronic device (100) of FIG. 1) according to one embodiment. As an example, the operation in FIG. 7 may be performed by at least one processor (e.g., the CPU (211), NPU (213), or GPU (215) of FIG. 2) included in the electronic device (100). As an example, the operation in FIG. 7 may be performed by an AI model (e.g., the LLM (240) of FIG. 2) included in the electronic device (100).

[0088] Referring to FIG. 7, the electronic device (100) can monitor tokens (token #i (731), token #i+1 (733), ....., token #t, where i and t are natural numbers, and t is a natural number greater than i) continuously generated by one arbitrary high-speed accumulator included in a plurality of high-speed accumulators (e.g., MAC #1 (560), MAC #2 (570), MAC #3 (580), ......, MAC #N (590) of FIG. 5, where N is a natural number greater than or equal to 2) for a specific batch. The specific batch may be one of a plurality of batches included in a multi-batch. Here, token #t (735) may be a token generated at the current time (720) by the arbitrary high-speed accumulator.

[0089] The electronic device (100) can predict the probability that EOT will occur with token #t+1 (737) to be generated after the current time point (720) by the arbitrary high-speed accumulator. The electronic device (100) can make the prediction of the probability that EOT will occur in response to the generation of a new token #t (735) at the current time point (720). As an example, the electronic device (100) can use the new token #t (735) or previously generated tokens (e.g., token #i (731), token #i+1 (733), ......, token #t) to predict candidate tokens to be generated next (e.g., token #t'+1 (737-1), EOT (737-2), token #t''+1 (737-3), token #t'''+1 (737-4)). The electronic device (100) can predict probability values ​​(739) (e.g., P(1)(739-1), P(2)(739-2), P(3)(739-3), P(4)(739-4)) regarding the possibility of generation for each of the predicted candidate tokens. For example, the electronic device (100) can predict P(1)(739-1), P(2)(739-2), P(3)(739-3), and P(4)(739-4) as probability values ​​corresponding to token #t'+1(737-1), EOT(737-2), token #t''+1(737-3), and token #t'''+1(737-4) predicted as candidate tokens, respectively. For example, if P(1)(739-1), P(2)(739-2), P(3)(739-3), and P(4)(739-4) are predicted as “0.1”, “0.5”, “0.2”, and “0.3”, the electronic device (100) can predict that EOT(737-2) will be generated as the next token because the probability value of EOT(737-2) is the highest. The electronic device (100) can predict that EOT(737-2) will be generated as the next token even if the probability value of EOT(737-2) is above a threshold level.

[0090] If the electronic device (100) predicts that an EOT (737-2) will be generated as the next token, it can notify any high-speed accumulator that performs an operation for the corresponding batch that an EOT will be generated as the next token and that an operation blocking command (e.g., a done signal or a skip signal) for power cutoff can be received. For example, the electronic device (100) can transmit a readiness signal to any high-speed accumulator in advance. This enables the initial communication procedure with the NPU (213) required to receive an operation blocking command after EOT generation to be performed in advance. In addition, when the readiness signal is transmitted to any high-speed accumulator, an operation for compressing and storing the corresponding batch information as DRAM in the internal cache (L1) of the NPU (213) and / or SRAM can be processed in advance. This enables the space of the SRAM to be secured in advance.

[0091] As described above, by providing a readiness signal in advance before outputting an action blocking command (e.g., a done signal or a skip signal), it is possible to reduce the delay that may occur in the initial communication between the NPU (213) and the arbitrary high-speed accumulator, and the probability that a bug may occur as a result.

[0092] FIG. 8 is a block diagram of an electronic device (801) (e.g., electronic device (100) of FIG. 1) within a network environment (800) according to various embodiments.

[0093] Referring to FIG. 8, in a network environment (800), an electronic device (801) may communicate with an electronic device (802) via a first network (898) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (804) or a server (808) via a second network (899) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (801) may communicate with the electronic device (804) via the server (808). According to one embodiment, the electronic device (801) may include a processor (820), a memory (830), an input module (850), an audio output module (855), a display module (860), an audio module (870), a sensor module (876), an interface (877), a connection terminal (878), a haptic module (879), a camera module (880), a power management module (888), a battery (889), a communication module (890), a subscriber identification module (896), or an antenna module (897). In some embodiments, the electronic device (801) may omit at least one of these components (e.g., the connection terminal (878)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (876), the camera module (880), or the antenna module (897)) may be integrated into one component (e.g., the display module (860)).

[0094] The processor (820) may, for example, execute software (e.g., a program (840)) to control at least one other component (e.g., a hardware or software component) of the electronic device (801) connected to the processor (820) and perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculations, the processor (820) may store commands or data received from other components (e.g., a sensor module (876) or a communication module (890)) in a volatile memory (832), process the commands or data stored in the volatile memory (832), and store result data in a non-volatile memory (834). According to one embodiment, the processor (820) may include a main processor (821) (e.g., a central processing unit or an application processor) or a secondary processor (823) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor)) that can operate independently or together therewith. For example, if the electronic device (801) includes a main processor (821) and a secondary processor (823), the secondary processor (823) may be configured to use less power than the main processor (821) or to be specialized for a given function. The secondary processor (823) may be implemented separately from the main processor (821) or as a part thereof.

[0095] The auxiliary processor (823) may control at least a portion of functions or states associated with at least one component (e.g., the display module (860), the sensor module (876), or the communication module (890)) of the electronic device (801), for example, on behalf of the main processor (821) while the main processor (821) is in an inactive (e.g., sleep) state, or together with the main processor (821) while the main processor (821) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (823) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (880) or a communication module (890)). In one embodiment, the auxiliary processor (823) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (801) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (808)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0096] The memory (830) can store various data used by at least one component (e.g., the processor (820) or the sensor module (876)) of the electronic device (801). The data can include, for example, software (e.g., the program (840)) and input data or output data for commands related thereto. The memory (830) can include volatile memory (832) or non-volatile memory (834).

[0097] The program (840) may be stored as software in the memory (830) and may include, for example, an operating system (842), middleware (844), or an application (846).

[0098] The input module (850) can receive commands or data to be used in a component of the electronic device (801) (e.g., a processor (820)) from an external source (e.g., a user) of the electronic device (801). The input module (850) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0099] The audio output module (855) can output audio signals to the outside of the electronic device (801). The audio output module (855) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0100] The display module (860) can visually provide information to an external party (e.g., a user) of the electronic device (801). The display module (860) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (860) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0101] The audio module (870) can convert sound into an electrical signal, or vice versa. According to one embodiment, the audio module (870) can acquire sound through the input module (850), output sound through the sound output module (855), or an external electronic device (e.g., electronic device (802)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (801).

[0102] The sensor module (876) can detect the operating status (e.g., power or temperature) of the electronic device (801) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (876) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0103] The interface (877) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (801) with an external electronic device (e.g., the electronic device (802)). In one embodiment, the interface (877) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0104] The connection terminal (878) may include a connector through which the electronic device (801) may be physically connected to an external electronic device (e.g., the electronic device (802)). According to one embodiment, the connection terminal (878) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0105] The haptic module (879) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (879) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0106] The camera module (880) can capture still images and videos. According to one embodiment, the camera module (880) may include one or more lenses, image sensors, image signal processors, or flashes.

[0107] The power management module (888) can manage power supplied to the electronic device (801). According to one embodiment, the power management module (888) can be implemented, for example, as at least a part of a power management integrated circuit (PMIC).

[0108] A battery (889) may power at least one component of the electronic device (801). In one embodiment, the battery (889) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0109] The communication module (890) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (801) and an external electronic device (e.g., electronic device (802), electronic device (804), or server (808)), and the performance of communication through the established communication channel. The communication module (890) may operate independently from the processor (820) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (890) may include a wireless communication module (892) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (894) (e.g., a local area network (LAN) communication module, or a power line communication module). Any of these communication modules may communicate with an external electronic device (804) via a first network (898) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (899) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (892) may use subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (896) to verify or authenticate the electronic device (801) within a communication network such as the first network (898) or the second network (899).

[0110] The wireless communication module (892) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimizing terminal power and connecting multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (892) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (892) may support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (892) may support various requirements specified in the electronic device (801), an external electronic device (e.g., the electronic device (804)), or a network system (e.g., the second network (899)). According to one embodiment, the wireless communication module (892) may support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0111] The antenna module (897) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (897) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (897) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (898) or the second network (899), may be selected from the plurality of antennas by, for example, the communication module (890). A signal or power may be transmitted or received between the communication module (890) and an external electronic device via the selected at least one antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (897).

[0112] According to various embodiments, the antenna module (897) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band.

[0113] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0114] According to one embodiment, commands or data may be transmitted or received between the electronic device (801) and an external electronic device (804) via a server (808) connected to a second network (899). Each of the external electronic devices (802, or 104) may be the same or a different type of device as the electronic device (801). According to one embodiment, all or part of the operations executed in the electronic device (801) may be executed in one or more of the external electronic devices (802, 104, or 108). For example, when the electronic device (801) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (801) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (801). The electronic device (801) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (801) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In one embodiment, the external electronic device (804) may include an Internet of Things (IoT) device. The server (808) may be an intelligent server using machine learning and / or a neural network. According to one embodiment, the external electronic device (804) or the server (808) may be included in the second network (899).The electronic device (801) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0115] Figure 9 is a block diagram of an exemplary AI system (900) capable of performing the operations described in this document. The AI ​​system (900) may be a generative AI system, but will be referred to as the "AI system (900)" hereinafter.

[0116] Referring to FIG. 9, the AI ​​system (900) may include a User Query / Response Interface (910) (e.g., I / F (220) of FIG. 2) (hereinafter, referred to as 'I / F (910)'), an AI framework (920), a generative AI model (930), a database (940), or an Application / Service Component (950).

[0117] The above I / F (910) can receive input (e.g., data acquired or generated by an electronic device (e.g., the electronic device (100) of FIG. 1 or the electronic device (801) of FIG. 8) (hereinafter referred to as 'electronic device (100)') or user input, etc.). The data acquired or generated by the electronic device (100) can include image or video data generated using a processor (e.g., the processor (110) of FIG. 1 or the processor (820) of FIG. 8), values ​​transmitted through a sensor or sensor hub (e.g., external illuminance, an angle of the terminal, a display (e.g., the display (140) of FIG. 1) or the temperature of the electronic device (100), display (140) size or expansion / reduction information, a captured image of an image sensor (e.g., the image sensor (150) of FIG. 1), etc.). The user input may be in the form of natural language, touch coordinates or stylus coordinates obtained through a touch panel or digitizer included in the display (140), images, and / or videos. In addition, context information may also be transmitted when the user input is transmitted. The context information may include various additional information at the time of the user input. The additional information may include, for example, information on the application currently being used by the user or information on the user's location. In addition, the user input may also be in the form of a mixture of the above-described natural language, images, sounds, and context information. In addition, the user input may also be in the form of a non-natural language, such as selecting a menu. The I / F (910) may provide the user with the result of analyzing the AI ​​system (900) and / or the input as an output. The output may be in the form of natural language or a specific content. The output may also be provided in the form of an action requested by the user. The output may also be provided in the form of a specific value specified by the user. The above I / F (910) can output the results of the generative AI system (900) to the user.The output may be in natural language or in a specific content format. The output may also be provided in the form of a user-requested action, etc.

[0118] The AI ​​framework (920) can receive user input and coordinate and control each component necessary to carry out the user's intention based on the user's query. For example, the AI ​​framework (920) can include a prompt design component (921), an API / plug-in management component (923), or an output modification component (or refiner component) (925).

[0119] The user input received from the I / F (910) may be transmitted to the prompt design component (921). The prompt design component (921) may be used to generate a prompt (e.g., the prompt (310) of FIG. 3) suitable for inputting the user input into the generative AI model (930) (e.g., LLM, LVM (large vision model), or LMM (large multimodal model)). The prompt design component (921) may be an AI component that uses a machine learning algorithm or a neural network to develop a better prompt (310) over time. The prompt design component (921) may access user preference data (943), a prompt library (941), and a knowledge component including prompt examples based on the user input to generate the prompt (310), and transmit the generated prompt (310) to the generative AI model (930).

[0120] The API / plug-in management component (923) may communicate with external information when a request for additional information is made when transmitting user input as input to the generative AI model (930). The API / plug-in management component (923) establishes a channel for communicating with the outside of the AI ​​interface through an API, and enables access to various data sources (e.g., knowledge repositories (945)) through the established channel.

[0121] The API / plug-in management component (923) may request the application / service component (923) to perform an action that ultimately requires user input, rather than an intermediate result, through an API when the action needs to be performed by the application or service. Information obtained from an external source may be used to generate a prompt (310) in the prompt design component (921) along with user input, or may be passed as input to the generative AI model (930).

[0122] The output adjustment component (925) (also referred to as a refiner component) can fine-tune or reprocess the results output from the generative AI model (930). The output adjustment component (925) can verify, for example, whether the content generated through the generative AI model (930) is irrelevant, does not contain biased content, or does not contain harmful content. The output adjustment component (925) can also determine to what extent the content matches the result desired by the user and, if additional processing is required, can proceed with the process. The output adjustment component (925) can additionally configure and provide the user with hints to avoid unwanted output.

[0123] The generative AI model (930) may generally refer to an AI neural network that creates new types of data based on user input information. The generative AI model (930) may include a model that generates images and / or a model that generates language. The model that generates images may include, for example, a generative adversarial network (GAN) or a variational auto encoder (VAE). The model that generates images may be, for example, a diffusion-based AI model that uses a VAE and a transformer structure. The model that generates language may be a model trained to output the most statistically appropriate output value based on an input value. Representative examples thereof include models such as CHAT-GPT 3 and CHAT-GPT 4. In addition, there is also an LMM as an AI model (930) that can recognize various types of data input such as text, images, voices, and videos and generate new data corresponding thereto.

[0124] The technical problems to be achieved in the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by a person having ordinary knowledge in the technical field to which the present disclosure pertains.

[0125] According to one example, the electronic device (100) may include a memory (230) including one or more storage media for storing instructions, and at least one processor (210) including processing circuitry. The instructions, when individually or collectively executed by the at least one processor (210), may cause the electronic device (100) to perform at least one operation. The at least one operation may include generating tokens in multiple batches in a plurality of high-speed accumulators (MACs) (560, 570, 580, 590). The at least one operation may include identifying that token generation has reached an end of token (EOT) in at least one of the plurality of high-speed accumulators. The operation may include controlling power consumption of the identified MAC.

[0126] According to one example, when the instructions are individually or collectively executed by at least one processor (210), the electronic device (100) may perform an operation of inputting an operation blocking command (done signal or skip signal) to a control pin provided in the identified MAC to prevent the identified MAC from processing a matrix process.

[0127] In one example, the instructions, when individually or collectively executed by at least one processor (210), may cause the electronic device (100) to perform an operation of stopping weight loading into the identified MAC to block power consumption in the identified MAC.

[0128] In one example, the instructions, when executed individually or collectively by at least one processor (210), may cause the electronic device (100) to perform an action of physically cutting off power to the identified MAC.

[0129] In one example, the instructions, when executed individually or collectively by at least one processor (210), may cause the electronic device (100) to perform an operation of brokering a matrix process in the plurality of high-speed accumulators (MACs) (560, 570, 580, 590) in response to an end of token (EOT) generation.

[0130] In one example, when the instructions are individually or collectively executed by at least one processor (210), the electronic device (100) may cause each of the plurality of high-speed accumulators (MACs) (560, 570, 580, 590) to predict an occurrence probability of the end of token (EOT) and, taking into account the predicted occurrence probability, transmit a readiness signal prior to powering off the identified high-speed accumulators (MACs).

[0131] According to one example, a storage medium (230) storing computer-readable instructions may cause the instructions, when executed by at least a portion of at least one processor (210) of the electronic device (100), to cause the electronic device (100) to perform at least one operation. The at least one operation may include: generating tokens in multiple batches in a plurality of high-speed accumulators (MACs) (560, 570, 580, 590); identifying that token generation has been completed (end of token, EOT) in at least one of the plurality of high-speed accumulators; and controlling power consumption of the identified high-speed accumulator (identified MAC).

[0132] In one example, the at least one operation may include: inputting an operation blocking command (done signal or skip signal) to a control pin provided in the identified MAC to prevent the identified MAC from processing a matrix process.

[0133] In one example, the at least one operation may include: stopping weight loading into the identified MAC to block power consumption in the identified MAC.

[0134] In one example, the at least one operation may include: physically cutting off power to the identified MAC.

[0135] In one example, the at least one operation may include: brokering a matrix process in the plurality of high-speed accumulators (MACs) (560, 570, 580, 590) in response to an end of token (EOT) occurrence.

[0136] In one example, the at least one operation may include: predicting a probability of occurrence of the end of token (EOT) for each of the plurality of high-speed accumulators (MACs) (560, 570, 580, 590); and transmitting a readiness signal prior to powering off the identified high-speed accumulators (MACs) based on the predicted probability of occurrence.

[0137] According to an example, an operating method of an electronic device (100) may include: generating tokens in multiple batches from a plurality of high-speed accumulators (MACs) (560, 570, 580, 590); identifying that token generation is complete (end of token, EOT) from at least one of the plurality of high-speed accumulators; and controlling power consumption of the identified high-speed accumulator (identified MAC).

[0138] According to one example, the method may include an operation of inputting an operation blocking command (done signal or skip signal) to a control pin provided in the identified MAC to prevent the identified MAC from processing a matrix process.

[0139] In one example, the method may include an operation of stopping weight loading into the identified MAC to block power consumption in the identified MAC.

[0140] In one example, the method may include physically cutting off power to the identified MAC.

[0141] In one example, the method may include an operation of brokering a matrix process in the plurality of high-speed accumulators (MACs) (560, 570, 580, 590) in response to an end of token (EOT) occurrence.

[0142] According to one example, the method may include an operation of predicting an occurrence probability of the end of token (EOT) for each of the plurality of high-speed accumulators (MACs) (560, 570, 580, 590); and an operation of transmitting a readiness signal prior to powering off the identified high-speed accumulators (MACs) in consideration of the predicted occurrence probability.

[0143] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned will be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains.

[0144] It should be understood that the embodiments of this document and the terminology used herein are not intended to limit the technical features described in this document to a specific embodiment, but include various modifications, equivalents, or substitutes of the embodiment. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the item, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0145] The term "module" used in one embodiment of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0146] An embodiment of the present document may be implemented as software including one or more instructions stored in a storage medium (e.g., memory (230)) readable by a machine (e.g., electronic device (100)). For example, a processor (e.g., processor (210)) of the machine (e.g., electronic device (100)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0147] According to one embodiment, the method according to one embodiment disclosed in the present document may be provided as a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0148] According to one embodiment, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to one embodiment, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to one embodiment, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device (100), A memory (230) including one or more storage media for storing instructions; and At least one processor (210) comprising a processing circuit, Here, the instructions, when individually or collectively executed by at least one processor (210), cause the electronic device (100) to perform at least one operation, At least one of the above actions, Generate tokens in multi-batch on multiple high-speed MACs (560, 570, 580, 590), Identifying that token generation has been completed (end of token, EOT) in at least one of the above multiple high-speed accumulators, An electronic device (100) comprising an operation for controlling power consumption of the identified high-speed accumulator (MAC).

2. In paragraph 1, When the above instructions are individually or collectively executed by at least one processor (210), the electronic device (100) causes: An electronic device (100) that causes an operation of inputting an operation blocking command (done signal or skip signal) to a control pin provided in the identified high-speed accumulator (MAC) to prevent the identified high-speed accumulator (MAC) from processing a matrix process.

3. In paragraph 1 or 2, When the above instructions are individually or collectively executed by at least one processor (210), the electronic device (100) causes: An electronic device (100) that causes an operation to be performed to stop loading weights into the identified MAC in order to block power consumption in the identified MAC.

4. In any one of paragraphs 1 to 3, When the above instructions are individually or collectively executed by at least one processor (210), the electronic device (100) causes: An electronic device (100) that causes a plurality of high-speed accumulators (MACs) (560, 570, 580, 590) to perform an operation of blocking a matrix process in response to an end of token (EOT) generation.

5. In any one of paragraphs 1 to 4, When the above instructions are individually or collectively executed by at least one processor (210), the electronic device (100) causes: Predict the occurrence probability of the end of token (EOT) for each of the above multiple high-speed accumulators (MAC) (560, 570, 580, 590), An electronic device (100) that causes an operation of transmitting a readiness signal prior to blocking power consumption of the identified high-speed accumulator (MAC) in consideration of the predicted occurrence probability.

6. In a storage medium (230) that stores instructions that can be read by a computer, The above instructions, when executed by at least a part of at least one processor (210) of the electronic device (100), cause the electronic device (100) to perform at least one operation; At least one of the above actions: The operation of generating tokens in multi-batches on multiple high-speed MACs (560, 570, 580, 590); An operation for identifying that token generation has been completed (end of token, EOT) in at least one of the above multiple high-speed accumulators; and A recording medium (230) comprising an operation for controlling power consumption of the identified high-speed accumulator (MAC).

7. In paragraph 6, At least one of the above actions: A recording medium (230) including an operation of inputting an operation blocking command (done signal or skip signal) to a control pin provided in the identified high-speed accumulator (MAC) to prevent the identified high-speed accumulator (MAC) from processing a matrix process.

8. In paragraph 6 or 7, At least one of the above actions: A recording medium (230) comprising an operation of stopping weight loading into the identified MAC to block power consumption in the identified MAC.

9. In any one of paragraphs 6 to 8, At least one of the above actions: A recording medium (230) comprising an operation of blocking a matrix process in a plurality of high-speed accumulators (MACs) (560, 570, 580, 590) in response to an end of token (EOT) generation.

10. In any one of paragraphs 6 to 9, At least one of the above actions: An operation of predicting the occurrence probability of the end of token (EOT) for each of the above multiple high-speed accumulators (MACs) (560, 570, 580, 590); and A recording medium (230) comprising an operation of transmitting a readiness signal prior to blocking power consumption of the identified high-speed accumulator (MAC) in consideration of the predicted occurrence probability.

11. In the operating method of an electronic device (100), The operation of generating tokens in multi-batches on multiple high-speed MACs (560, 570, 580, 590); An operation for identifying that token generation has been completed (end of token, EOT) in at least one of the above multiple high-speed accumulators; and A method comprising an operation for controlling power consumption of the identified high-speed accumulator (MAC).

12. In paragraph 11, A method comprising an operation of inputting an operation blocking command (done signal or skip signal) to a control pin provided in the identified high-speed accumulator (MAC) to prevent the identified high-speed accumulator (MAC) from processing a matrix process.

13. In paragraph 11 or 12, A method comprising an operation of stopping weight loading into the identified MAC to block power consumption in the identified MAC.

14. In any one of paragraphs 11 to 13, A method comprising an operation of blocking a matrix process in a plurality of high-speed accumulators (MACs) (560, 570, 580, 590) in response to an end of token (EOT) occurrence.

15. In any one of paragraphs 11 to 14, An operation of predicting the occurrence probability of the end of token (EOT) for each of the above multiple high-speed accumulators (MACs) (560, 570, 580, 590); and A method comprising an operation of transmitting a readiness signal prior to blocking power consumption of the identified high-speed accumulator (MAC) considering the predicted occurrence probability.

Images