Apparatus and method for executing a program including a plurality of operators

Abstract

Methods and apparatus for executing a program (e.g., a neural network inference or a deep learning model) comprising a plurality of operators are provided. The apparatus can include an executor and an analyzer. The executor is configured to execute the program with at least a first operator of the plurality of operators, wherein the first operator is loaded onto a second memory from a first memory storing the plurality of operators, the executor generates an indication signal based on a progress of executing the program with the first operator. The analyzer is coupled to the executor, the first memory and the second memory, configured to load at least a second operator stored in the first memory to the second memory based on the indication signal from the executor and an execution plan stored in the second memory before the executor completes executing the program with the first operator.

Description

Technical Field

[0001] The present invention relates to memory, and more particularly, to apparatus and methods for dynamically loading neural network inference stored in memory and for managing the power supplied to the memory and the hardware accelerator that performs the neural network inference. Background Technology

[0002] Unless otherwise stated herein, the methods described in this section do not constitute prior art with respect to the listed claims, nor are they considered prior art by virtue of being included in this section.

[0003] During a process called training, a neural network (NN) learns from various examples of a specific task. After learning, the task can be performed on new data during a process called inference. NN inference involves a large number of weights and activations, which need to be stored in sufficiently large memory, such as dynamic random access memory (DRAM). During the entire execution of NN inference, the DRAM consumes a significant amount of power while powered on. Summary of the Invention

[0004] Various aspects of the present invention provide an apparatus and method for executing a program comprising a plurality of operators. The apparatus may include an executor and an analyzer. The executor is configured to execute the program with at least a first operator of the plurality of operators, wherein the first operator is loaded from a first memory storing the plurality of operators into a second memory, and the executor generates an indication signal based on the progress of executing the program with the first operator. The analyzer is coupled to the executor, the first memory, and the second memory, and is configured to load at least a second operator stored in the first memory into the second memory based on the indication signal from the executor and an execution scheme stored in the second memory, before the executor completes execution of the program with the first operator.

[0005] In some embodiments, the apparatus further includes an estimator coupled to an analyzer, a first memory, and a second memory, wherein the analyzer controls the estimator to manage the power applied to the first memory based on the signals from the actuator and an execution scheme stored in the second memory, before the actuator completes execution of the program with a first operator. For example, the estimator can manage the power supplied to the first memory by powering on / off the first memory, configuring the operating mode of the first memory, or scaling the voltage and / or frequency applied to the first memory.

[0006] Various aspects of the present invention provide another apparatus for executing a program comprising multiple operators. The apparatus includes a first actuator, a second actuator, and an analyzer. The first actuator is configured to execute the program using at least a first operator of the multiple operators and to generate an indication signal based on the progress of executing the program using the first operator. The second actuator is configured to execute the program using at least a second operator. An analyzer coupled to the first and second actuators is configured to manage power supplied to the second actuator before the first actuator completes executing the program with the first operator, based on the execution scheme and signals from the first actuator. For example, the second actuator may be a DLA or an accelerated processing unit (APU).

[0007] In some embodiments, the apparatus further includes an estimator coupled to an analyzer and a second actuator, the estimator being configured to manage the power supplied to the second actuator by powering on / off the second actuator, configuring the operating mode of the second actuator, or scaling the voltage and / or frequency applied to the second actuator.

[0008] Various aspects of the present invention provide a method for executing a program comprising a plurality of operators. The method includes loading at least a first operator of the plurality of operators from a first memory storing the plurality of operators into a second memory; executing the program with the first operator loaded in the second memory and generating a signal based on the execution progress of the program; and loading at least a second operator of the plurality of operators stored in the first memory into the second memory before the execution of the program using the first operator is completed, based on the signal and an execution scheme stored in the second memory.

[0009] In some embodiments, the method further includes managing the power applied to the first memory based on the signal and the execution scheme stored in the second memory before the execution of the program with the first operator is completed.

[0010] In some embodiments, the first operator and the second operator are executed by a first actuator and a second actuator, respectively. Based on the execution scheme and the signals from the first actuator, the power applied to the second actuator is managed before the first actuator completes executing the program with the first operator.

[0011] In this invention, loading the second operator into the second memory before the executor completes the execution program with the first operator can reduce the delay of the execution program.

[0012] These and other objects of the invention will be readily understood by those skilled in the art upon reading the following detailed description of the preferred embodiments illustrated in the accompanying drawings. A detailed description will be given in the following embodiments with reference to the accompanying drawings. Attached Figure Description

[0013] A more complete understanding of the invention can be obtained by reading the following detailed description and referring to the examples given in the accompanying drawings, wherein: the included drawings are used to provide a further understanding of the embodiments of the present disclosure, and the drawings are incorporated into and constitute a part of the embodiments of the present disclosure. The drawings illustrate implementations of the embodiments of the present disclosure and, together with the description, serve to explain the principles of the embodiments of the present disclosure. It will be understood that the drawings are not necessarily drawn to scale, as some components may be shown out of scale to clearly illustrate the concepts of the embodiments of the present disclosure. Various embodiments of the invention presented as examples will be described in detail with reference to the following drawings, wherein the same reference numerals denote the same components.

[0014] Figure 1A This is a power timing diagram illustrating the execution of a DL model on a TCM (or SRAM).

[0015] Figure 1B This is a power timing diagram illustrating the execution of a DL model on a combination of TCM (or SRAM) and DRAM.

[0016] Figure 2 This is a power timing diagram illustrating the execution of a DL model on a combination of TCM and DRAM according to some embodiments of the present disclosure.

[0017] Figure 3 This is a functional block diagram of an exemplary apparatus for executing a program (e.g., a DL model) according to some embodiments of the present disclosure.

[0018] Figure 4A This is a power timing diagram of an exemplary hardware accelerator (such as a mini DLA and APU) operating in working mode.

[0019] Figure 4B This is a power timing diagram of an exemplary hardware accelerator switching from standby mode to operating mode.

[0020] Figure 5 This is a power timing diagram of an exemplary hardware accelerator (e.g., a mini DLA and an APU) switching from standby mode to operating mode according to some embodiments of this disclosure.

[0021] Figure 6 This is a functional block diagram of an exemplary apparatus for executing a program (e.g., a DL model) according to some embodiments of the present disclosure.

[0022] Figure 7 This is a flowchart illustrating an exemplary method for executing a program involving multiple operators according to some embodiments of the present disclosure.

[0023] In the following detailed description, numerous specific details are set forth for illustrative purposes so that those skilled in the art can more thoroughly understand the embodiments of the invention. However, it will be apparent that one or more embodiments may be practiced without these specific details, and different embodiments may be combined as needed, and should not be limited to the embodiments illustrated in the accompanying drawings. Detailed Implementation

[0024] The following description illustrates preferred embodiments of the present invention, and is intended only to exemplify the technical features of the invention, not to limit the scope of the invention. Throughout this specification and claims, certain terms are used to refer to specific elements. Those skilled in the art should understand that manufacturers may use different names for the same element. Therefore, this specification and claims do not distinguish elements by differences in name, but rather by differences in function. The terms "element," "system," and "device" used in this invention can refer to computer-related entities, where the computer can be hardware, software, or a combination of hardware and software. The terms "comprising" and "including" as used in the following description and claims are open-ended terms and should be interpreted as "comprising, but not limited to...". Furthermore, the term "coupled" refers to an indirect or direct electrical connection. Therefore, if a device is described as coupled to another device, it means that the device can be directly electrically connected to the other device, or indirectly electrically connected to the other device through other devices or connection means.

[0025] Unless otherwise indicated, the corresponding numbers and symbols in the various figures generally refer to the corresponding parts. The figures are drawn to clearly illustrate the relevant parts of the embodiments and are not necessarily drawn according to this example.

[0026] The terms "basically" or "roughly" as used in this document mean that, within an acceptable range, a person skilled in the art can solve the technical problem to be solved and basically achieve the desired technical effect. For example, "roughly equal to" means a method that a person skilled in the art can accept with a certain margin of error from "exactly equal to" without affecting the correctness of the result.

[0027] This specification discloses detailed embodiments and implementations of the claimed subject matter. However, it should be understood that the disclosed embodiments and implementations are merely illustrative of the claimed subject matter, which can be embodied in various forms. For example, embodiments of this disclosure can be implemented in many different forms and should not be construed as limited to the exemplary embodiments and implementations set forth herein. Rather, these exemplary embodiments and implementations are provided so that the description of embodiments of this disclosure is thorough and complete, and will fully convey the scope of embodiments of this disclosure to those skilled in the art. In the following description, details of well-known features and technologies may be omitted to avoid unnecessarily obscuring the presented embodiments and implementations.

[0028] Neural networks (NNs) are used to extract meaningful information or features from large amounts of data. Without being specifically programmed, a neural network (NN) can learn from various samples of a specific task during a process called training. After learning, the task can be performed on new data during a process called inference. During training, a set of optimal weights that optimize the accuracy of NN inference can be estimated. A neural network (NN) is a program that includes a large number of weights and activations; therefore, NN inference is too large to be stored in, for example, tightly-coupled memory (TCM), which has limited bandwidth. For example, NN inference can be stored in dynamic random access memory (DRAM), and the NN inference will be loaded portion-by-portion from DRAM into TCM for execution by the inference executor. Throughout the loading and execution of NN inference, DRAM remains powered on, consuming a significant amount of power.

[0029] Neural network inference is also too large to be performed by a single general-purpose executor. For example, neural network inference can be partitioned into multiple partitions and distributed across heterogeneous hardware (such as mini DLAs and APUs) to accelerate inference computation. During neural network inference, the hardware accelerator can operate in work mode. To conserve power, the hardware accelerator can operate in standby mode and switch to work mode when performing neural network inference. Switching from standby mode to work mode introduces additional latency.

[0030] According to some aspects of this disclosure, an analyzer and an execution scheme can be introduced. The execution scheme can be manually created or created by an offline optimizer or an online / runtime optimizer. In one embodiment, the execution scheme can be used to tell the analyzer when and how to load a portion of a program (e.g., an operator of a deep learning (DL) model) from DRAM to the TCM and manage the power supplied to the DRAM. For example, when the analyzer looks up the execution scheme and learns that an operator of the DL model needs to be loaded from DRAM to the TCM, the analyzer can turn on the DRAM, load the operator of the DL model from the DRAM to the TCM, and turn off the DRAM after the operator of the DL model has been loaded into the TCM. In one embodiment, the operator of the DL model can be loaded into the TCM before the execution of the DL model using the previous operator is complete, to reduce latency.

[0031] In another embodiment, the execution plan can also tell the analyzer when to switch the hardware accelerator from standby mode to active mode. For example, when the analyzer looks up the execution plan and learns that the hardware accelerator is about to use an operator to execute a DL model, the analyzer can activate the hardware accelerator before the previous DL model execution using the operator is complete, in order to reduce latency.

[0032] Deep learning (DL) has achieved tremendous success in various applications of machine learning, such as robotics, autonomous vehicles, augmented reality, natural language processing, and big data analytics. DL models consist of various types of deep neural networks (DNNs), such as fully connected neural networks (FCNNs), convolutional neural networks (CNNs), autoencoders (AEs), generative adversarial networks (GANs), and recurrent neural networks (RNNs).

[0033] In FCNN, all neurons (or nodes) in one layer are connected to neurons in the next layer. Therefore, the output of each layer, the multi-layer perceptron (MLP), is fed into the next layer as its input. FCNN is a supervised artificial neural network used for image recognition and classification. However, FCNN is computationally expensive and prone to overfitting.

[0034] CNNs add additional filtering layers, where, in addition to the weights and biases of each neuron, filter weights (or convolution kernels) are learned. CNNs also add pooling layers, which compute the maximum / minimum or average value of a certain number of samples to reduce the dimensionality added by the convolution operations of the filtering layers. CNNs can extract features while reducing model complexity, thus reducing the risk of overfitting and have found significant applications in image processing.

[0035] An Artificial Neural Network (AE) is an unsupervised artificial neural network (NN) used to learn efficient encodings for unlabeled data. An AE consists of two stacked NNs. The first NN (the encoder) learns or encodes representative features of the input. The second NN (the decoder) takes these encoded representative features as input and reconstructs the original input approximately as the final output. AEs can be used for dimensionality reduction and information retrieval.

[0036] Generative Adversarial Networks (GANs) are also a type of unsupervised artificial neural network (NN). They treat unsupervised problems as being supervised by two sub-models: a generator and a discriminator. The generator is trained to generate samples from the training data, while the discriminator is trained to distinguish whether the generated samples are real or fake. These two sub-models are trained together adversarially in a zero-sum game until a Nash equilibrium is found when the discriminator is fooled half the time, meaning the generator is generating plausible samples. GANs can be applied to image synthesis, including synthesizing near-perfect faces, restoring color and quality from old videos, and generating realistic deepfake videos.

[0037] RNNs are designed specifically for sequential data prediction. Each neuron in an RNN receives information not only from the previous layer but also from its own preceding channel. RNNs can predict future information or recover missing parts of sequential data. RNNs are commonly used for sequential or temporal problems such as language translation, natural language processing (NLP), speech recognition, and image captioning, and have been incorporated into popular applications such as Siri, voice search, and Google Translate.

[0038] To further improve accuracy, DNNs have become deeper, requiring extremely intensive computation and very large memory footprints for numerous weights and biases. A common approach to meet these computational and storage requirements is to leverage cloud computing. However, training and inferring DL models in the cloud requires devices to transmit large amounts of data to the cloud, consuming significant network bandwidth. Real-time inference is crucial for many applications. For example, voice-based intelligent assistant applications, such as Siri, need to quickly parse and understand user queries and return responses. However, sending data to the cloud and accessing cloud services (such as for inference and training) can result in additional queuing and propagation latency from the network, which may not be sufficient to meet the stringent low-latency requirements of real-time interactive applications. Furthermore, users who own data risk privacy by sending it to the cloud.

[0039] Edge computing is a solution that addresses the aforementioned challenges of cost, latency, and privacy because edge devices are closer to the user than the cloud. For example, by being closer to data sources on end devices such as smartphones, wearables, and Internet of Things (IoT) devices, edge computing can reduce latency, enabling real-time services such as real-time inference. As another example, edge computing allows data to be analyzed closer to the data source, thus avoiding exposure to privacy and security attacks.

[0040] However, there are insufficient resources in terminal devices to support raw, large-scale deep learning models. To meet cost and latency requirements, various mechanisms and architectures have been proposed, such as model design, model compression, hardware selection, model partitioning, and model frameworks, to enable terminal devices to perform inference quickly.

[0041] MobileNets (e.g., MobileNet V2) are an exemplary model design aimed at reducing memory and execution latency in mobile embedded vision applications. Spatially separable convolutions can split a large kernel into two smaller kernels. For example, a 3x3 kernel can be split into a 3x1 kernel and a 1x3 kernel. Therefore, instead of performing a single convolution on the 3x3 kernel, two simple convolutions can be performed, one on the 3x1 kernel and one on the 1x3 kernel (a total of 6 multiplications), to achieve the same result. One of the most well-known spatially separable convolutions is the Sobel kernel, which can be used for edge detection. However, not all kernels can be precisely split into two smaller kernels. Depthwise separable convolutions can handle kernels that cannot be perfectly separable into two smaller kernels. MobileNets are based on a streamlined architecture that uses depthwise separable convolutions to decompose or separate convolutional filters into two simpler operations to reduce the amount of computation required. Depthwise separable convolution involves sequentially performed depthwise (DW) convolution and pointwise (PW) convolution. Ordinary convolution convolves a 12x12x3 image (where 3 is the number of channels in the input image, e.g., red (R), green (G), and blue (B)) with 256 5x5x3 kernels (256 is the number of channels in the output image) to obtain an 8x8x256 image. Unlike ordinary convolution, depthwise separable convolution first convolves the 12x12x3 image with three 5x5x1 kernels in the depthwise convolution to obtain an intermediate 8x8x3 image, and then convolves the intermediate 8x8x3 image with 256 1x1x3 kernels in the pointwise convolution to obtain an 8x8x256 image. In a regular convolution, 256 5x5x3 kernels each move 8x8 times, resulting in 256x5x5x3x8x8 = 1,228,800 multiplications. In contrast, in a depthwise separable convolution, 3 5x5x1 kernels each move 8x8 times, and 256 1x1x3 kernels each move 8x8 times, resulting in 3x5x5x1x8x8 + 256x1x1x3x8x8 = 53,952 multiplications, far fewer than 1,228,800 multiplications.

[0042] DNN models can be compressed in several ways, such as parameter quantization (lower precision), parameter pruning (fewer weights), and knowledge distillation. Parameter quantization approximates an existing DNN using floating-point numbers by using a new DNN with low-bitwidth numbers, thus reducing storage requirements and computational costs. For example, the activation function of each convolutional layer in a DNN can be replaced with a quantization function, immediately converting the activation to binary values ​​+1 and -1 in a low-bitwidth binary neural network (BNN) before each convolution. Parameter pruning involves removing redundant or less important parameters (e.g., weights) from the DNN that contribute little to the output, such as those very close to 0. Parameter pruning will make the DNN smaller while aiming to maintain the accuracy of the initially larger DNN. Knowledge distillation involves creating a smaller DNN that can imitate the behavior of a larger, more powerful DNN. For example, a small DNN can be precisely trained step-by-step to do what it should do by using the output predictions generated from a larger DNN that has already been trained.

[0043] In addition to existing central processing units (CPUs) and graphics processing units (GPUs), accelerated processing units (APUs), deep learning accelerators (DLAs, such as mini DLAs), custom application-specific integrated circuits (ASICs), and field-programmable gate array (FPGA)-based DNN accelerators have been developed to accelerate deep learning inference. APUs combine CPUs and GPUs on a single chip to form a combined processing unit, reducing cost and increasing efficiency. By providing dedicated processing units, DLAs can be designed and optimized to improve the speed, efficiency, and accuracy of computers and mobile devices executing deep learning algorithms. Custom ASICs can focus on efficient memory access to reduce latency and power consumption. FPGAs can deliver fast computation while maintaining reconfigurability. Custom ASICs and FPGA-based DNN accelerators are generally more energy-efficient than CPUs and GPUs. Hardware selection metrics can be based on accuracy, power consumption, throughput, and cost.

[0044] DL models can be partitioned into multiple partitions and then distributed across heterogeneous hardware, such as mini DLAs and APUs, to accelerate DL computation by reducing the resource cost of each partition. For example, each layer of a DNN can be sliced ​​to increase parallelism and reduce memory footprint, and these slices can be executed layer by layer.

[0045] Running DL models on static random access memory (SRAM) or tightly-coupled memory (TCM) can achieve better energy savings compared to dynamic random access memory (DRAM). TCM is one of many on-chip memories that can typically be embedded in low-power processors to accelerate data processing within the processor.

[0046] Figure 1AThis is a power timing diagram illustrating the execution of DL model 100 on TCM (or SRAM). Execution of DL model 100 begins at time t1 and ends at time t6. DL model 100 involves multiple operators stored in the TCM, such as operators OP1 to OP5. The executor can execute operators OP1 to OP5 of DL model 100 sequentially from time t1 to time t5. For example, DL model 100 may include MobileNet V2, and operators OP1 to OP5 may include 3x3 convolution, 1x1 convolution, 3x3 DW convolution, 1x1 convolution, and fully convolution (FC), respectively. Executing DL model 100 on the TCM consumes power p1, and therefore consumes p1 x (t6 - t1) of energy.

[0047] Although TCMs are energy-efficient and can be accessed quickly, they are area expensive, making it uneconomical to embed large TCMs into processors. To address this issue, operators OP1 to OP5 can be stored in DRAM, and the operators OP1 to OP5 scheduled for execution can be loaded from DRAM into the TCM for the executor to run.

[0048] Figure 1BThis is a power timing diagram illustrating the execution of DL model 100 on a combination of TCM (or SRAM) and DRAM. For example, operators OP1 and OP2 are already stored in the TCM, which is large enough to store a maximum of two operators, while operators OP3 through OP5 remain stored in DRAM. The actuator can begin executing operators OP1 and OP2 of DL model 100 at times t1 and t2, respectively. After executing DL model 100 with operators OP1 and OP2, i.e., at time t3, the actuator is scheduled to execute operators OP3 and OP4 of DL model 100 and begins loading operators OP3 and OP4 from DRAM into the TCM. Loading operators from DRAM into the TCM takes time. For example, it takes time Δt for operators OP3 and OP4 to be loaded from DRAM into the TCM, and the actuator cannot begin executing operator OP3 of DL model 100 until time t3 + Δt when operator OP3 is loaded into the TCM. After executing operators OP3 and OP4 of DL model 100, i.e., at time t5+Δt, the executor is scheduled to execute operator OP5 of DL model 100 and begins loading operator OP5 from DRAM to TCM. Similarly, loading operator OP5 from DRAM to TCM takes time Δt, and the executor will execute operator OP5 at time t5+2Δt. Therefore, the execution of DL model 100 on the combination of TCM and DRAM begins at time t1 and ends at time t6+2Δt, with a delay (i.e., t6-t1) adding an additional 2Δt. During the loading and execution of operators OP3 to OP5 of DL model 100, DRAM remains powered on and continuously consumes, for example, power p2. Therefore, DRAM is not powered off until time t6+2Δt, and the execution of DL model 100 on the combination of TCM and DRAM consumes energy of p1x(t6-t1)+p1x2Δt+p2x(t6-t3+2Δt).

[0049] Figure 2 This is a power timing diagram illustrating the execution of DL model 100 on a combination of TCM and DRAM according to some embodiments of the present disclosure. For example, operators OP1 and OP2 are already stored in the TCM. To avoid adding latency (i.e., t6-t1), the DRAM is powered on no later than time t3-Δt, at which time the actuator can begin loading operators OP3 and OP4 from the DRAM into the TCM, and operators OP3 and OP4 can be stored in the TCM no later than time t3; the DRAM is powered on no later than time t5-Δt, at which time the actuator can begin loading operator OP5 from the DRAM into the TCM, and operator OP5 can be stored in the TCM no later than time t5. Therefore, the actuator can immediately execute DL model 100 with operators OP3 and OP5 after completing the execution of DL model 100 with operators OP2 and OP4 respectively, and the execution of DL model 100 can end at time t6 without incurring additional latency.

[0050] To conserve energy, the DRAM can be shut down immediately after the operators are loaded and stored in the TCM. For example, the DRAM is powered on at time t3-Δt, at which point operators OP3 and OP4 begin loading from the DRAM into the TCM. The DRAM is powered off at time t3, at which point the loading of operators OP3 and OP4 from the DRAM into the TCM ends. The DRAM is powered on again at time t5-Δt, at which point operator OP5 begins loading from the DRAM into the TCM. The DRAM is powered off again at time t5, at which point the loading of operator OP5 from the DRAM into the TCM ends. Therefore, executing DL model 100 on the combination of TCM and DRAM consumes energy p1x(t6-t1)+p2x2Δt. Figure 1B The energy consumed by executing DL model 100 shown is less than p1x2Δt+p2x(t6-t3).

[0051] Figure 3 This is a functional block diagram of an exemplary device 300 for executing a program (e.g., DL model 100) according to some embodiments of the present disclosure. For example, device 300 may be a smartphone, wearable device, or any Internet of Things (IoT) device. In one embodiment, device 300 may include an actuator 310 (e.g., an inference actuator), an analyzer 320 coupled to the actuator 310, and an estimator 321 coupled to the analyzer. The actuator 310, analyzer 320, and estimator 321 may include software and / or hardware. In one embodiment, the actuator 310, analyzer 320, and estimator 321 may be included on a single chip. In another embodiment, the actuator 310, analyzer 320, and estimator 321 may be three different chips. For example, the actuator 310 may be a CPU or a DL, and the analyzer 320 may be a microcontroller unit (MCU) or a CPU. As another example, executor 310 may be a CPU with an interrupt descriptor table. When an exception occurs and the CPU interrupts its current operation, analyzer 320 may execute a specific exception handler function listed in the interrupt descriptor table.

[0052] In one embodiment, analyzer 320 and estimator 321 may be coupled between first memory 330 and second memory 340. In one embodiment, first memory 330 may be used to store programs, such as a DL model involving multiple operators OP1 to OP5, which may be, for example, 3x3 convolution, 1x1 convolution, 3x3 DW convolution, 1x1 convolution, and FC. For example, first memory 330 may be DRAM, on-bus SRAM, or serial flash memory. In another embodiment, second memory 340 may be used to store at least one of the operators OP1 to OP5 loaded from first memory 330. For example, second memory 340 may be TCM or SRAM.

[0053] The executor 310 may be further coupled to the second memory 340. The executor 310 may execute the DL model 100 using operators loaded on the second memory 340 and switched by the OP hub 380, and generate signals (or indication signals) based on the execution progress of the operators of the DL model 100. For example, the signal may include the node identifier (ID) of the operation, and may also include information such as program tensors, kernels and / or time.

[0054] In one embodiment, execution scheme (or descriptor) 350 may also be stored in the second memory 340. For example, execution scheme 350 may include scripts, rules, or models. In one embodiment, execution scheme 350 may be a script, and analyzer 320 may be a script player. In another embodiment, rules may generate scripts for use in a layer of a program (e.g., DL model 100) based on the input tensor of the previous layer of the program. Analyzer 320 may load, for example, operators OP1 to OP5 from the first memory 330 to the second memory 340 via direct memory access (DMA) controller 360, and control estimator 321 to manage the power supplied to the first memory 330 based on this signal and execution scheme 350. In one embodiment, estimator 321 may enable power controller 370 to manage the power supplied to the first memory 330 by powering on / off the first memory 330. In another embodiment, estimator 321 may enable power controller 370 to manage the power supplied to the first memory 330 by configuring the operating mode of the first memory 330. For example, estimator 321 may enable power controller 370 to configure the first memory 330 to operate in a high-performance mode by scaling the data rate and row cycle time of the first memory 330. In some other embodiments, estimator 321 may enable power controller 370 to manage the power supplied to the first memory 330 by dynamically scaling the voltage and / or frequency (DVFS) applied to the first memory 330, because power consumption is related to V... 2 xf is correct. For example, estimator 321 can manage the power supplied to the first memory 330 by reducing the voltage applied to the first memory 330. Although reducing the voltage may increase the signal propagation delay, and the increase in signal propagation delay can lead to errors when using unmodified timing parameters (e.g., the time required to reliably detect and amplify data in the row buffer), if the system used to model the neural network is highly tolerant of errors, the neural network is a universal approximator, and it can still work well.

[0055] The operation of device 300 can be described as follows (see also reference) Figure 2 ).

[0056] Before time t1, when the executor 310 is ready to execute the DL model 100, the analyzer 320 can generate an execution plan 350, or retrieve the execution plan 350 from the second memory 340, and initiate the DMA controller 360 to load operators OP1 (e.g., 3x3 convolution) and OP2 (e.g., 1x1 convolution) from the first memory 330 to the second memory 340. The size of the second memory 340 is, for example, set to store a maximum of two operators. For example, the analyzer 320 can control the estimator 321 to provide power. The estimator 321 can enable the power controller 370 to power on the first memory 330 by executing power management A, and then control the DMA controller 360 to load operators OP1 and OP2 from the first memory 330 to the second memory 340, and enable the power controller 370 to power off the first memory 330.

[0057] At time tl, executor 310 will use operator OP1 for inference (or model, e.g., DL model 100) and then send an indication signal (or information) to analyzer 320. For example, executor 310 begins to execute a 3x3 convolution loaded on a second memory 340.

[0058] At time t2, executor 310 has completed executing DL model 100 with operator OP1 and begins executing DL model 100 with operator OP2. Operator OP2, along with operator OP1, was loaded from the first memory 330 to the second memory 340 before time t1 and is already stored in the second memory 340. For example, executor 310 begins executing a 1x1 convolution loaded in the second memory 340.

[0059] At time t3-Δt, while executor 310 is still executing DL model 100 using operator OP2 (e.g., 1x1 convolution), analyzer 320, based on execution scheme 350 and indication signals (or information) sent from executor 310, knows it is time to load operators OP3 and OP4 from first memory 330 to second memory 340 to reduce latency. For example, estimator 321 may execute power management A at time t3-Δt to power on first memory 330 via power controller 370, then initiate DMA controller 360 to load operators OP3 (e.g., 3x3 DW convolution) and OP4 (e.g., 1x1 convolution) from first memory 330 to second memory 340, and power off first memory 330 at time t3. In one embodiment, only 3x3 DW convolution is loaded from first memory 330 to second memory 340 because 1x1 convolution has already been loaded into second memory 340.

[0060] At time t3, when operators OP3 and OP4 have been stored in the second memory 340, the actuator 310 can begin executing the DL model 100 with operator OP3, and then send a signal (or information) to the analyzer 320. For example, the actuator 310 begins executing a 3x3 DW convolution loaded on the second memory 340.

[0061] At time t4, when executor 310 has completed the execution of DL model 100 using operator OP3, executor 310 can begin executing DL model 100 using operator OP4. Before time t3, operator OP4, together with operator OP3, has been loaded from the first memory 330 into the second memory 340 and is already stored in the second memory 340. For example, executor 310 begins executing a 1x1 convolution loaded on the second memory 340 and sends a signal (or information) to analyzer 320.

[0062] At time t5-Δt, while the actuator 310 is still executing the DL model 100 using operator OP4 (e.g., 1x1 convolution), the analyzer 320, based on the execution scheme 350 and signals (or information) sent from the actuator 310, can determine when it is time to load operator OP5 from the first memory 330 to the second memory 340 to reduce latency. For example, the estimator 321 can perform power management A at time t5-Δt, powering on the first memory 330 again via the power controller 370, then initiating the DMA controller 360 to load operator OP5 (e.g., FC) from the first memory 330 to the second memory 340, and powering off the first memory 330 again at time t5.

[0063] At time t5, when the executor 310 has completed the execution of the DL model 100 using operator OP4, the executor 310 can begin executing the DL model 100 using operator OP5, which has been loaded into the second memory 340, and then send a signal (or information) to the analyzer 320. For example, the executor 310 begins executing the FC loaded into the second memory 340.

[0064] At time t6, executor 310 has completed the execution of DL model 100 with operator OP5.

[0065] Since the next operator used by the actuator 310 to execute DL model 100 is already stored in the second memory 340 before the actuator 310 completes the execution of DL model 100 with the previous operators, the actuator 310 can complete the execution of DL model 100 with all operators OP1 to OP5 at time t6 without incurring additional delay. Furthermore, since the first memory 330 is only powered on during the loading of operators from the first memory 330 to the second memory 340, the device 300 consumes less energy compared to the prior art where DRAM is always powered on and continuously consumes power.

[0066] As mentioned earlier, the DL model can be partitioned into multiple partitions and then distributed across heterogeneous hardware, such as general-purpose processors (e.g., CPUs) and dedicated hardware accelerators (e.g., mini DLAs and APUs), to accelerate DL computation by reducing the resource cost of each partition.

[0067] Figure 4A This is a power timing diagram of an exemplary hardware accelerator (e.g., a mini DLA and an APU) operating in its working mode, during which the mini DLA and APU consume power p1 and power p2, respectively. For example, DL model 400 (e.g., an autoencoder model) involves multiple operators OP1 to OP5, where operators OP1, OP3, and OP5 are not complex and can be executed by a general-purpose processor (e.g., a CPU), while operators OP2 and OP4 are computationally demanding and need to be executed by dedicated mini DLAs and APUs, respectively, to accelerate the execution speed of DL model 400.

[0068] However, these hardware accelerators consume a significant amount of power when powered on and operating in working mode. To address this issue, these hardware accelerators can switch to standby mode when not performing DL calculations, or switch to working mode when performing DL calculations.

[0069] Figure 4B This is a power timing diagram illustrating the switching from standby mode to operating mode for an exemplary hardware accelerator. During standby mode, the hardware accelerator consumes lower power p0. The mini DLA and APU, operating in operating mode, consume higher power p1 and power p2, respectively. Since it takes time, e.g., Δt2, for the lower power p0 to increase to the higher power p1, the mini DLA begins operating in operating mode at time t2+Δt2, thus incurring an overhead Δt2. Similarly, it takes time, e.g., Δt4, for the lower power p0 to increase to the higher power p2, and the APU does not begin operating in operating mode until time t4+Δt2+Δt4.

[0070] Figure 5This is a power timing diagram of an exemplary hardware accelerator (e.g., a mini DLA and an APU) switching from standby mode to operating mode according to some embodiments of this disclosure. For example, when a mini DLA is scheduled to operate in operating mode at time t2, the mini DLA can switch from standby mode to operating mode no later than time t2-Δt2. Therefore, at time t2 the mini DLA is already in operating mode, which is precisely the exact time it is scheduled to execute the computationally demanding operator OP2. Similarly, an APU can switch from standby mode to operating mode no later than time t4-Δt4 and is ready to run at time t4, i.e., the exact time it is scheduled to execute the computationally demanding operator OP4. In some other embodiments, the exemplary hardware accelerator can be switched to operate in operating mode or powered off.

[0071] Figure 6 This is a functional block diagram of an exemplary device 600 for executing a program (e.g., DL model 400) according to some embodiments of the present disclosure. For example, device 600 may be a smartphone, wearable device, or any Internet of Things (IoT) device. In one embodiment, device 600 may include an actuator 610 (or a first actuator, such as an inference actuator), a hardware accelerator 690 (or a second actuator), an analyzer 620 coupled to actuator 610, and an estimator 621 coupled to analyzer 620 and hardware accelerator 690. Actuator 610, analyzer 620, and estimator 621 may be similar to actuator 310, analyzer 320, and estimator 321 of device 300, respectively.

[0072] In one embodiment, the analyzer 620 may be coupled between a first memory 330 and a second memory 340. In one embodiment, the first memory 330 may be used to store programs (e.g., the DL model 400 relating to operators OP1 to OP5), which may be loaded into the second memory 340 for execution by the executor 610. In one embodiment, to simplify the description of the operation of the exemplary device 600, all operators OP1 to OP5 of the DL model 400 have been loaded from the first memory 330 and stored in the second memory 340.

[0073] In one embodiment, execution scheme (or descriptor) 650 may also be stored in second memory 340. For example, execution scheme 650 may include scripts, rules, or models. In one embodiment, execution scheme 650 may be a script, and analyzer 620 may be a script player. In another embodiment, rules may generate scripts for specific layers of a program (e.g., DL model 400) based on input tensors from the previous layer of the program. Analyzer 620 may control estimator 621 to manage the power supplied to hardware accelerator 690 (e.g., mini DLA and APU) based on indication signals and execution scheme 650. In one embodiment, estimator 621 may manage the power applied to hardware accelerator 690 by switching hardware accelerator 690 between standby and operating modes via power management B.

[0074] refer to Figure 5 The operation of device 600 can be described as follows.

[0075] Before time tl, when the executor 610 is ready to execute the DL model 400, the analyzer 620 can generate an execution plan 650 or retrieve the execution plan 650 from the second memory 340.

[0076] At time t1, executor 610 (e.g., CPU) begins inference on the program (or model, e.g., DL model 400) using operator OP1, and then sends an indication signal (or information) to analyzer 620. For example, executor 610 begins executing DL model 400 using operator OP1.

[0077] At time t2-Δt2, while actuator 610 is still executing DL model 400 using operator OP1, analyzer 620, based on execution scheme 650 and indication signals (or information) sent from actuator 610, knows that DL model 400 will be executed next with operator OP2, and it is time to switch hardware accelerator 690 (e.g., mini DLA) from standby mode to operating mode, which takes time Δt2. For example, estimator 621 can perform power management B to provide more power to the mini DLA, thereby switching the mini DLA to operate in operating mode.

[0078] At time t2, when the actuator 610 has completed the execution of DL model 400 with operator OP1 and the mini DLA has switched to the working mode, the OP hub 380 can switch to operator OP2 (the mini DLA will execute DL model 400 with operator OP2), and the actuator 610 can start the mini DLA to begin executing DL model 400 with operator OP2 and send an indication signal to the analyzer 620.

[0079] At time t3, when the mini-DLA has finished executing the DL model 400 with operator OP2, the estimator 621 can manage the power supplied to the mini-DLA through power management B, switch the mini-DLA back to standby mode to save power, and the actuator 610 starts executing the DL model 400 with operator OP3.

[0080] At time t4-Δt4, which may be earlier than or the same as time t3, analyzer 620, based on execution scheme 650 and signals (or information) sent from the mini-DLA, knows that DL model 400 will be executed next with operator OP4, and it is time to switch the hardware accelerator 690 (e.g., APU) from standby mode to operating mode. For example, analyzer 620 may execute power management B to provide more power to the APU, thereby switching the APU to operate in operating mode.

[0081] At time t4, when the actuator 610 has finished executing the DL model 400 with operator OP3 and the APU has switched to operating mode, the OP hub 380 can switch to operator OP4, the APU will execute the DL model 400 with OP4, and the actuator 610 can cause the APU to start executing the DL model 400 with operator OP4 and send a signal to the analyzer 620.

[0082] At time t5, when the APU has completed the execution of DL model 400 with operator OP4, the estimator 621 can manage the power supplied to the APU through power management B to switch the APU back to standby mode to save power, and the executor 610 starts executing DL model 400 with operator OP5.

[0083] Since the hardware accelerator 690 has switched to operating mode and is ready to execute the DL model using operators OP2 and OP4 before the executor 610 completes the execution of the DL model 400 using the previous operators OP1 and OP3, no additional delay is generated.

[0084] The above is for reference. Figure 5 The operations performed by device 600 are described. Device 600 can also refer to... Figure 2 and Figure 5 Execute the DL model. This can be combined with the above... Figure 2 Description and Figure 5 Based on the relevant descriptions, the operation of device 600 is derived. The following is only a brief description of an example.

[0085] At time t1, executor 610 uses operator OP1 to infer the program (or model, e.g., DL model);

[0086] At time t2-Δt2, power management B begins to switch the hardware accelerator 690 (e.g., mini DLA) from standby mode to operating mode, which takes time Δt2;

[0087] At time t2, the mini DLA begins executing DL model 400 with operator OP2, and after the mini DLA finishes executing the DL model with operator OP2, power management B is executed to switch the mini DLA back to running in standby mode;

[0088] At time t3-Δt, power management A is powered on to start loading operators OP3 and OP4 from the first memory 330 to the second memory 340, and the first memory 330 is powered off after loading is completed at time t3.

[0089] At time t3, executor 610 can begin executing the DL model with operator OP3;

[0090] At time t4-Δt4, power management B begins to switch the hardware accelerator 690 (e.g., APU) to operating mode;

[0091] At time t4, when the APU has switched to running in working mode, the APU will execute the DL model with OP4; and after the APU has finished executing the DL model with operator OP4, power management B will be executed to switch the APU back to running in standby mode.

[0092] At time t5-Δt, power management A is powered on again to start loading operator OP5 from the first memory 330 to the second memory 340; at time t5, actuator 610 can start executing the DL model with operator OP5.

[0093] The above description is not a limiting description of the present invention, and those skilled in the art can make any appropriate changes to the above embodiments. For example, the first memory can store, for example, three operators, or it can be operators OP3 or OP5 (or other operators, etc.) that have high computational requirements and require hardware accelerators for execution. Based on the foregoing, those skilled in the art can determine the appropriate timing for powering on / off the first memory and the appropriate timing for switching the operating mode of the hardware accelerator.

[0094] Figure 7This is a flowchart illustrating an exemplary method 700 for executing a program involving multiple operators according to some embodiments of the present disclosure. In various embodiments, some steps of the illustrated method 700 may be performed simultaneously in a different order than shown, may be replaced by other method steps, or may be omitted. Additional method steps may also be performed as needed. Aspects of method 700 may be implemented by a smartphone, wearable device, or any IoT device, such as the devices 300 and 600 shown and described in the preceding figures.

[0095] In step S710, at least the first operator among the plurality of operators can be loaded from the first memory storing the plurality of operators into the second memory. For example, operators OP1 and OP2 can be loaded from the first memory 330 into the second memory 340.

[0096] In step S720, the program can be executed using the first operator loaded onto the second memory, and an indication signal can be generated based on the program's execution progress. For example, the executor 310 can execute the DL model using operators OP1 and OP2, and generate an indication signal (or information) which is then sent to the analyzer 320.

[0097] In step S730, based on the indication signal and execution plan, at least a second operator from a plurality of operators stored in the first memory can be loaded into the second memory before the program execution with the first operator is completed. For example, when the executor 310 is still executing the DL model 100 using operator OP2, the analyzer 320 can know, based on the execution plan 350 and the indication signal (or information) sent by the executor 310, that it is time to load operators OP3 and OP4 from the first memory 330 into the second memory 340 to reduce latency. In one embodiment, based on the indication signal and execution plan, power can be supplied to the first memory before the program execution with the first operator is completed. For example, the estimator 321 can perform power management A at time t3-Δt to power on the first memory 330 via the power controller 370, then initiate the DMA controller 360 to load operators OP3 and OP4 from the first memory 330 into the second memory 340, and power off the first memory 330 at time t3.

[0098] In one embodiment, the program can be executed by a first executor and a second executor using a first operator and a second operator, respectively. For example, executor 610, such as a CPU, can use operators OP1 and OP3 to execute DL model 400, while hardware accelerator 690, such as a mini DLA or APU, can use operators OP2 and OP4 to execute the DL model. In one embodiment, power can be supplied to the second executor based on the execution scheme and signals from the first executor before the first executor completes executing the program with the first operator. For example, estimator 621 can manage the power supplied to hardware accelerator 690 via power management B by powering on / off hardware accelerator 690, configuring the operating mode of hardware accelerator 690, or scaling the voltage and / or frequency applied to hardware accelerator 690.

[0099] Devices 300 and 600 may optionally include other components, such as input and output devices, additional signal processing circuitry, etc. Therefore, devices 300 and 600 are capable of performing additional functions, such as executing application programs and processing alternative communication protocols.

[0100] The processes and functions described herein can be implemented as a computer program, which, when executed by one or more processors, causes the one or more processors to perform corresponding methods and functions. The computer program can be stored or distributed on a suitable medium, such as an optical storage medium or solid-state medium provided together with or as part of other hardware. The computer program can also be distributed in other forms, for example, via the Internet or other wired or wireless telecommunications systems. For example, the computer program can be acquired and loaded into a device, including through physical media or distributed systems (including, for example, from a server connected to the Internet).

[0101] A computer program can be accessed from a computer-readable medium that provides program instructions for use by or in connection with a computer or any instruction execution system. The computer-readable medium may include any means of storing, communicating, propagating, or transmitting the computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable medium may be a magnetic, optical, electronic, electromagnetic, infrared, or semiconductor system (or apparatus or device) or a propagation medium. The computer-readable medium may include computer-readable non-transitory storage media, such as semiconductor or solid-state memory, magnetic tape, removable computer disks, random access memory (RAM), read-only memory (ROM), magnetic disks, and optical disks. Computer-readable non-transitory storage media may include all types of computer-readable media, including magnetic storage media, optical storage media, flash memory media, and solid-state storage media.

[0102] While the invention has been described by way of example and according to preferred embodiments, it should be understood that the invention is not limited to the disclosed embodiments. Rather, it is intended to cover various variations and similar structures (as will be apparent to those skilled in the art), such as combinations or substitutions of different features in different embodiments. Therefore, the scope of the appended claims should be given the broadest interpretation to cover all such variations and similar structures.

Claims

1. An apparatus for executing a program comprising a plurality of operators, the apparatus comprising: An executor is configured to execute the program with at least a first operator of the plurality of operators, wherein the first operator is loaded from a first memory storing the plurality of operators to a second memory, and the executor generates an indication signal based on the progress of executing the program with the first operator; as well as An analyzer, coupled to the executor, the first memory, and the second memory, is configured to load at least a second operator stored in the first memory into the second memory based on an indication signal from the executor and an execution scheme stored in the second memory, before the executor completes execution of the program with the first operator; An estimator, coupled to the analyzer, the first memory, and the second memory, wherein the analyzer controls the estimator to perform power management operations based on the execution scheme stored in the second memory and the indication signal from the executor, before the executor completes execution of the program with the first operator; The power management operation includes managing the power applied to the first memory; The analyzer loads the second operator into the second memory simultaneously, according to the execution scheme stored in the second memory, during the time period when the executor is executing the program with the first operator; The second operator is stored in the second memory before the executor finishes execution with the first operator, so that the executor can immediately execute the program with the second operator after finishing execution with the first operator; The analyzer is configured to control the power applied to the first memory such that the first memory is powered on when the second operator begins to load, and the first memory is powered off immediately when the second operator finishes loading.

2. The apparatus according to claim 1, wherein, The estimator manages the power supplied to the first memory by powering on / off the first memory, configuring the operating mode of the first memory, or scaling the voltage and / or frequency applied to the first memory.

3. The apparatus according to claim 1, wherein, The execution scheme includes scripts, rules, or models.

4. The apparatus according to claim 3, wherein, The rules generate scripts for specific layers of the program based on input tensors from the previous layer of the program.

5. The apparatus according to claim 1, wherein, The indication signals include the program's operation, node identifier, tensor, kernel, and / or time.

6. The apparatus according to claim 1, wherein, The second memory is a tightly coupled memory (TCM) or a static random access memory (SRAM).

7. The apparatus according to claim 1, wherein, The first memory is dynamic random access memory (DRAM), on-bus SRAM, or serial flash memory.

8. The apparatus according to claim 1, wherein: The actuator is a central processing unit (CPU), and the analyzer is a microcontroller unit (MCU) or an exception handler, or... The actuator is a deep learning accelerator (DLA), and the analyzer is an MCU.

9. The apparatus according to claim 1, wherein, The analyzer and the actuator are included on a single chip.

10. The apparatus of claim 1, further comprising a direct memory access (DMA) controller coupled to the analyzer, the first memory, and the second memory, wherein the analyzer is further configured to instruct the DMA controller to load the second operator stored in the first memory into the second memory.

11. The apparatus according to claim 1, wherein, The execution scheme is either manually created or created by an offline optimizer or an online / runtime optimizer.

12. An apparatus for executing a program involving a plurality of operators, the apparatus comprising: A first executor is configured to execute the program with at least a first operator of the plurality of operators, and to generate an indication signal based on the progress of executing the program with the first operator; A second executor is configured to execute the program using at least a second operator; as well as An analyzer coupled to the first actuator and the second actuator, The estimator is coupled to the analyzer and the second memory; The analyzer is configured to control the estimator to perform power management operations based on the execution scheme and the indication signal from the first executor before the first executor completes the execution of the program with the first operator; The power management operation includes managing the power applied to the second actuator; The power management operation includes managing the power applied to the second actuator to power on the second actuator by performing the power management operation; This ensures that when the first executor completes execution with the first operator, the second executor has already been powered on, so that the second executor can immediately execute the program with the second operator after the first executor has completed execution.

13. The apparatus of claim 12, wherein the estimator is configured to manage the power supplied to the second actuator by powering on / off the second actuator, configuring the operating mode of the second actuator, or scaling the voltage and / or frequency applied to the second actuator.

14. The apparatus according to claim 12, wherein, The second actuator is a DLA or an acceleration processing unit (APU).

15. A method for executing a program involving multiple operators, comprising: Load at least the first operator of the plurality of operators from the first memory storing the plurality of operators into the second memory; The program is executed using the first operator loaded on the second memory, and an indication signal is generated based on the execution progress of the program; as well as Based on the indication signal and the execution scheme stored in the second memory, before the program is completed using the first operator, at least the second operator among the plurality of operators stored in the first memory is loaded into the second memory; Based on the indication signal and the execution scheme stored in the second memory, a power management operation is performed before the program executed with the first operator is completed; The power management operation includes managing the power applied to the first memory; The analyzer loads the second operator into the second memory simultaneously, according to the execution scheme stored in the second memory, during the time period when the executor is executing the program with the first operator; The second operator is stored in the second memory before the executor finishes execution with the first operator, so that the executor can immediately execute the program with the second operator after finishing execution with the first operator; The analyzer is configured to control the power applied to the first memory such that the first memory is powered on when the second operator begins to load, and the first memory is powered off immediately when the second operator finishes loading.

16. The method according to claim 15, wherein, The first operator and the second operator are executed by the first executor and the second executor, respectively; The power management operation further includes managing the power applied to the second actuator.

17. The method of claim 16, wherein managing the power applied to the second actuator includes powering on / off the second actuator, configuring the operating mode of the second actuator, or scaling the voltage and / or frequency applied to the second actuator.

Images