Device and method for executing a program

Abstract

Various aspects of the present invention provide methods and apparatus for executing a program (e.g., neural network (NN) reasoning). For example, the apparatus may include an executor and a dynamic loading agent. The executor is coupled to a second memory and is configured to execute a portion of the NN reasoning that has been loaded from a first memory storing the NN reasoning on the second memory, and to generate a signal based on the execution progress of the NN reasoning. The dynamic loading agent is coupled to the executor, the first memory, and the second memory, and is configured to load the next portion of the NN reasoning stored in the first memory to the second memory, and manage the power provided to the first memory based on the signal from the executor and the reasoning execution scheme stored in the second memory.

Description

Technical Field

[0001] The present application relates generally to memory, and more particularly to apparatus and methods capable of dynamically loading neural network inference stored in memory and managing power supplied to the memory. Background Art

[0002] Unless otherwise indicated herein, the approaches described in this section do not constitute prior art to the listed claims and are not admitted to be prior art by inclusion in this section.

[0003] In a process called training, a neural network (NN) can learn from examples of a specific task. After learning, the task can be performed on new data in a process called inference. Neural network (NN) inference has a large number of weights and activations, and needs to be stored in a large enough memory, such as dynamic random access memory (DRAM). During the execution of the entire NN inference, the DRAM is powered on, which consumes a lot of power. Summary of the invention

[0004] The following summary of the invention is illustrative only and is not intended to be limiting in any way. That is, the following overview is provided to introduce the concepts, highlights, benefits, and advantages of the novel and non-obvious technologies described herein. Selected embodiments are further described in the detailed description below. Therefore, the following summary of the invention is neither intended to identify the necessary features of the claimed subject matter nor to be used to determine the scope of the claimed subject matter. The present invention provides an apparatus and method for executing a program (e.g., neural network (NN) reasoning) that can reduce power consumption.

[0005] Aspects of the present invention provide a device for executing a program. The device may include an executor and a dynamic loading agent. The executor is coupled to a second memory, and the executor is configured to execute a portion of the program (wherein the portion of the program has been loaded from a first memory for storing the program on the second memory), and to generate a signal based on the execution progress of the program. The dynamic loading agent may be coupled to the executor, the first memory, and the second memory, and the dynamic loading agent is configured to load the next portion of the program stored in the first memory to the second memory, and manage the power provided to the first memory based on the signal from the executor and the execution scheme stored in the second memory.

[0006] For example, the dynamic loading agent can manage the power provided to the first memory by powering the first memory on or off, configuring the operating mode of the first memory, or scaling the voltage and / or frequency applied to the first memory. As another example, the execution scheme may include a script, a rule, or a model. For example, the rule may generate a script for a specific layer of the program based on an input tensor from a previous layer of the program. As yet another example, the signal may include an operation of the program, a node identification (ID), a tensor, a kernel, and / or a time.

[0007] In one embodiment, the second memory may be a tightly coupled memory (TCM) or a static random access memory (SRAM). In another embodiment, the first memory may be a dynamic random access memory (DRAM), a bus SRAM, or a serial flash memory. For example, the executor may be a central processing unit (CPU), and the dynamic loading agent may be a microcontroller unit (MCU). For another example, the executor may be a CPU, and the dynamic loading agent may be a CPU exception. For another example, the executor may be a deep learning accelerator (DLA), and the dynamic loading agent may be an MCU.

[0008] In one embodiment, the dynamic loading agent and the executor may be included on a single chip. In another embodiment, the apparatus may further include a direct memory access (DMA) controller coupled to the dynamic loading agent, the first memory, and the second memory, and the dynamic loading agent may be further configured to instruct the DMA controller to load a next portion of the program stored in the first memory into the second memory.

[0009] Aspects of the present invention also provide a method for executing a program. The method may include loading a portion of the program from a first memory storing the program to a second memory. The method may further include executing the portion of the program loaded on the second memory, and generating a signal based on the execution progress of the program. The method may also include loading the next portion of the program stored in the first memory to the second memory, and managing the power provided to the first memory based on the signal and the execution scheme stored in the second memory.

[0010] Those skilled in the art will no doubt understand these and other objects of the present invention after reading the following detailed description of the preferred embodiments shown in the accompanying drawings. The detailed description will be given in the following embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The present invention may be more fully understood by reading the subsequent detailed description and by referring to the examples given in the accompanying drawings, wherein: the accompanying drawings are included to provide a further understanding of the embodiments of the present disclosure, and the accompanying drawings are incorporated into and constitute a part of the embodiments of the present disclosure. The accompanying drawings illustrate implementation methods of the embodiments of the present disclosure and are used together with the specification to explain the principles of the embodiments of the present disclosure. It is understood that the drawings are not necessarily drawn to scale, as some components may be shown to be out of proportion to the dimensions in actual implementation to clearly illustrate the concepts of the embodiments of the present disclosure. Various embodiments of the present invention presented as examples will be described in detail with reference to the following drawings, wherein the same reference numerals represent the same components.

[0012] Figure 1 A functional block diagram of an exemplary apparatus for executing a program (eg, neural network (NN) inference) according to some embodiments of the present invention is shown.

[0013] Figure 2 According to some embodiments of the present invention, a flow through a first memory (storage Figure 1 Timing diagram of the current of NN inference).

[0014] Figure 3 A functional block diagram of another exemplary apparatus for performing NN inference is shown according to some embodiments of the present invention.

[0015] Figure 4 According to some embodiments of the present invention, a flow through a first memory (storage Figure 3 Timing diagram of the current of NN inference).

[0016] Figure 5 According to some embodiments of the present invention, a flow through a first memory (storage Figure 3 Another current timing diagram of NN inference).

[0017] Figure 6 A functional block diagram of yet another exemplary apparatus for performing NN inference is shown according to some embodiments of the present invention.

[0018] Figure 7 A functional block diagram of yet another exemplary apparatus for performing NN inference is shown according to some embodiments of the present invention.

[0019] Figure 8 A flowchart diagram of an exemplary method for performing NN reasoning is shown according to some embodiments of the present invention.

[0020] Fig. 9 A flowchart diagram of another exemplary method for performing NN reasoning according to some embodiments of the present invention is shown.

[0021] In the following detailed description, for the purpose of illustration, many specific details are set forth so that those skilled in the art can more thoroughly understand the embodiments of the present invention. However, it is apparent that one or more embodiments may be implemented without these specific details, and different embodiments may be combined as needed, and should not be limited to the embodiments listed in the accompanying drawings. DETAILED DESCRIPTION

[0022] The following description is a preferred embodiment of the present invention, which is only used to illustrate the technical features of the present invention, and is not used to limit the scope of the present invention. Certain words are used throughout the specification and claims to refer to specific components. It should be understood by those skilled in the art that manufacturers may use different names to refer to the same components. Therefore, the present specification and claims do not use the difference in name as a way to distinguish components, but use the difference in function of the components as a basis for distinction. The terms "component", "system" and "device" used in the present invention can be entities related to computers, wherein the computer can be hardware, software, or a combination of hardware and software. The terms "including" and "comprising" mentioned in the following description and claims are open-ended terms, so they should be interpreted as "including, but not limited to...". In addition, the term "coupled" means an indirect or direct electrical connection. Therefore, if a device is described as coupled to another device, it means that the device can be directly electrically connected to the other device, or indirectly electrically connected to the other device through other devices or connection means.

[0023] Corresponding numbers and symbols in the various figures of the drawings generally refer to corresponding parts unless otherwise indicated. The drawings are drawn to clearly illustrate the relevant parts of the embodiments and are not necessarily drawn to scale.

[0024] The term "substantially" or "approximately" as used herein means that within an acceptable range, a person skilled in the art can solve the technical problem to be solved and basically achieve the technical effect to be achieved. For example, "approximately equal to" means that a certain error from "completely equal to" is acceptable to a person skilled in the art without affecting the correctness of the result.

[0025] This specification discloses detailed embodiments and implementations of the claimed subject matter. However, it should be understood that the disclosed embodiments and implementations are merely illustrations of the claimed subject matter, which may be embodied in various forms. For example, the disclosed embodiments may be implemented in many different forms and should not be construed as being limited to the exemplary embodiments and implementations set forth herein. Rather, these exemplary embodiments and implementations are provided so that the description of the disclosed embodiments is thorough and complete, and will fully convey the scope of the disclosed embodiments to those skilled in the art. In the following description, details of well-known features and technologies may be omitted to avoid unnecessarily obscuring the presented embodiments and implementations.

[0026] Neural networks (NN) are used to extract meaningful information or features from a large amount of data. Without being specifically programmed, a neural network (NN) can learn from various samples of a specific task in a process called training. After learning, the task can be performed on new data in a process called inference. During the training process, a set of optimal weights that optimize the accuracy of NN inference can be estimated. Neural networks (NN) are programs that include a large number of weights and activations, so NN inferences are too large to be stored in, for example, tightly coupled memory (TCM), which has limited bandwidth. For example, neural network (NN) inferences can be stored in dynamic random access memory (DRAM), and the neural network (NN) inferences will be loaded from the dynamic random access memory (DRAM) to the tightly coupled memory (TCM) portion by portion for execution by the inference executor. During the entire process of loading and executing the NN inference, the DRAM is always powered on, which consumes a lot of power.

[0027] According to some aspects of the present invention, a dynamic loading agent and an executing scheme are introduced, for example, an inference executing scheme. In one embodiment, the inference executing scheme can be used to tell the dynamic loading agent when to load a portion of a program (e.g., NN inference) from DRAM to TCM and how to manage the power provided to the DRAM. For example, when the dynamic loading agent looks up the inference executing scheme and learns that a portion of the NN inference needs to be loaded from DRAM to TCM, the dynamic loading agent can turn on or power on the DRAM, load a portion of the NN inference from DRAM to TCM, and turn off or power off the DRAM after the portion of the NN inference is loaded into the TCM. According to some aspects of the present invention, the dynamic random access memory (DRAM) will be powered off for most of the time when the NN inference is executed.

[0028] Smart devices, wearable devices, and other Internet of things (IoT) devices may include always-on continuous sensing apps and various sensors (e.g., accelerometers, microphones, ambient light sensors, biometric sensors, etc.), and use the data collected by these sensors to implement gesture- or voice-based control, monitor user activity and security, and provide interpretation of the user's context. These IoT devices may perform analysis and interpretation of the collected data on a main processor, which therefore needs to be frequently powered on to address these continuous sensing tasks. These IoT devices may also utilize dedicated low-power microprocessors (called sensor hubs) to process the data to offload much of the data processing work from the main processor and reduce power consumption.

[0029] Tightly coupled memory (TCM) is a type of on-chip memory that can be embedded in low-power processors to speed up data processing in the processor. Tightly coupled memory (TCM) is expensive, so it is not cost-effective to embed a large size of TCM in a processor. Tightly coupled memory (TCM) is implemented very close to the central processing unit (CPU) and has access latency and size close to that of the L1 cache.

[0030] Neural Networks (NNs), based on biological modeling of the human brain, are used to extract important information or features from large amounts of data. Without being specifically programmed, a NN can learn from a variety of samples of a specific task in a process called training. After learning, the task can be performed on new data in a process called inference. During the training process, an optimal set of weights that optimize the accuracy of NN inference can be estimated.

[0031] Artificial neural network (ANN) is a type of neural network in deep learning (DNN). An artificial neural network (ANN) may include a plurality of interconnected neurons (or nodes or units) arranged in layers. There are connections or edges between nodes in adjacent layers, and the connections have weights associated with them. Each node is able to receive inputs from some other nodes connected to it from adjacent layers or the outside world, and calculate outputs. Each input has an associated weight, which is assigned based on its relative importance relative to other inputs. The node weights the inputs with associated weights, sums the weighted inputs and a trainable constant value (called a bias), and applies a nonlinear activation function to the sum to calculate the output. The activation function may be a sigmoid function, a hyperbolic tangent (tanh) function, or a rectified linear unit (ReLU). The nodes of artificial neural network (ANN) are divided into input nodes, hidden nodes and output nodes, which are arranged in different layers (or called operations or operators), including input layer, hidden layer and output layer. Input nodes can receive data from the outside world and pass the data to hidden nodes without any calculation. As the name suggests, hidden nodes have no direct connection with the outside world. Hidden nodes calculate the data transmitted from input nodes and transmit the calculation results to output nodes. Output nodes can transmit the calculation results to the outside world.

[0032] Convolutional neural network (CNN) is one of the most popular neural networks (NN) in deep learning. Compared with artificial neural networks (ANN), which process input pixels of images that are far and near each other exactly the same and the layers are fully connected, convolutional neural networks (CNN) take into account the spatial structure of images through three basic concepts, including local receptive fields, shared weights, and pooling. Each neuron in the first hidden layer can be connected to the local receptive field of the input neuron corresponding to some input pixels of the image. The local receptive field can slide over the entire image at a time by a certain step size (for example, one or two pixels), and the first hidden layer is established. The hidden neurons in the first hidden layer constructed in this way can have the same weights and biases, and only detect exactly the same features at different locations in the image. The mapping from the input layer to the hidden layer is called a feature map. The weights and biases that define the feature map are called shared weights and shared biases, respectively, which in turn define the kernel or filter. Shared weights and shared biases can greatly reduce the number of parameters involved in convolutional networks.

[0033] Convolutional neural networks (CNNs) can be composed of a variety of different layers, including convolutional layers, pooling layers, and fully-connected layers. Each layer is defined by a weight matrix that is learned in a one-time training process performed before the convolutional neural network (CNN) is ready for inference. Each convolutional layer can have a kernel (or filter) and convolve the input with the kernel to obtain an output. For example, operations such as 2D convolution or addition (also called layers or operators) take one or more input tensors and produce a single output. Pooling layers are usually used immediately after convolutional layers, and convolutional layers are pooled to reduce the number of parameters of the input tensor. Each neuron in the pooling layer can summarize a region of 2x2 neurons (e.g., in the previous convolutional layer). For example, a pooling neuron only outputs the maximum or average activation value in the 2x2 input region. The last convolutional layer or pooling layer is flattened and output to a fully connected layer, which forms the last few layers in a convolutional neural network (CNN).

[0034] Neural network (NN) inference includes a large number of weights and activation values. As, for example, the model of NN inference becomes larger, it becomes more difficult to deploy the entire NN inference to TCM. In practice, NN inference is stored in an external memory (e.g., DRAM), and a portion of the NN inference will be moved to TCM each time it is executed.

[0035] The present invention provides a method for dynamically loading neural network inference at a first memory (eg, DRAM, bus SRAM, serial flash memory) to optimize power. Figure 1A functional block diagram of an exemplary apparatus 100 for executing a program (e.g., NN inference) is shown according to some embodiments of the present invention. For example, the apparatus 100 may be a smart phone, a wearable device, or any IoT device. In one embodiment, the apparatus 100 may include a controller 130 and an executor, such as an inference executor 140. For example, the inference executor 140 may be a central processing unit (CPU) or a microprocessor unit (MCU). Due to its low cost, wide availability, and moderate power consumption, the microprocessor unit (MCU) is an attractive platform for building a smartphone. In one embodiment, the controller 130 may be coupled between the first memory 110 and the second memory 120, and the inference executor 140 is coupled to the second memory 120. For example, the first memory 110 may be a dynamic random access memory (DRAM), an on-bus static random access memory (SRAM), or a serial flash memory. As another example, the second memory 120 may be a cache, and the controller 130 may therefore be a cache controller. It should be noted that, in some embodiments, the first memory 110, the second memory 120, the controller 130 and the executor 140 are schematically illustrated by taking DRAM, TCM, cache controller and inference executor 140 as examples, respectively, but the present invention is not limited to this example. It can be understood that the bus static random access memory requires a bus to access. In an embodiment of the present invention, the power consumption of the first memory and the second memory is relative, more specifically, the first memory has high power consumption and the second memory has low power consumption, for example, the first memory is a dynamic memory and the second memory is a static memory. More specifically, for example, if the first memory is DRAM, the second memory may be TCM / cache / SRAM, and for another example, if the first memory is an on-bus SRAM, the second memory may be TCM / cache. It should be noted that this example is not a limitation of the present invention.

[0036] In one embodiment, the NN reasoning 150 is stored in a first memory (e.g., DRAM) 110. The controller (e.g., cache controller) 130 loads a portion 151 of the NN reasoning 150 stored in the first memory (e.g., DRAM) 110 to the second memory (e.g., cache) 120 at a time. The inference executor 140 executes the portion 151 of the NN reasoning 150 loaded on the second memory (e.g., cache) 120. For example, when the data to be accessed exists in the second memory (e.g., cache) 120, the inference executor 140 executes the portion 151 of the NN reasoning 150 loaded on the second memory (e.g., cache) 120. As another example, when the data to be accessed is not inside the second memory (e.g., cache) 120 (miss), which may indicate that the inference executor 140 is about to execute the next portion 152 of the NN inference 150, the controller (e.g., cache controller) 130 may access the first memory (e.g., DRAM) 110 and load the next portion (nextportion) 152 of the NN inference 150 stored in the first memory (e.g., DRAM) 110 into the second memory (e.g., cache) 120, and then the inference executor 140 executes the next portion 152 of the NN inference 150 that has been loaded into the second memory (e.g., cache) 120.

[0037] Figure 2 According to some embodiments of the present invention, the flow through during the execution of the entire NN reasoning 150 is shown. Figure 1 The timing diagram of the current of the first memory (eg, DRAM) 110 is shown. Figure 2 As shown, when the data to be accessed is not inside the second memory (e.g., cache) 120, the controller (e.g., cache controller) 130 needs to access the first memory (e.g., DRAM) 110. For example, the controller (e.g., cache controller) 130 can power on the first memory (e.g., DRAM) 110 and then load the next part 152 of the NN reasoning 150 to the second memory (e.g., cache) 120. Once the first memory (e.g., DRAM) 110 is powered on / powered up, it will not be powered off until the entire NN reasoning 150 is fully executed. Therefore, during the loading and execution of the entire NN reasoning 150, the first memory (e.g., DRAM) 110 is always powered on. Figure 2 It is shown that the average current flowing through the first memory (eg, DRAM) 110 is approximately 24 mA.

[0038] Figure 3According to some embodiments of the present invention, a functional block diagram of another device 300 for performing NN reasoning is shown. The device 300 may also be a smart phone, a wearable device, or any IoT device. The device 300 may include a dynamic loading agent 330 and an inference executor 340, and the inference executor 340 is coupled to the dynamic loading agent 330. For example, the dynamic loading agent 330 and the inference executor 340 may be hardware or software. In one embodiment, the dynamic loading agent 330 and the inference executor 340 may be included on a single chip. In another embodiment, the dynamic loading agent 330 and the inference executor 340 may be two different chips. For example, the inference executor 340 may be a CPU, and the dynamic loading agent 330 may be an MCU. As another example, the inference executor 340 may be a deep learning accelerator (DLA), and the dynamic loading agent 330 may be a microprocessor unit (MCU). As another example, the inference executor 340 may be a central processing unit (CPU), and the dynamic loading agent 330 may be a CPU exception, wherein the CPU exception belongs to the behavior of the software, a software process triggered by a hardware interrupt or a software trigger. That is, the inference executor 340 may be a first software program in the CPU, and the dynamic loading agent 330 may be a second software program in the same CPU, wherein the first software program and the second software program are different. As another example, the inference executor 340 may be a CPU provided with an interrupt descriptor table, and the dynamic loading agent 330 may execute a specific exception handling function listed in the interrupt descriptor table when an exception occurs and the CPU interrupts its current work. In one embodiment, the inference executor 340 may be coupled to the second memory 320. For example, the second memory 320 may be a tightly coupled memory (TCM) or a static random access memory (SRAM). In another embodiment, the dynamic loading agent 330 may be coupled between the first memory 110 and the second memory 320. In yet another embodiment, the dynamic loading agent 330 may also be coupled to a power controller 350 , and the power controller 350 may be coupled to the first memory (eg, DRAM) 110 .

[0039] The second memory (e.g., TCM) 320 may be configured to store a portion 151 of the NN reasoning 150 loaded from the first memory (e.g., DRAM) 110. The inference executor 340 may execute the portion 151 of the NN reasoning 150 loaded into the second memory (e.g., TCM) 320 and generate a signal based on the execution progress of the NN reasoning 150. For example, the signal may include a node identification (ID) of the NN reasoning 150. In one embodiment, the inference execution scheme 370 may also be stored in the second memory (e.g., TCM) 320. The dynamic loading agent 330 may load the next portion 152 of the NN reasoning 150 from the first memory (e.g., DRAM) 110 to the second memory (e.g., TCM) 320, and manage the power / power provided to the first memory (e.g., DRAM) 110 via the power controller 350 based on the signal and the inference execution scheme 370. For example, the dynamic loading agent 330 may manage the power / electricity provided to the first memory (e.g., DRAM) 110 by powering on / off the first memory (e.g., DRAM) 110. In one embodiment, the dynamic loading agent 330 may be a script player, and the speculative execution scheme 370 may be a script, for example, which may list the following:

[0040] Node 0: DRAM powered on

[0041] Node 0: data 0-12 loaded to TCM

[0042] Node 0: DRAM powered off

[0043] Node 13: DRAM powered on

[0044] Node 13:data 13and 14loaded to TCM

[0045] Node 13: DRAM powered off

[0046] …

[0047] 5ms after Node 14:DRAM powered on

[0048] Node 15: data 15 loaded to TCM

[0049] Node 15: DRAM powered off

[0050] …

[0051] Node 30: DRAM powered on

[0052] Node 30&Convolution OP:do special op

[0053] Node 30: DRAM powered off

[0054] …

[0055] In some embodiments, the inference execution scheme 370 is illustrated by taking a script as an example, but the present invention is not limited to this example scenario. Figure 4 According to some embodiments of the present invention, the flow of the above script 370 stored in the second memory (eg, TCM) 320 during the execution of the entire NN reasoning 150 is shown. Figure 3 A timing diagram of the current flowing through the first memory (e.g., DRAM) 110. Initially, the inference executor 340 needs to execute a part of the NN inference and generate a signal, for example, Node 0. After receiving the signal of Node 0, the dynamic loading agent 330 searches for the script 370 stored in the second memory (e.g., TCM) 320. Then, the first memory (e.g., DRAM) 110 is powered on based on the script content “Node 0: DRAM powered on”, data 0-12 stored in the first memory (e.g., DRAM) 110 is loaded to the second memory (e.g., TCM) 320 based on the script content “Node 0: data 0-12 moved to TCM”, and the first memory (e.g., DRAM) 110 is powered off based on the script content “Node 0: DRAM powered off”. Therefore, since the first memory (e.g., DRAM) 110 is powered on, the current flowing through the first memory (e.g., DRAM) 110 has a spike at Node 0, as shown in FIG. Figure 4 As shown in “Node 0” in FIG. 1 , and since the first memory (e.g., DRAM) 110 is powered off after Node 0, the current becomes very small after Node 0. Then, the inference executor 340 executes a portion 151 of the NN inference 150 loaded on the second memory (e.g., TCM), e.g., data 0-12.

[0056] When executing a portion 151 of the NN reasoning 150 loaded on the second memory (e.g., TCM) 320, the reasoning executor 340 sequentially generates signals of nodes 1 to 12 based on the execution progress of the NN reasoning 150. Since the script 370 does not list any content indexed by nodes 1 to 12, the dynamic loading agent 330 does not perform any operation upon receiving the signals of nodes 1 to 12, and the first memory (e.g., DRAM) 110 remains in a powered-off state, as shown in FIG. Figure 4 As shown in "Node 1-12" in the figure.

[0057] Then, for example, when the inference executor 340 is executed at the node 13 and needs to execute the next part 152 of the NN inference 150, the inference executor 340 generates a signal of the node 13. After receiving the signal of the node 13, the dynamic loading agent 330 searches for the script 370, and sequentially powers on the first memory (e.g., DRAM) 110 based on the script content “Node 13: DRAM powered on”, loads the data 13 and 14 stored in the first memory (e.g., DRAM) 110 to the second memory (e.g., TCM) 320 based on the script content “Node 13: data13and 14moved to TCM”, and powers off the first memory (e.g., DRAM) 110 based on the script content “Node 13: DRAM powered off”. Therefore, since the first memory (e.g., DRAM) 110 is powered on again, the current flowing through the first memory (e.g., DRAM) 110 has a spike at the node 13, as shown in FIG. Figure 4 As shown in “Node 13” in FIG. 1 , and since the first memory (e.g., DRAM) 110 is powered off again after node 13, the current becomes very small after node 13. Then, the inference executor 340 executes the next part 152 of the NN inference 150 loaded on the second memory (e.g., TCM) 320, for example, data 13 and 14.

[0058] Then, for example, when the inference executor 340 is executed at the node 14 and needs to execute another next part of the NN inference 150, the inference executor 340 generates a signal of the node 14. Once the signal of the node 14 is received, the dynamic loading agent 330 can look up the script 370, and based on the script content "5ms after Node 14: DRAM powered on", the first memory (e.g., DRAM) 110 is powered on 5ms after receiving the signal of the node 14. Therefore, the current flowing through the first memory (e.g., DRAM) 110 has a spike at the position 5ms after the node 14, as shown in FIG. Figure 4As shown in "Node 14 5ms later".

[0059] Then, for example, when the inference executor 340 is executed on the node 15 and needs to execute another next part of the NN inference 150, the inference executor 340 generates a signal of the node 15. After receiving the signal of the node 15, the dynamic loading agent 330 can find the script 370, and, sequentially, load the data 15 stored in the first memory (e.g., DRAM) 110 to the second memory (e.g., TCM) 320 based on the script content “Node 15: data 15 moved to TCM”, and power off the first memory (e.g., DRAM) 110 based on the script content “Node 15: DRAM powered off”. In this case, the first memory (e.g., DRAM) 110 has been powered on before the inference executor 340 generates the signal of the node 15, and the dynamic loading agent 330 is able to load the data 15 from the first memory (e.g., DRAM) 110 to the second memory (e.g., TCM) 320 immediately after receiving the signal of the node 15. In an embodiment where the dynamic loading agent 330 and the speculative executor 340 are two different chips, the dynamic loading agent 330, after receiving the signal from node 14, can power on the first memory (e.g., DRAM) 110 when the speculative executor 340 executes at node 15.

[0060] Then, for example, when the inference executor 340 is executed at the node 30 and needs to perform an operation associated with the node 30, the inference executor 340 generates a signal of the node 30. Upon receiving the signal of the node 30, the dynamic loading agent 330 can look up the script 370, and sequentially power on the first memory (e.g., DRAM) 110 based on the script content “Node 30: DRAM powered on”, pass settings or configurations to the convolution operator (e.g., indicated by the channel number, data address, buffer size, etc.) based on the script content “Node 30&Convolution OP: do special op”, perform some specific operations (e.g., fine-grained operators, including loading a small amount of data and performing fine-grained convolution operations) based on the script content “Node 30&Convolution OP: do special op”, and power off the first memory (e.g., DRAM) 110 based on the script content “Node 30: DRAM powered off”.

[0061] With the flow Figure 1 Compared to the average current of 24 mA flowing through the first memory (eg, DRAM) 110, Figure 3The average current of the first memory (eg, DRAM) 110 is approximately 4.7 mA because the first memory (eg, DRAM) 110 is powered off most of the time when the NN inference 150 is executed.

[0062] In one embodiment, the signal may also include the time of the NN reasoning 150. For example, the signal of node 14 may also include the time of 5ms. In another embodiment, the signal may also include the operation of the NN reasoning 150. For example, the signal of node 30 may also include a convolution operation, which may include a convolution channel number. In yet another embodiment, the signal may also include a tensor of the NN reasoning 150. For example, if there is some weighted data that needs to be processed, the signal may include a "tensor", and the dynamic loading agent 330 may perform processing of the weighted data after receiving the tensor. In yet another embodiment, the signal may further include a kernel of the NN reasoning 150. For example, the signal may include a kernel and a tensor, and the dynamic loading agent 330 may execute the tensor by utilizing the kernel after receiving the kernel and the tensor.

[0063] In one embodiment, the dynamic loading agent 330 may also manage the power provided to the first memory (e.g., DRAM) 110 by configuring the operating mode of the first memory (e.g., DRAM) 110. For example, the dynamic loading agent 330 may configure the first memory (e.g., DRAM) 110 to operate in a high-performance mode by scaling the data rate and row cycle time of the first memory (e.g., DRAM) 110. In another embodiment, the dynamic loading agent 330 may also manage the power provided to the first memory (e.g., DRAM) 110 by dynamically scaling the voltage and / or frequency (DVFS) applied to the first memory (e.g., DRAM) 110, because power consumption is related to V 2 xf is proportional. For example, the dynamic loading agent 330 can manage the power provided to the first memory (e.g., DRAM) 110 by reducing the voltage applied to the first memory (e.g., DRAM) 110. Although reducing the voltage increases the propagation delay of the signal, which can cause errors when using unmodified timing parameters (e.g., the time required for data to be reliably detected and amplified in the row buffer), neural networks are universal approximators, and if the system used to model the neural network has a high error tolerance, the neural network can still work well.

[0064] In one embodiment, the reasoning execution scheme 370 may include a rule. In some embodiments, the reasoning execution scheme 370 is illustrated by taking a rule as an example, but the present invention is not limited to this example scenario. In one embodiment, the rule 370 generates a script for a certain layer of the NN reasoning 150 based on the input tensor from the previous layer of the NN reasoning 150. For example, Figure 5 As shown, this is the flow when the inference execution scheme 370 is a rule Figure 3 In another embodiment, the inference execution scheme 370 may include a model. For example, the model 370 is well trained to generate rules or scripts for the dynamic loading agent 330 to find. As another example, the dynamic loading agent 330 may be a NN executor of a policy model. In another embodiment, the inference execution scheme 370 may be man-made, such as a rule or script 370, or created by an offline optimizer or an online / runtime optimizer. Such an optimizer may analyze the model and find one of the best rules. For example, the online / runtime optimizer may create the rules 370 at runtime. As another example, the offline optimizer may first create the rules 370, and the dynamic loading agent 330 may apply the rules 370 to know when to load a portion of the NN reasoning 150 from the first memory (e.g., DRAM) 110 to the second memory (e.g., TCM) 320 and how to manage the power provided to the first memory (e.g., DRAM) 110.

[0065] Figure 6 A functional block diagram of another exemplary apparatus 600 for performing NN reasoning is shown according to some embodiments of the present invention. Figure 3The difference between the apparatus 300 and the apparatus 600 is that the apparatus 600 further includes a direct memory access (DMA) controller 680, which couples the first memory (e.g., DRAM) 110, the second memory (e.g., TCM) 320, and the dynamic loading agent 330. In one embodiment, the dynamic loading agent 330 can instruct the DMA controller 680 to load a portion 151 and a next portion 152 of the NN reasoning 150 directly to the second memory (e.g., TCM) 320 without the involvement of the dynamic loading agent 330. For example, after receiving a signal from the speculative executor 340, the dynamic loading agent 330 searches for the speculative execution scheme 370 stored in the second memory (e.g., TCM) 320 and learns that some data stored in the first memory (e.g., DRAM) 110 needs to be loaded into the second memory (e.g., TCM) 320, and issues an instruction to the DMA controller 680 by sending some information to the DMA controller 680, including a read instruction, the number of words to be read, the starting position in the first memory (e.g., DRAM) 110 to be read from, and the address of the second memory (e.g., TCM) 320. Then, the DMA controller 680 loads the data directly from the first memory (e.g., DRAM) 110 to the second memory (e.g., TCM) 320 based on the information without passing through the dynamic loading agent 330. After the loading is completed, the DMA controller 680 can use the system bus to send an interrupt signal to the dynamic loading agent 330 to notify the DMA controller 680 that it has been completed. Compared with the device 300, the device 600 can have better speed performance. Therefore, Figure 6 The first memory (eg, DRAM) 110 is Figure 3 The first memory (eg, DRAM) 110 consumes less power.

[0066] Figure 7 A functional block diagram of another exemplary apparatus 700 for performing NN reasoning is shown according to some embodiments of the present invention. Figure 6The difference between the apparatus 600 and the apparatus 700 is that the apparatus 700 may further include a cache controller 790, which is coupled to the dynamic loading agent 330. In one embodiment, the cache controller 790 is also coupled between the first memory (e.g., DRAM) 110 and the cache 720. In another embodiment, the cache 720 is also coupled to the inference executor 340. In yet another embodiment, the cache controller 790 may load another portion 721 of the NN inference 150 stored in the first memory (e.g., DRAM) 110 to the cache 720, and the inference executor 340 may execute another portion 721 of the NN inference 150 that has been loaded from the first memory (e.g., DRAM) 110 to the cache 720.

[0067] In one embodiment, the apparatus 100, 300, 600 and 700 may be configured to perform various functions according to one or more embodiments or examples described herein. Thus, the apparatus 100, 300, 600 and 700 may provide means for implementing the techniques, processes, functions, components, systems described herein. The apparatus 100, 300, 600 and 700 may be a general purpose computer in some embodiments, and, in other embodiments, may be an apparatus including specially designed circuits to implement the various functions, components or processes described herein. The apparatus 100, 300, 600 and 700 may optionally include other components, such as input and output devices, additional or signal processing circuits, etc. Thus, the apparatus 100, 300, 600 and 700 may be capable of performing other additional functions, such as executing applications, and processing alternative communication protocols.

[0068] The processes and functions described herein can be implemented as a computer program, which, when executed by one or more processors, can cause one or more processors to perform the corresponding methods and functions. The computer program can be stored or distributed on a suitable medium, such as an optical storage medium or solid-state medium provided together with other hardware or as a part thereof. The computer program can also be distributed in other forms, for example, via the Internet or other wired or wireless telecommunications systems. For example, a computer program can be obtained and loaded into a device, including obtaining the computer program through a physical medium or a distributed system (including, for example, from a server connected to the Internet).

[0069] A computer program may be accessed from a computer-readable medium that provides program instructions for use by or in conjunction with a computer or any instruction execution system. The computer-readable medium may include any device that stores, communicates, propagates, or transmits the computer program for use by or in conjunction with an instruction execution system, device, or apparatus. The computer-readable medium may be a magnetic, optical, electronic, electromagnetic, infrared, or semiconductor system (or device or apparatus) or propagation medium. The computer-readable medium may include a computer-readable non-transitory storage medium, such as a semiconductor or solid-state memory, a magnetic tape, a removable computer disk, a random access memory (RAM), a read-only memory (ROM), a magnetic disk, and an optical disk. Computer-readable non-transitory storage media may include all types of computer-readable media, including magnetic storage media, optical storage media, flash memory media, and solid-state storage media.

[0070] Figure 8 A flowchart of an exemplary method 800 for performing NN reasoning is shown according to some embodiments of the present invention. Aspects of method 800 can be performed on smart phones, wearable devices, and other IoT devices, such as the devices 100, 300, 600, and 700 shown and described in the previous figures. In various embodiments, some steps of the method 800 shown may be performed simultaneously or in an order different from the order shown, may be replaced by other method steps, or may be omitted. Additional method steps may also be performed as needed. Method 800 may include steps 810, 820, and 830.

[0071] In step 810, a portion of the NN reasoning is loaded from a first memory storing the NN reasoning to a second memory. For example, the second memory may be a TCM or an SRAM. For another example, the first memory may be a DRAM, a bus SRAM, or a serial flash memory. In one embodiment, a portion 151 of the NN reasoning 150 may be loaded from a first memory (e.g., DRAM) 110 to a second memory (e.g., TCM) 320.

[0072] In step 820, the portion of the NN reasoning loaded on the second memory is executed, and a signal is generated based on the execution progress of the NN reasoning. In one embodiment, the portion 151 of the NN reasoning 150 loaded on the second memory (e.g., TCM) 320 can be executed by the reasoning executor 340, and the reasoning executor 340 generates signals for nodes 13 and 30 based on the execution progress of the NN reasoning 150.

[0073] In step 830, the next portion of the NN reasoning stored in the first memory is loaded into the second memory, and the power supplied to the first memory is managed based on the signal and the reasoning execution scheme stored in the second memory. For example, the power supplied to the first memory is managed by powering on / off the first memory, configuring the operating mode of the first memory, or scaling the voltage and / or frequency applied to the first memory. As another example, the reasoning execution scheme can be a script, a rule, or a model. As yet another example, the signal can include an operation, a node ID, a tensor, a kernel, and / or a time for the NN reasoning. In one embodiment, the next portion 152 of the NN reasoning 150 stored in the first memory (e.g., DRAM) 110 is loaded into the second memory (e.g., TCM) 320, and the power provided to the first memory (e.g., DRAM) 110 is managed by powering on / off the first memory (e.g., DRAM) 110 based on the script 370 (e.g., DRAM powered on, DRAM powered off, data 0-12 moved to TCM and data 13 and 14 moved to TCM) and the signal (e.g., signal of node 0, node 13, node 14, node 15, node 30). The method 800 may continue to perform steps 820 and 830 until the entire NN reasoning 150 is completed.

[0074] Fig. 9 A flowchart of another exemplary method 900 for performing NN reasoning is shown according to some embodiments of the present invention. Aspects of method 900 can be performed on smart phones, wearable devices, and other IoT devices, such as the devices 100, 300, 600, and 700 shown and described in the aforementioned figures. In various embodiments, some steps of the method 900 shown may be performed in an order different from the order shown or performed simultaneously, may be replaced by other method steps, or may be omitted. Additional method steps may also be performed as needed. Method 900 may include steps 910, 920, 930, 940, and 950.

[0075] In step 910, the inference executor (e.g., the inference executor 340) executes a portion of the NN inference that has been loaded on the second memory, for example, a portion 151 of the NN inference 150 that has been loaded on the second memory (e.g., TCM) 320. In step 920, the inference executor 340 sends a signal based on the execution progress of the NN inference 150, for example, a node ID, time, operation, tensor, kernel, etc. In step 930, the dynamic loading agent (e.g., the dynamic loading agent 330) receives the signal. In step 940, after receiving the signal, the dynamic loading agent 330 searches for an inference execution scheme stored in the second memory 320 and indexed by the signal, for example, the inference execution scheme 370 stored in the second memory (e.g., TCM) 320 is indexed by the node ID. In step 950, the dynamic loading agent 330 performs corresponding actions based on the inference execution scheme 370 indexed by the signal, such as driving a DMA controller (e.g., DMA controller 680) to load data (e.g., the next part 152 of the NN inference 150) at a specified time (preloading), or loading multiple data (batch processing) from a first memory 110 (e.g., DRAM 110) to a second memory (e.g., TCM) 320, enabling or disabling AP (application processor) / DRAM / DMA power (scheduling power on / off), and / or passing configurations or settings to operators and performing special operations (fine-grained operators). Steps 910, 920, 930, 940, and 950 of the method 900 may continue to be executed until the entire NN inference 150 has been loaded from the first memory (e.g., DRAM) 110 to the second memory (e.g., TCM) 320. In one embodiment, steps 910, 940, and 950 may be performed synchronously. For example, the dynamic loading agent 330 and the inference executor 340 may be included on a single chip, and the single chip may sequentially perform steps 910, 940, and 950. In another embodiment, step 910 and steps 940 and 950 may be performed asynchronously. For example, the dynamic loading agent 330 and the inference executor 340 may be two different chips, and the two different chips may respectively perform steps 910, 940, and 950 in parallel.

[0076] The subject matter described herein sometimes describes different components contained in other different components, or different components connected to other different components. It should be understood that the described structure is only used as an example, and in fact, the same function can also be achieved by implementing other structures. Conceptually, any component configuration that can achieve the same function is effectively "associated" to achieve the desired function. Therefore, any two components combined herein to achieve a specific function can be regarded as "associated" with each other to achieve the desired function, regardless of their structure or intermediate components. Similarly, any two components associated in this way can also be regarded as "operationally connected" or "operationally coupled" to achieve the desired function, and any two components that can be associated in this way can also be regarded as "operationally couplable" to each other to achieve the desired function. Specific examples of operational coupling include but are not limited to physically pairable and / or physically interactive components and / or wirelessly interactive and / or wirelessly interactive components and / or logically interactive and / or logically interactive components.

[0077] In addition, for any plural and / or singular words used in this article, those skilled in the art can convert the plural to the singular and / or the singular to the plural according to whether the context and / or application scenario is appropriate. For the sake of clarity, various substitutions between singular and plural in this article are clearly stipulated here.

[0078] In addition, it will be understood by those skilled in the art that, in general, the words used herein, and in particular in the appended claims, for example, the words used in the body of the claims generally have an "open" meaning, for example, the word "comprising" should be understood as "including but not limited to", the word "having" should be understood as "having at least", the word "including" should be understood as "including but not limited to", etc. It will be further understood by those skilled in the art that if an introductory claim listing intends to include a specific value, such intention will be explicitly listed in the claim, and if it is not listed, such intention does not exist. To assist understanding, for example, the appended claims may contain introductory phrases such as "at least one" and "one or more" to introduce claim listings. However, such a phrase should not cause the claim list to be interpreted as if the introduction of the indefinite article "a" means that any particular claim containing such an introductory claim list is limited to embodiments containing only one such list, even when the same claim includes the introductory phrase "one or more" or "at least one" and an indefinite article such as "a", that is, "a" should be interpreted as "at least one" or "one or more". The same is true for the use of definite articles to introduce claim lists. In addition, even if a specific value is explicitly listed in an introductory claim list, those skilled in the art will recognize that such a list should be understood to include at least the listed value, for example, when only "two lists" without any other limitation, it means at least two lists, or two or more lists. In addition, if the phrase "at least one of A, B, and C, etc." is used, it is generally understood by those skilled in the art that, for example, "a system having at least one of A, B, and C" will include, but is not limited to, a system having only A, a system having only B, a system having only C, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B, and C, etc. If the phrase "at least one of A, B, or C, etc." is used, it is generally understood by those skilled in the art that, for example, "a system having at least one of A, B, or C" will include, but is not limited to, a system having only A, a system having only B, a system having only C, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B, and C, etc. It is further understood by those skilled in the art that almost all separating words and / or phrases connecting two or more alternative words appearing in the specification, claims, or drawings should be understood to take into account all possibilities, that is, including one of all words, either of the two words, or both words. For example, the phrase "A or B" should be understood to include the following possibilities: "A", "B", or "A and B".

[0079] Although the present invention has been described by way of example and in terms of preferred embodiments, it should be understood that the present invention is not limited to the disclosed embodiments. Instead, it is intended to cover various modifications and similar structures (as will be apparent to those skilled in the art), for example, combinations or replacements of different features in different embodiments. Therefore, the scope of the appended claims should be given the broadest interpretation to cover all such modifications and similar structures.

Claims

1. A device for executing a program, characterized in that The device includes: an executor configured to execute a portion of the program and generate a signal based on the execution progress of the program, wherein the program is a neural network (NN) inference, and the portion of the program has been loaded from a first memory storing the program to a second memory; and A dynamic loading agent is coupled to the executor, the first memory and the second memory, and is configured to: determine whether a next part of the program stored in the first memory needs to be loaded into the second memory based on the signal from the executor and the speculative execution scheme stored in the second memory, and manage the power provided to the first memory by controlling power on / off of the first memory.

2. The device according to claim 1, characterized in that The dynamic loading agent is also configured to manage power provided to the first memory by configuring an operating mode of the first memory or scaling a voltage and / or frequency provided to the first memory.

3. The device according to claim 1, characterized in that The reasoning execution scheme includes scripts, rules or models.

4. The device according to claim 3, characterized in that This rule generates a script for a specific layer of a program based on the input tensors from the previous layer of the program.

5. The device according to claim 1, characterized in that The signal includes the program's operations, node IDs, tensors, kernels, and / or time.

6. The device according to claim 1, characterized in that The second memory is a tightly coupled memory TCM or a static random access memory SRAM.

7. The device according to claim 1, characterized in that The first memory is a dynamic random access memory DRAM, a bus SRAM or a serial flash memory.

8. The device according to claim 1, characterized in that: The executor is a central processing unit CPU, the dynamic loading agent is a microcontroller unit MCU or the CPU is abnormal; or, The executor is a deep learning accelerator DLA, and the dynamic loading agent is an MCU.

9. The device according to claim 1, characterized in that The dynamic loading agent and the executor are located in a single chip.

10. The device according to claim 1, characterized in that The device also includes a direct memory access (DMA) controller coupled to the dynamic loading agent, the first memory, and the second memory, wherein the dynamic loading agent is further configured to instruct the DMA controller to load the next portion of the program stored in the first memory into the second memory.

11. The device according to claim 1, characterized in that The speculative execution plan is manually set or created by an offline optimizer or an online / runtime optimizer.

12. A method for executing a program, characterized in that include: Loading a portion of the program from a first memory storing the program to a second memory, wherein the program is a neural network (NN) inference; executing the portion of the program loaded on the second memory and generating a signal based on the progress of the execution of the program; and, Based on the signal and the speculative execution scheme stored in the second memory, it is determined whether a next portion of the program stored in the first memory needs to be loaded into the second memory, and power supplied to the first memory is managed by controlling power on / off of the first memory.

13. The method according to claim 12, characterized in that Managing power provided to the first memory also includes: configuring an operating mode of the first memory; or scaling a voltage and / or frequency provided to the first memory.

14. The method according to claim 12, characterized in that The reasoning execution scheme includes scripts, rules or models.

15. The method according to claim 14, characterized in that This rule generates a script for a specific layer of a program based on the input tensors from the previous layer of the program.

16. The method according to claim 12, characterized in that The signal includes the program's operations, node IDs, tensors, kernels, and / or time.

17. The method according to claim 12, characterized in that The second memory is a tightly coupled memory TCM or a static random access memory SRAM.

18. The method according to claim 12, characterized in that The first memory is a dynamic random access memory DRAM, a bus SRAM or a serial flash memory.

19. The method of claim 12, wherein: The speculative execution plan is manually set or created by an offline optimizer or an online / runtime optimizer.

20. The method of claim 12, wherein: The operations of executing the portion of the program, loading a next portion of the program stored in the first memory to the second memory, and managing power supplied to the first memory are asynchronously performed.

Images